General

OK so I am trying to pull from the Microsoft Graph API. To do that, I must first POST to get a bearer token. I assume I can use discover type of HTTP Request to get it. A JSON response will be returned with the field access_token containing what I need.How do I use the data from access_token as a collect header for Authorization? This is a first for me so I need some hand holding here.Thanks!

I was previously using SPLUNK to get data via a SPLUNK add-on that used an API query to pull down detections from the CrowdStrike platform. We are moving away from SPLUNK and trying to use CRIBL to get CrowdStrike data. CrowdStrike have a SIEM connector but this requires additional infrastructure and I am trying to see if I can use CRIBL to query CrowdStrike via the API to pull this information down. Has anyone attempted or tried this? I was stuck with the client id and key with the REST API, but I think this needs to be done separately and only use an authentication token (bearer) to make the connection.

The Microsoft Windows Events pack is currently dropping "SourceName=AD FS Auditing" events from the Security logs. I found the two lines that are not properly filtering the events.Pipeline: Windows Classic Events, "Final Cleanup" lines 27 & 28, (Serialize & Eval).I turned it off, but still working to get the events to not drop and convert to json.Q: Wondering if this filter is currently being updated/corrected?

Hi all, this is my first post on this forum, so hello. We have just begun to use CRIBL and I am still in a learning phase. I am wondering if there are more parsers that can be added to the library or whether you have to create them yourself? As an example I have a CRIBL instance that is receiving json data from another CRIBL instance (with the data having been prebaked SPLUNK) for a Cisco ASA log. The content of the log entry is in either the raw or Message field and I would like to extract into key value pairs. I have tried the some functions but it looks like I have to do custom regex (and I am not good with regex).

Can someone assist in explaining the difference between setting the Cribl General TLS settings versus the distributed TLS settings and what server certificates need to be applied? Where do we need to put the server.pem certs versus the web certs? Currently I am using mutual authentication and each host has their own self-signed certificates.

I am trying to pull event data from a remote location with http rest collector.It requires 2 stages to pull the data.Discovery will pull an array of job id's (and details).Collection will pull the results of each job from the array.When I set the collector up for discovery, I get the correct information including the ID.Using this id variable in the collection url I only get 1 event per id. If I hard code this id in the url AND disable collection, I get all events for that job.The throughput looks ok.But the capture window only captured 2 events. 1 from each job id.

Hello,I have a few general questions on Persistent Queues(PQ). I see PQ settings for Source & Destination, and further my question are specific to size & compression. 1. If I enable PQ and let the default size to be 1 MB & with compression turned on and if I get a burst of 100MB of events which Cribl Stream is unable to handle, does it create a 100 compressed files of 1 MB each? I understand my events will get delayed in getting indexed in Splunk, but will this guarantee that I eventually will get these events(Assuming I have PQ enabled on the destination side too).2. Is compression given as an option just so to mitigate disk utilization? 3. When I turn on PQ, I do see a few ndjson.*.gz files in the queues directory, but these are from about 20-60 minutes ago when I first enabled PQ, when i do a cursory check of the data, I am seeing all the events in Splunk. Does this mean the Cribl is busy and eventually will get through processing these gz files may be in a few ho

Hi there,Is a Pulsar connector for input/output analog Kafka planned?This would be a important addition to cribl stream, as pulsar is a fast growing platform and (for me) a successor to kafka.Best regards,

I am trying to setup SSO for our Cribl environments using Okta. I have successfully completed this within our DEV environment (which does not use agentless SSO). When testing the SSO setup in Prod, the login process gets stuck at the agentless desktop SSO step and will not progress past it.Does anyone have any ideas/recommendations as to what can be changed within Cribl, Okta, or somewhere else, that may help resolve this problem? We have not seen this issue with any other application at our enterprise except for Cribl.Thank you,Andrew

Hi, I have problem with deploy config from Leader to Worker. I have distributed deployment with 2 Workers and 1 worker group (testing, free license used). I configured secure connection between Leader and Workers according to https://docs.cribl.io/stream/securing-communications 1. For the first look everything is OK - both Workers are visible on Manage Worker Nodes on Leader as alive, I can connect to Workers GUI (Remote UI Access is on), config version is OK etc.Problem is when I want to deploy config changes to Workers. On Manage Worker Nodes I see yellow exclamation mark and neverending rotating circle in Config version column and config changes are not propagated to Workers.In log on Leader I see this:{"time":"2022-06-17T13:01:21.350Z","cid":"api","channel":"CriblMaster","level":"info","message":"sending config update request","group":"default","version":"8627dcf","logStreamEnv":"master","worker":"ff5ea4c4-51eb-4aa4-a650-2e6c8251bb21"}{"time":"2022-06-17T13:01:21.363Z","cid":"api",

Are the available Redis commands in the dropdown list or should I go by the complete list at the URL provided when I hover over the ‘?’?I’m trying to set multiple values for a hash and I’m having trouble grabbing the key-value pairs from my event to send. I’m not sure if it’s a limitation or I’m doing something incorrectly. I’m using ‘__e’ to grab the entire event (Args section) and specifying a key. The Command section allows me to put in hmset.

Can I use self-signed certificate with a REST collector? Does Cribl perform any TLS certificate validity checks?

If I need to accept an SNMP trap from a source, and send it to multiple trap destinations, will Cribl spoof the source IP and send the trap “un-altered” to the final destinations? Meaning will it preserve the source IP address from the original trap IP header?

I’ve made the switch from a single instance to a distributed setup (1 leader and 2 worker nodes). My old server, which receives logs from a variety of sources, is now acting as a worker. What happens if this worker goes down, because all of the sources are sending logs to one single worker IP? What should I do to guarantee appropriate high availability?

Not a question but rather something useful. I have data that is in key:value format, using key=value in the extract format did not work. I was able to extract every key and value with this one regex extract.(?<_NAME_0>.*):\s(?<_VALUE_0>.*)

i have a use case where we have a nfs share that applications writes logs too i would like to monitor these while being written to (file monitor?) but distribute the load across workers. is that possible?

What does this message indicate on the Splunk TCP source:message:Failed to parse s2s payload{}error:αmessage:Cannot advance offset 2023514992 past end of event 74αstack:Error: Cannot advance offset 2023514992 past end of event 74at p.advance (/opt/cribl/bin/cribl.js:14:12271888)at h._readEvents (/opt/cribl/bin/cribl.js:14:12274323)at h._transform (/opt/cribl/bin/cribl.js:14:12275791)at h.Transform._read (_stream_transform.js:205:10)at h.Transform._write (_stream_transform.js:193:12)at writeOrBuffer (_stream_writable.js:352:12)at h.Writable.write (_stream_writable.js:303:10)at Socket.ondata (_stream_readable.js:719:22)at Socket.emit (events.js:315:20)at Socket.EventEmitter.emit (domain.js:486:12)Show less

Hi Guys,I have setup a test environment on my laptop using VBOX and with a Centos7 edge leader node and worker using ports 9000(ui) and 4200 for internal comms.I’ve installed the worker node using curl. Seems there is an issue with the worker node connecting to the leader as shown below. From the worker node, telnet to the leader nodes port 9000 and 4200 connect fine. But not sure what additional stuff to check based on the below messages:{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"info","message":"attempting to connect","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"will retry to connect","nextConnectTime":1653267018346}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"connecting","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.264Z","cid":"api","channel":"input:D

We have a collector script (python script) that can take up to 10 min to pull down all the data (the script will cache it) and then “print” it out so that Cribl can then ingest it. Ends up being equivalent to a 650MB CSV file. The first ad hoc full run seemed to have timed out even though I had 0 set for job timeout. Debug shows a lot of “failed to pop task” messages while it is running. Any ideas?

So I have a script that will collect data from an API. The data is returns is CSV formatted, the first line will be the list of fields. How do you tell Stream to use that first line as the headers?

One of the neatest features of the product is the live preview. This works great for relatively small events.If I have a massive event with a pretty big intensive pipeline to boot, things time out and it really is a PITA to debug my code. How do people work through such a work flow?I usually increase the timeout from 10 seconds to 60 seconds and "play" the pipeline, but I wanted to know if people had some other nifty hacks.Thank you!b1scu1t

Hi, not sure if this is a FREE license thing or something I have wrong with docker. Here is my docker compose.version: 3.5services: master: image: ${CRIBL_IMAGE:-cribl/cribl:latest} environment: - CRIBL_DIST_MODE=master - CRIBL_DIST_MASTER_URL=tcp://criblmaster@0.0.0.0:4200 - CRIBL_VOLUME_DIR=/share/homes/chris/cribl:/opt/cribl/config-volume ports: - 19000:9000 hostname: cribl_leader volumes: - /share/homes/chris/cribl:/opt/cribl/config-volume workers: image: ${CRIBL_IMAGE:-cribl/cribl:latest} depends_on: - master hostname: cribl_worker environment: - CRIBL_DIST_MODE=worker - CRIBL_DIST_MASTER_URL=tcp://criblmaster@0.0.0.0:4200 ports: - 9000Every-time I modify the compose and redeploy, I have to re-input config (user / password, registration, etc). The homes DIR is on the host and has open permissions. I do see data being written in there. Is there a setting I may be missing? Thank you Chris

when i try to upgrade splunk app via search head deployer i get an error that it fails to apply bundle because cribl is busyError while deploying apps to first member, aborting apps deployment to all members: Error while updating app=cribl on target=https://10.150.22.183:8089: Non-200/201 status _code=500; {"messages":[{"type":"ERROR","text":"\n In handler localapps: Error installing application: Failed to copy: /opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa 723/cribl/bin to /opt/splunk/etc/apps/cribl/bin. 2 errors occurred. Description for first 2: [{operation:\"copying source to destination\", error:\"Text file busy\", src:\" /opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa723/cribl/bin/cribl\", dest:\"/opt/splunk/etc/apps/cribl/bin/cribl\"}, {operation:\"transfering contents from source to d estination\", error:\"Text file busy\", src:\"/opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa723/cribl/bin\", dest:\"/opt/splunk/etc/apps/cribl/bin\"}]"}]}

When adding a GeoIP function to a pipeline, where do I upload .mmdb files (or where are they stored if I grabbed a pack that includes one)?

I wrote a pretty elaborate pipeline that extracts a bunch of metrics from a pretty hideously ugly event and further creates a multi metric that is further shipped to a metric index in splunk thanks to @david & @bdalpe.I noticed today when I was making some minor edits to this pipeline it would time out with the default 10 second, but when I made it 60 seconds, it ran to success. When I promote this pipeline into production, how can I make sure none of my pipelines timed out and dropped the data? When I query Splunk I do see the metrics, but I just want to make sure I am not losing stuff.Thanks!