Skip to main content

Products

Get answers, share tips, and explore best practices across the entire Cribl product suite.

  • 760 Topics
  • 0 Replies
760 Topics
D
Draco3New Participant
asked in General
CrowdStrike API

I was previously using SPLUNK to get data via a SPLUNK add-on that used an API query to pull down detections from the CrowdStrike platform. We are moving away from SPLUNK and trying to use CRIBL to get CrowdStrike data. CrowdStrike have a SIEM connector but this requires additional infrastructure and I am trying to see if I can use CRIBL to query CrowdStrike via the API to pull this information down. Has anyone attempted or tried this? I was stuck with the client id and key with the REST API, but I think this needs to be done separately and only use an authentication token (bearer) to make the connection.

22
D
T
talantacpNew Participant
asked in General
AD FS Auditing Events Dropping from Windows Pack

The Microsoft Windows Events pack is currently dropping "SourceName=AD FS Auditing" events from the Security logs. I found the two lines that are not properly filtering the events.Pipeline: Windows Classic Events, "Final Cleanup" lines 27 & 28, (Serialize & Eval).I turned it off, but still working to get the events to not drop and convert to json.Q: Wondering if this filter is currently being updated/corrected?

01
J
D
Draco3New Participant
asked in General
CRIBL - Parser Libraries

Hi all, this is my first post on this forum, so hello. We have just begun to use CRIBL and I am still in a learning phase. I am wondering if there are more parsers that can be added to the library or whether you have to create them yourself? As an example I have a CRIBL instance that is receiving json data from another CRIBL instance (with the data having been prebaked SPLUNK) for a Cisco ASA log. The content of the log entry is in either the raw or Message field and I would like to extract into key value pairs. I have tried the some functions but it looks like I have to do custom regex (and I am not good with regex).

02
K
T
talantacpNew Participant
asked in General
Certificates and Enabling TLS

Can someone assist in explaining the difference between setting the Cribl General TLS settings versus the distributed TLS settings and what server certificates need to be applied? Where do we need to put the server.pem certs versus the web certs? Currently I am using mutual authentication and each host has their own self-signed certificates.

22
T
A
amillerInspiring
asked in General
Rest Collector with/without discovery results inconsistent

I am trying to pull event data from a remote location with http rest collector.It requires 2 stages to pull the data.Discovery will pull an array of job id's (and details).Collection will pull the results of each job from the array.When I set the collector up for discovery, I get the correct information including the ID.Using this id variable in the collection url I only get 1 event per id. If I hard code this id in the url AND disable collection, I get all events for that job.The throughput looks ok.But the capture window only captured 2 events. 1 from each job id.

01
A
G
Govardhanen GopalParticipating Frequently
asked in General
General Peristent Queue Questions

Hello,I have a few general questions on Persistent Queues(PQ). I see PQ settings for Source & Destination, and further my question are specific to size & compression. 1. If I enable PQ and let the default size to be 1 MB & with compression turned on and if I get a burst of 100MB of events which Cribl Stream is unable to handle, does it create a 100 compressed files of 1 MB each? I understand my events will get delayed in getting indexed in Splunk, but will this guarantee that I eventually will get these events(Assuming I have PQ enabled on the destination side too).2. Is compression given as an option just so to mitigate disk utilization? 3. When I turn on PQ, I do see a few ndjson.*.gz files in the queues directory, but these are from about 20-60 minutes ago when I first enabled PQ, when i do a cursory check of the data, I am seeing all the events in Splunk. Does this mean the Cribl is busy and eventually will get through processing these gz files may be in a few ho

11
B
R
redukNew Participant
asked in Stream
Populating lookup table using incoming events

As the title says, I’m looking to create and update a lookup table based on incoming events, somewhat similar to the outputlookup function in splunk. Is there any way to update or create a table based on fields from events passing through a pipeline?

03
P
0
0xbaemNew Participant
asked in General
Pulsar Connector

Hi there,Is a Pulsar connector for input/output analog Kafka planned?This would be a important addition to cribl stream, as pulsar is a fast growing platform and (for me) a successor to kafka.Best regards,

11
J
A
astvinceNew Participant
asked in General
Cribl Authentication with Okta and Agentless Desktop SSO

I am trying to setup SSO for our Cribl environments using Okta. I have successfully completed this within our DEV environment (which does not use agentless SSO). When testing the SSO setup in Prod, the login process gets stuck at the agentless desktop SSO step and will not progress past it.Does anyone have any ideas/recommendations as to what can be changed within Cribl, Okta, or somewhere else, that may help resolve this problem? We have not seen this issue with any other application at our enterprise except for Cribl.Thank you,Andrew

02
D
B
Brandon McCombsEmployee
asked in Stream
How to properly distribute UDP syslog packets across worker nodes and processes when using F5 LBs?

An F5 load-balancer was setup and should be sending syslog traffic to some Cribl Stream workers using round-robin. Currently there is a single source of events, for testing purposes. The events are almost exclusively going to a single worker, which is unexpected.The expectation is that the events would be load-balanced and that there would be a fair distribution across the worker nodes. According to F5 team, the connection gets load-balanced, so as more sources added, traffic will start showing on the other workers. That is to be expected but since UDP syslog is being used the events from a single source should already getting load balanced.

12
E
D
ddukeKnown Participant
asked in Stream
Methods supported in Unroll Function

Does the unroll function support any other methods than split() ?Please give me your examples 🙂

07
P
S
StevengoossensNew Participant
asked in Stream
Problem sending data to Elastic Data Stream

Hi,I’ve configured an index template in Elastic for a specific log source and set it to use a data stream instead of the indices.Whenever Cribl starts sending data, the data stream is created, but no data is added to is. I presume this has something to do with the _bulk API, since data streams only support the op_type create.Does anyone have a working configuration for logging towards data streams in Elastic?

01
K
L
Lukáš MečířNew Participant
asked in General
Problem with deploy config from Leader to Workers

Hi, I have problem with deploy config from Leader to Worker. I have distributed deployment with 2 Workers and 1 worker group (testing, free license used). I configured secure connection between Leader and Workers according to https://docs.cribl.io/stream/securing-communications 1. For the first look everything is OK - both Workers are visible on Manage Worker Nodes on Leader as alive, I can connect to Workers GUI (Remote UI Access is on), config version is OK etc.Problem is when I want to deploy config changes to Workers. On Manage Worker Nodes I see yellow exclamation mark and neverending rotating circle in Config version column and config changes are not propagated to Workers.In log on Leader I see this:{"time":"2022-06-17T13:01:21.350Z","cid":"api","channel":"CriblMaster","level":"info","message":"sending config update request","group":"default","version":"8627dcf","logStreamEnv":"master","worker":"ff5ea4c4-51eb-4aa4-a650-2e6c8251bb21"}{"time":"2022-06-17T13:01:21.363Z","cid":"api",

12
L
D
dvanshNew Participant
asked in Stream
Azure Event Hub JSON

HiWe have setup event hub to send data stream to cribl and our now trying to extract the json into individual events. We have tried the event breaker JSON Array method and JSON unroll method. Neither seems to work. Has anyone tried this successfully or have pointers?Thank you

32
D
D
dmtrsNew Participant
asked in General
POST to URL using Javascript

Is there a way to set up a Destination as a POST request (w/authentication)? I tried using Code and modifying the code for DNS lookup on this page (https://cribl.io/blog/extending-cribl-building-custom-functions/). I got errors about using Promise.

00
J
S
Shane DanielsEmployee
asked in General
Can I use self-signed certificate with a REST collector?

Can I use self-signed certificate with a REST collector? Does Cribl perform any TLS certificate validity checks?

01
J
J
jccurtisNew Participant
asked in General
SNMP Trap Forwarding Question

If I need to accept an SNMP trap from a source, and send it to multiple trap destinations, will Cribl spoof the source IP and send the trap “un-altered” to the final destinations? Meaning will it preserve the source IP address from the original trap IP header?

01
S
S
Shane DanielsEmployee
asked in General
Changing from single instance to distributed setup - high availability

I’ve made the switch from a single instance to a distributed setup (1 leader and 2 worker nodes). My old server, which receives logs from a variety of sources, is now acting as a worker. What happens if this worker goes down, because all of the sources are sending logs to one single worker IP? What should I do to guarantee appropriate high availability?

12
S
E
ekalbyNew Participant
asked in General
Regex to Grab Key:Value

Not a question but rather something useful. I have data that is in key:value format, using key=value in the extract format did not work. I was able to extract every key and value with this one regex extract.(?<_NAME_0>.*):\s(?<_VALUE_0>.*)

02
P
S
SonOfBuziInspiring
asked in General
Technical question on stream file monitor

i have a use case where we have a nfs share that applications writes logs too i would like to monitor these while being written to (file monitor?) but distribute the load across workers. is that possible?

02
B
M
mikeyleeNew Participant
asked in Stream
Can only see splunk metrics.log events and not other events from monitored files via the Splunk UF

Hi guys,I’m testing cribl stream for the 1st time by getting a splunk UF to forward to a Worker on the same centos7 host (ip: 10.0.2.9).When checking Live Data, I only see events from /opt/splunkforwarder/var/log/splunk/metrics.logI was expecting to see events from /var/log/audit/audit.log and /var/log/secure which are listed as being monitored files:/opt/splunkforwarder/bin/splunk list monitorYour session is invalid. Please login.Splunk username: adminPassword:Monitored Directories:$SPLUNK_HOME/var/log/splunk/opt/splunkforwarder/var/log/splunk/audit.log/opt/splunkforwarder/var/log/splunk/btool.log/opt/splunkforwarder/var/log/splunk/conf.log/opt/splunkforwarder/var/log/splunk/configuration_change.log/opt/splunkforwarder/var/log/splunk/dfm_stderr.log/opt/splunkforwarder/var/log/splunk/dfm_stdout.log/opt/splunkforwarder/var/log/splunk/first_install.log/opt/splunkforwarder/var/log/splunk/health.log/opt/splunkforwarder/var/log/splunk/license_usage.log/opt/splunkforwarder/var/log/sp

23
J
F
fshelton79New Participant
asked in General
Odd error message

What does this message indicate on the Splunk TCP source:message:Failed to parse s2s payload{}error:αmessage:Cannot advance offset 2023514992 past end of event 74αstack:Error: Cannot advance offset 2023514992 past end of event 74at p.advance (/opt/cribl/bin/cribl.js:14:12271888)at h._readEvents (/opt/cribl/bin/cribl.js:14:12274323)at h._transform (/opt/cribl/bin/cribl.js:14:12275791)at h.Transform._read (_stream_transform.js:205:10)at h.Transform._write (_stream_transform.js:193:12)at writeOrBuffer (_stream_writable.js:352:12)at h.Writable.write (_stream_writable.js:303:10)at Socket.ondata (_stream_readable.js:719:22)at Socket.emit (events.js:315:20)at Socket.EventEmitter.emit (domain.js:486:12)Show less

01
B
M
mikeyleeNew Participant
asked in General
Problem with Worker connecting to Leader node

Hi Guys,I have setup a test environment on my laptop using VBOX and with a Centos7 edge leader node and worker using ports 9000(ui) and 4200 for internal comms.I’ve installed the worker node using curl. Seems there is an issue with the worker node connecting to the leader as shown below. From the worker node, telnet to the leader nodes port 9000 and 4200 connect fine. But not sure what additional stuff to check based on the below messages:{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"info","message":"attempting to connect","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"will retry to connect","nextConnectTime":1653267018346}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"connecting","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.264Z","cid":"api","channel":"input:D

64
M
G
Govardhanen GopalParticipating Frequently
asked in General
Fundamental question on git deployment in distributed setup

I am new to the distributed world, so please bear with me. I have a distributed setup with 2 worker groups(default & 2ndgroup). I have assigned nodes to each of the above 2 groups. In the leader ui when I navigate into the "2ndgroup" and make changes to my pipelines and commit and distribute the changes, it gets pushed to my git repo(remote) & the worker nodes.When I further navigate into the worker nodes and do a live capture of events and make changes to the pipelines, how do I push these changes to the git remote and further when I view it in the leader node under group view reflect the changes that were done at the node level? May be I am missing something obvious, can somebody shed some light on what I am not getting right in this flow?

00
B
S
Shawn CannonKnown Participant
asked in General
Python Collector Script issue

We have a collector script (python script) that can take up to 10 min to pull down all the data (the script will cache it) and then “print” it out so that Cribl can then ingest it. Ends up being equivalent to a 650MB CSV file. The first ad hoc full run seemed to have timed out even though I had 0 set for job timeout. Debug shows a lot of “failed to pop task” messages while it is running. Any ideas?

43
B

Badge winners

Show all badges
Terms & ConditionsCookie settings