Skip to main content

General

Can't find what you're looking for? Ask it here!

  • 425 Topics
  • 2,587 Replies
425 Topics
A
AustinrNew Participant
asked in General
AWS OpenSearch Service and Cribl

Has anyone connected up an AWS Opensearch Service instance to Cribl? I created a local backend user in my domain and gave it what I thought were good permissions but I still get a 401 error when I test the connection. Can anyone share what they did to get this to work?

103
B
D
dmtrsNew Participant
asked in General
Using HMSET to update Redis

Are the available Redis commands in the dropdown list or should I go by the complete list at the URL provided when I hover over the ‘?’?I’m trying to set multiple values for a hash and I’m having trouble grabbing the key-value pairs from my event to send. I’m not sure if it’s a limitation or I’m doing something incorrectly. I’m using ‘__e’ to grab the entire event (Args section) and specifying a key. The Command section allows me to put in hmset.

11
B
G
GeoffBNew Participant
asked in General
Capturing Cribl Login attempts

I am trying to capture login attempts (successful/unsuccessful) to Cribl. I can see that the cribl.log file contains logs for these (with a type of “auth”).I have enabled CriblLogs as a source.The log level for channel “auth” is set to Info.When I do a capture on the CriblLogs source I don’t see these “auth” events, I do however see (some) other events - any suggestions as to what I might be doing wrong?Thanks.

63
B
T
Tony REmployee
asked in General
Using segment in path to eval host

I am currently bringing in a bunch of Apache logs. The sites are divided up in /var/log/httpd (eg. /var/log/httpd/site1/access.log, /var/log/httpd/site2/access.log). It would be amazing to set the base folder, the max depth, the allow list, and the host segment of the path. Is there a way to read in the path of the file to set the host field? Currently doing this in Splunk with the “host_segment” function in the inputs.conf file.

12
A
S
Shawn CannonKnown Participant
asked in General
Using Discover in Rest collector to populate a collect header

OK so I am trying to pull from the Microsoft Graph API. To do that, I must first POST to get a bearer token. I assume I can use discover type of HTTP Request to get it. A JSON response will be returned with the field access_token containing what I need.How do I use the data from access_token as a collect header for Authorization? This is a first for me so I need some hand holding here.Thanks!

51
B
D
Draco3New Participant
asked in General
CrowdStrike API

I was previously using SPLUNK to get data via a SPLUNK add-on that used an API query to pull down detections from the CrowdStrike platform. We are moving away from SPLUNK and trying to use CRIBL to get CrowdStrike data. CrowdStrike have a SIEM connector but this requires additional infrastructure and I am trying to see if I can use CRIBL to query CrowdStrike via the API to pull this information down. Has anyone attempted or tried this? I was stuck with the client id and key with the REST API, but I think this needs to be done separately and only use an authentication token (bearer) to make the connection.

72
D
T
talantacpNew Participant
asked in General
AD FS Auditing Events Dropping from Windows Pack

The Microsoft Windows Events pack is currently dropping "SourceName=AD FS Auditing" events from the Security logs. I found the two lines that are not properly filtering the events.Pipeline: Windows Classic Events, "Final Cleanup" lines 27 & 28, (Serialize & Eval).I turned it off, but still working to get the events to not drop and convert to json.Q: Wondering if this filter is currently being updated/corrected?

01
J
D
Draco3New Participant
asked in General
CRIBL - Parser Libraries

Hi all, this is my first post on this forum, so hello. We have just begun to use CRIBL and I am still in a learning phase. I am wondering if there are more parsers that can be added to the library or whether you have to create them yourself? As an example I have a CRIBL instance that is receiving json data from another CRIBL instance (with the data having been prebaked SPLUNK) for a Cisco ASA log. The content of the log entry is in either the raw or Message field and I would like to extract into key value pairs. I have tried the some functions but it looks like I have to do custom regex (and I am not good with regex).

22
K
T
talantacpNew Participant
asked in General
Certificates and Enabling TLS

Can someone assist in explaining the difference between setting the Cribl General TLS settings versus the distributed TLS settings and what server certificates need to be applied? Where do we need to put the server.pem certs versus the web certs? Currently I am using mutual authentication and each host has their own self-signed certificates.

132
T
A
amillerInspiring
asked in General
Rest Collector with/without discovery results inconsistent

I am trying to pull event data from a remote location with http rest collector.It requires 2 stages to pull the data.Discovery will pull an array of job id's (and details).Collection will pull the results of each job from the array.When I set the collector up for discovery, I get the correct information including the ID.Using this id variable in the collection url I only get 1 event per id. If I hard code this id in the url AND disable collection, I get all events for that job.The throughput looks ok.But the capture window only captured 2 events. 1 from each job id.

21
A
G
Govardhanen GopalParticipating Frequently
asked in General
General Peristent Queue Questions

Hello,I have a few general questions on Persistent Queues(PQ). I see PQ settings for Source & Destination, and further my question are specific to size & compression. 1. If I enable PQ and let the default size to be 1 MB & with compression turned on and if I get a burst of 100MB of events which Cribl Stream is unable to handle, does it create a 100 compressed files of 1 MB each? I understand my events will get delayed in getting indexed in Splunk, but will this guarantee that I eventually will get these events(Assuming I have PQ enabled on the destination side too).2. Is compression given as an option just so to mitigate disk utilization? 3. When I turn on PQ, I do see a few ndjson.*.gz files in the queues directory, but these are from about 20-60 minutes ago when I first enabled PQ, when i do a cursory check of the data, I am seeing all the events in Splunk. Does this mean the Cribl is busy and eventually will get through processing these gz files may be in a few ho

21
B
0
0xbaemNew Participant
asked in General
Pulsar Connector

Hi there,Is a Pulsar connector for input/output analog Kafka planned?This would be a important addition to cribl stream, as pulsar is a fast growing platform and (for me) a successor to kafka.Best regards,

11
J
A
astvinceNew Participant
asked in General
Cribl Authentication with Okta and Agentless Desktop SSO

I am trying to setup SSO for our Cribl environments using Okta. I have successfully completed this within our DEV environment (which does not use agentless SSO). When testing the SSO setup in Prod, the login process gets stuck at the agentless desktop SSO step and will not progress past it.Does anyone have any ideas/recommendations as to what can be changed within Cribl, Okta, or somewhere else, that may help resolve this problem? We have not seen this issue with any other application at our enterprise except for Cribl.Thank you,Andrew

22
D
L
Lukáš MečířNew Participant
asked in General
Problem with deploy config from Leader to Workers

Hi, I have problem with deploy config from Leader to Worker. I have distributed deployment with 2 Workers and 1 worker group (testing, free license used). I configured secure connection between Leader and Workers according to https://docs.cribl.io/stream/securing-communications 1. For the first look everything is OK - both Workers are visible on Manage Worker Nodes on Leader as alive, I can connect to Workers GUI (Remote UI Access is on), config version is OK etc.Problem is when I want to deploy config changes to Workers. On Manage Worker Nodes I see yellow exclamation mark and neverending rotating circle in Config version column and config changes are not propagated to Workers.In log on Leader I see this:{"time":"2022-06-17T13:01:21.350Z","cid":"api","channel":"CriblMaster","level":"info","message":"sending config update request","group":"default","version":"8627dcf","logStreamEnv":"master","worker":"ff5ea4c4-51eb-4aa4-a650-2e6c8251bb21"}{"time":"2022-06-17T13:01:21.363Z","cid":"api",

32
L
D
dmtrsNew Participant
asked in General
POST to URL using Javascript

Is there a way to set up a Destination as a POST request (w/authentication)? I tried using Code and modifying the code for DNS lookup on this page (https://cribl.io/blog/extending-cribl-building-custom-functions/). I got errors about using Promise.

20
J
S
Shane DanielsEmployee
asked in General
Can I use self-signed certificate with a REST collector?

Can I use self-signed certificate with a REST collector? Does Cribl perform any TLS certificate validity checks?

11
J
J
jccurtisNew Participant
asked in General
SNMP Trap Forwarding Question

If I need to accept an SNMP trap from a source, and send it to multiple trap destinations, will Cribl spoof the source IP and send the trap “un-altered” to the final destinations? Meaning will it preserve the source IP address from the original trap IP header?

11
S
S
Shane DanielsEmployee
asked in General
Changing from single instance to distributed setup - high availability

I’ve made the switch from a single instance to a distributed setup (1 leader and 2 worker nodes). My old server, which receives logs from a variety of sources, is now acting as a worker. What happens if this worker goes down, because all of the sources are sending logs to one single worker IP? What should I do to guarantee appropriate high availability?

42
S
E
ekalbyNew Participant
asked in General
Regex to Grab Key:Value

Not a question but rather something useful. I have data that is in key:value format, using key=value in the extract format did not work. I was able to extract every key and value with this one regex extract.(?<_NAME_0>.*):\s(?<_VALUE_0>.*)

22
P
S
SonOfBuziInspiring
asked in General
Technical question on stream file monitor

i have a use case where we have a nfs share that applications writes logs too i would like to monitor these while being written to (file monitor?) but distribute the load across workers. is that possible?

02
B
F
fshelton79New Participant
asked in General
Odd error message

What does this message indicate on the Splunk TCP source:message:Failed to parse s2s payload{}error:αmessage:Cannot advance offset 2023514992 past end of event 74αstack:Error: Cannot advance offset 2023514992 past end of event 74at p.advance (/opt/cribl/bin/cribl.js:14:12271888)at h._readEvents (/opt/cribl/bin/cribl.js:14:12274323)at h._transform (/opt/cribl/bin/cribl.js:14:12275791)at h.Transform._read (_stream_transform.js:205:10)at h.Transform._write (_stream_transform.js:193:12)at writeOrBuffer (_stream_writable.js:352:12)at h.Writable.write (_stream_writable.js:303:10)at Socket.ondata (_stream_readable.js:719:22)at Socket.emit (events.js:315:20)at Socket.EventEmitter.emit (domain.js:486:12)Show less

51
B
M
mikeyleeNew Participant
asked in General
Problem with Worker connecting to Leader node

Hi Guys,I have setup a test environment on my laptop using VBOX and with a Centos7 edge leader node and worker using ports 9000(ui) and 4200 for internal comms.I’ve installed the worker node using curl. Seems there is an issue with the worker node connecting to the leader as shown below. From the worker node, telnet to the leader nodes port 9000 and 4200 connect fine. But not sure what additional stuff to check based on the below messages:{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"info","message":"attempting to connect","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"will retry to connect","nextConnectTime":1653267018346}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"connecting","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.264Z","cid":"api","channel":"input:D

504
M
G
Govardhanen GopalParticipating Frequently
asked in General
Fundamental question on git deployment in distributed setup

I am new to the distributed world, so please bear with me. I have a distributed setup with 2 worker groups(default & 2ndgroup). I have assigned nodes to each of the above 2 groups. In the leader ui when I navigate into the "2ndgroup" and make changes to my pipelines and commit and distribute the changes, it gets pushed to my git repo(remote) & the worker nodes.When I further navigate into the worker nodes and do a live capture of events and make changes to the pipelines, how do I push these changes to the git remote and further when I view it in the leader node under group view reflect the changes that were done at the node level? May be I am missing something obvious, can somebody shed some light on what I am not getting right in this flow?

20
B
S
Shawn CannonKnown Participant
asked in General
Python Collector Script issue

We have a collector script (python script) that can take up to 10 min to pull down all the data (the script will cache it) and then “print” it out so that Cribl can then ingest it. Ends up being equivalent to a 650MB CSV file. The first ad hoc full run seemed to have timed out even though I had 0 set for job timeout. Debug shows a lot of “failed to pop task” messages while it is running. Any ideas?

113
B
S
Shawn CannonKnown Participant
asked in General
How to use first line in CSV as the headers for Parser function

So I have a script that will collect data from an API. The data is returns is CSV formatted, the first line will be the list of fields. How do you tell Stream to use that first line as the headers?

32
S

Badge winners

Show all badges
Terms & ConditionsCookie settings