Community Discussions

TL;DR Use VSCode to test out javascript "code" function locally before attempting it in Cribl Stream/Edge.I wanted to first give a shout out to @Jon Rust for pointing me in this direction. Paying it forward to other Cribl users who would like to give the code function a shot.I am using the example from the following pagehttps://docs.cribl.io/stream/usecase-code-function/#building-a-new-arrayI had a pipeline with about 40 functions which was used to take a bunch of metrics together and publish it as a multi-metric into splunk. I wrote this with David Maislins help about 3-4 years ago, and it was getting to be a bear to maintain and upate. The code function was the perfect candidate for this. I took 36 functions and consolidated into 1 code function which was much easier to debug. Please note I have NO experience with JS and this is just something I came up over 2 days

Hello, I would like to ask one thing: is it possible to use some database (SQL etc.) as a data source for lookup? One of our customers has a database that contains data that he would like to use to enrich events processed on Cribl Stream. The idea is that he would read the data from the database into Cribl Stream using a suitable connector, save it as a CSV file and then use that as the source for a lookup to use in the pipeline. Since I don't have experience with something like this yet, I would like to ask: Is there anyone who has experience with this? Is this a feasible way? And if so, is it suitable or are there better ones? What is there to be careful about performance, load on Cribl and the database, etc.? (Enrichment would involve about 200 events per second, the database is in the lower tens of thousands of rows, and the data in the lookup should be refreshed once per hour.) Any experience, ideas or recommendations would be greatly appreci

Hey All,Trying to setup TCP source to onboard our vendor saas logs in to our onprem splunk. With Cribl can we do IP whitelisting for the TCP source and allow only certain IPs instead of opening it up to public. My setup is onprem so want to see if there are possibilities available in the cribl side.

Hi,I'm trying to setup a Splunk HEC within Cribl Stream, and I'm encountering the error "malformed HEC event." I've encountered that error when setting up HECs in Splunk, and to correct the problem I have to go to Source type > Select Source Type > Structured > _json when editing the HEC's settings. I've been looking and reading search results, but so far I haven't found the equivalent setting in Cribl Stream. Where can I find that setting for the HEC please?

I’m attempting to leverage the REST collector to collect logs from Duo. The REST API requires authentication to be in the form of an HMAC-SHA1 signature in hex ASCII format. Instructions can be found here: Duo Admin API | Duo Security.At the very least, I’m curious if this is possible, and if so, how would one go about setting this up? Thanks!

I'm new to Cribl, so at this point we have single-instance Cribl Stream installed on premise. We just got a new SE, and he stated I need to move the Leader Node to the cloud as it's easier for him to troubleshoot. I have a cloud account created.I presume this is the distributed deployment, but the menus described in the documentation don't seem to align with what I see in the UI. https://docs.cribl.io/stream/deploy-distributed/ I'm not confident I have the correct software. Can Stream be converted into a worker node? Do I need the distributed deployment? Can anyone point me to a video that describes the process? I've looked on YouTube, but I've not had any luck.

Is there a way to move a worker to another group via GUI? And when not, why not?!

We are planning to enable gitops workflow for production and from the docs we understood that the production environment will be “read only” if the gitops workflow is set to “push”. What if there is outage with “git” and we can’t do the release? What is the alternative to push the code without git?

How I can Parse Array of array in cribl stream. Each array element contains multiple strings.

Hi,Excuse me for my "not knowing", maybe a easy answer for all of you.I am in the process of migrating our Heavy Forwaders to Cribl . One off the inputs is for Cisco IOS devices. The props.conf has a Regex in it to change the sourcetype with help of a transforms. The props uses a regex to select the events.Where and how do i put the regex which is used in the props.conf for filtering the events. Do i have to place this regex in the "filter expresion" in the route? Or in a pre-pipeline? And how do i do it?Thanks in advancegreetingsJari

I am trying to forward an exact duplicate of the syslog message. I have the route going thru passthru to not change the data. But I am getting the timestamp and original message host added to the message. Thoughts?Source: Syslog on 514/tcp/udp.Pipeline: passthruSource Data:<164>:Jul 23 12:34:15 CDT: %ASA-auth-4-987654: Group = 1234567890, Username = efghijk, IP = aaa.bbb.ccc.ddd, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:29s, Bytes xmt: 0, Bytes rcv: 0, Reason: Idle TimeoutDestination Data:<164>Jul 23 12:34:15 lll.mmm.nnn.ppp :Jul 23 12:34:15 CDT: %ASA-auth-4-987654: Group = 1234567890, Username = efghijk, IP = aaa.bbb.ccc.ddd, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:30m:29s, Bytes xmt: 0, Bytes rcv: 0, Reason: Idle Timeout

The cribl service should not be run as root, some users have run into file ownership problems if they start the service as root and subsequently chown the cribl home directory as described in the docs. If the files in $CRIBL_HOME are owned by user cribl and the service still won't start, check that the files in /tmp/cribl_stream are also owned by the cribl user.If not, you can use this command to change the owner. You may need to use sudo to do this if your user does not have permission.chown -R cribl /tmp/cribl_stream

I have an application which is running on a Virtual Machine & this app is responsible for generating syslog messages.My goal is to forward these syslog messages to a source configured on Cribl Cloud.I went through the documentation, however I am not able to understand that how can I forward these messages.I know the idea is to send the messages to some IP address and Port, but where do I find this IP and port number?Also, while configuring the source on cribl, what is the purpose of IP and Port there? Do I need to point it to my Virtual Machine?

You know how as you get better at Cribl Pipelines, you refactor as you learn better ways to do stuff? What's the coolest thing you've figured out how to do? Include a screenshot if you can.

Hi, I have an API endpoint which require parameter to use a time string in the format of %Y-%m-%d %H:%M, which e.g. look something like this: https://host/api/v1.0/endpoint?start=2024-06-21 14:00&end=2024-06-21 15:00However when I put this on the URL, it always get converted to a value of 2024-06-21%2014:00 using example above. The API endpoint is unable to accept %20 and always ended up throwing error.Is there a way to avoid space being converted in Discover/Collect URI in REST API collector?

I'm getting 400 responses on a POST Body that I'm trying to pass through the state.latestTime within the POST body. I would like to know what is being posted.I don't seem to be able to get a debug log out of my REST Collector to see what it is POSTing. What it's deriving from state.latestTime.Anyone know where I can find this? The debug logs seem to only show the URL, not the POST Body

I have a REST API Collector which look at retrieving a number of records containing an ID which then need to be fed into a collection task as input for a 2-stage API call.While the Discover task has successfully retrieved the records with the ID, and I can see that in debug log the collection task has pick up the ID and insert correctly into Collect URI using variable, the tasks simply get cancelled afterwards.Does anyone have experience resolving similar issue before or have some idea what this might have gone wrong?

Hi, have anyone successfully deployed Cribl on RHEL based host that comes with SELinux enabled?While I have managed to get Cribl manually run, I've hit into issues when trying to enable Cribl on boot as a service. Looking at the entries from "journal -xe", it's throwing up errors to do with SELinux denying the processs because of cribl.pid file unlinking permission.After running the suggested actions as interim fix for SELinux, systemd were able to launch Cribl process but only to see that it fail right after, with slightly different message about "protocol failure" which doesn't give much of a clue what went wrong.

I'd like to search a CloudTrail S3 bucket. What I want to search is "in all accounts" and only in "US-EAST-2" region for event name:"Assume Role". I'm not sure if I configured the Dataset correctly and/or how to do the search. Regarding the Dataset bucket path. This is the S3 folder layout: AWSLogs/subfolder/<ACCOUNT>/CloudTrail/<REGION>/YYYY/MM/DDIn the Dataset config, i'm not sure what to put in the bucket path. Especially for the <Account> and <Region> parts My guess is something like this AWSLogs/subfolder/${ACCOUNT}/CloudTrail/${REGION}/${_time:%Y}/${_time:%m}/${_time:%d}Regarding search: What I want to search is "in all accounts" and only in "US-EAST-2" region for event name:"Assume Role" When I did: dataset="CloudTrail_DATASET_NAME" | limit 1000 it returns no results. Not sure what I'm missingThanks

Hi, I am currently exploring a way to collect event data and looking to save it into a temporary staging storage (GCS Bucket) before transferring the data downstream for further processing.I am having difficulty to get a working configuration on Stream no matter what I tried. Stream worker is always throwing the following error: "failed to close file" where the detailed message is saying "Invalid argument." without much details except that the file that Stream worker is having issue to close is at this location: /opt/cribl/state/outputs/staging/prefix/Bucket_yyyymmdd_hhmm/yyyy/mm/dd/CriblOut-<random>.0.json.tmpI have checked that the credentials for accessing the bucket is valid and the permissions for the given credentials are alright for upload files into the bucket.Does anyone have any idea how I might be able to get around this problem?Thanks.

Hello,Here I have a small picture of how the environment is structured:Red arrow -> Source Splunk TCP (Cribl Stream) I'm trying to forward the journald data from the Splunk Universal Forwarder to the Cribl Worker (Black to blue box).I have configured the forwarding of the journald data using the instructions from Splunk.(Get data with the Journald input - Splunk Documentation) I can forward the journald data and it also arrives at the cribl worker.Problem: the cribl worker cannot distinguish the individual events from the journald data or does not know when a single event is over and thus combines several individual events into one la

I recently set up our SQS amazon queue in cribl. Events are forwarding to splunk, however when compared to the pre existing aws logs in Splunk from the Heavyforwarder TA, I noticed we are only pulling in events from only one region via cribl oppose to the 20 actually sending events and being received through the Splunk TA. Any advice on how to troubleshoot an solve this issue?

Hi,I have a very basic setup which has: Cribl stream leader node, 2 cribl workers, 1 admin nodesource: syslog tcp port 1514destination:filesystemroute filter:__inputId.startsWith('syslog:')route pipeline: mainroute output: filesystempipeline: main and CriblSyslog routes attachedI have a test machine that has it's rsyslog sending to remote location (a worker node) and also a script that randomly generates syslog data and sends to the same worker node every 1 second.I can see the data come in using tcpdump, I can see the received messages coming into the worker in the Sources → Syslog → Status menu but nothing is being processed. Outside of the status menu, none of the metrics show the incoming messages and no files are being written to the filesystem. I've confirmed the connection between the test host and worker node is open with telnet and netcat.Tried tailing Cribl log files, system log files, anyth

Which Azure Sentinel tables does Cribl Stream natively support? And does it also support sending data to custom tables?

Hi All, Posted this in the Slack channel but no bites.We are trying to filter out data from an WAF log to decrease the log size ingested. I created a parser in the existing WAF pipeline using the “Extract” Operation Mode but I cant seem to get it to function correctly.  I was told by support the "Mask Function" would work best, they said "Within that function you can specify the regex and replace it with "". " Not sure how to go about that route. I'm essentially trying to drop data in the log using the following regex but like i said it doesnt seem to function when I test with sample log. The regex is the following -> nContent-Security-Policy-Report-Only.*?DFCspReportFunction\+blob:.*