Stream

Hi, for most of my SaaS services, I have to use the REST collector to pull events periodically, as they do not support sending their events directly into Cribl. I wonder what the recommended strategy is to ensure that all events are getting collected. What schedule is common to run the REST collector on? every few hours or rather minutes?Most APIs support giving a "from" timestamp parameter - but what variable can I use within Cribl to reference "the last run time of a job" ?In general: Is it better to give the API precise instructions beforehand, so it will not return events already collected, or is it better to just collect duplicate events and then drop them in Cribl (I did it with a Suppression rule, but the thresholds need to be very long)Thanks a lot.

Why do we see this message when using Cribl v4.4.0 with S2S V4 and Splunk forwarder version 9.1.x:{"time": "2023-12-19T15:16:15.549Z","cid": "w0","channel": "input:in_splunk_tcp","level": "error","message": "closed connection","src": "10.10.10.10:62453","error": {"message": "Unsupported op-code 249.","stack": "Error: Unsupported op-code 249.\n at g._processData (/opt/cribl/bin/cribl.js:15:15721335)\n at g._transform (/opt/cribl/bin/cribl.js:15:15717080)\n at Transform._write (node:internal/streams/transform:175:8)\n at writeOrBuffer (node:internal/streams/writable:392:12)\n at _write (node:internal/streams/writable:333:10)\n at Writable.write (node:internal/streams/writable:337:10)\n at TLSSocket.ondata (node:internal/streams/readable:766:22)\n at TLSSocket.emit (node:events:513:28)\n at TLSSocket.emit (node:domain:489:12)\n at TLSSocket.<anonymous> (/opt/cribl/bin/cribl.js:15:12649570)\n at addChunk (node:internal/streams/readable:324:12)\n at readableAddChunk (node:internal/str

Posting this to get community feedback and track our progress in case anyone encounters this issue in the future.We are attempting to use the data cloning functionality on our F5 load balancer to duplicate data to our Cribl workers. We want to do this so that we can route, filter, and reduce production data in Cribl without impacting production operations. When the time comes, Cribl will be ready for the cutover without much impact at all, hopefully.Right now, the F5 is sending data to our syslog servers, which then send to Splunk. The end goal is to replace syslog with Cribl, but we want to be able to see all the data in Cribl before then so that we can operate on it. We are successfully cloning data that comes from Splunk UFs and HFs, but the cloned syslog data from the F5 is not being seen by the Cribl worker.The specific problem is this: The data is making it to the to the correct interface and port on the worker, but is not being picked up by Cribl for whatever reason.I have verif

Hello, has anyone encountered this error? i'm trying to forward my firewall logs to Cribl.I added Syslog Push via QuickConnect. Address: is public IP address of the firewallUDP port: 1514On my firewall Settings:Syslog Server Address: is the Ingress IP of Cribl CloudPort: 1514Are these the correct settings?

channel: func:dns_lookupcid: w11cribl_postproc_pipe_ts: 1702407784.731hits: 1399991keys: 3ksize: 50level: debugmessage: Cache statsmisses: 9time: 2023-12-12T19:03:04.731Zvsize: 240

What is the recommended method of restricting groupings of clients/servers to accessing event collector subscriptions, absent support for computer group control that WEC has?I thought about creating additional listeners and via GPO pointing particular groupings of clients at particular listeners. Is there a better alternative?

Would it be possible to confirm:What does Cribl do with the events when this issue occursWhat is the maximum character limit CRIBL can parse

Why does bootstrap command have HTML character codes?

Why do I see some worker processes having a higher usage than others for Kafka sources or destinations?

I am seeing this error when trying to access the worker group configuration from the UI:“The Config Helper service is not available because a configuration file doesn’t exist or the settings are invalid. Please fix it and restart Cribl server”:

In red below, the license still shows an old email address. This person is no longer part of our company and we want to replace the email address:

Hi all, i'm currently trying to configure Microsoft Sentinel as my Destination.I'm stuck configuring the Authentication. I don't really know what the Client ID is and the hover text "JavaScript expression used to compute the application (client) ID for the Azure application. Can be a constant." isn't very helpful. I have the Application ID as a mix of letters and numbers, but I can't insert it in this form? Can somebody help? Thanks :)

Hi Mates, I'm using cribl S3 collector to collect the logs from AWS S3 bucket. S3 bucket contains logs of Akamai datastream which has the log format as below.eg: s3://BUCKETNAME/APPNAME/ENV/ak-913478-1701139960-008071-ds.gzak- akamai file prefix913478 - random string170xxxxxxxx - EPOCH timestamp008071 - random stringds- file suffixWhen I schedule the S3 collector to run every 15 minutes, it collects all the log files from buckets every time. Looking for a suggestion, how do I collect only the new files from S3 not all the files every time.

Why an engine id is not included into a configuration for decrypting SNMP traps?

How do i calculate the amount of workers i need if i am using AWS Fargate arm processors? I am looking at 7tb/day input and output to 2 destination which is a total of 21tb/day throughput.Thanks

I'm quite new to Cribl, so please forgive me if I've got some easy and/or dumb questions!The first thing the Palo Alto Pack pipelines all do is extract the host from an RFC3164 formatted syslog message with a BSD time format.Eval: host = _raw.match(/[A-Z][a-z]{2}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2}\s([^\s]+)\s/)[1] || hostOurs are sending in RFC5424 w/ Octet count framing (not syslog'y enough to know if that's RFC6587 or not), so the host extraction fails and ends up sending to splunk as the IP.With the desire to use the Palo Pack, what is the best way to handle this?Right now I created a pre-processing pipeline and attached it to my source. All it does is extract the host from the header. That way I don't have to handle it in every pipeline inside in the pack.Also, is there a reason you cannot connect a pre-processing pipeline to a source that's inside a pack? In this case, it would be nice to be able to package my paloalto_syslog_pre-processing pipeline inside our customized PaloAlto

Can cribl stream carry out ldap queries to Microsoft AD and then save the AD information in a lookup table or something similar? I want to automate fetching AD information from AD servers and then feeding it to a SIEM like Splunk or Elastic.

Just upgraded a Stream cluster with 4.03. When performing the Commit to update the workers got "block timeout reached" message. Can be related to the amount of changes (195)? Is there any settings to change the timeout limit for updates?

I used a standalone test box to get my REST collector working like I wanted.  authentication, discover and collect worked great.  When I moved it into my distributed environment, it errors out but does pull in some events.  The discovery and collect phases are distrubuted across my 5 workers.  It seems like the authentication token may not be working properly when it runs on more than one worker?  I am guessing of course.  I get errors like 503 Service Temporaily Unavailable and 500 Internal Server Error.  This does not happen when I run it from my single standalone test server.  I may need some assistance trying to figure this one out.

I am getting this error in stream when trying to configure an IP address for a TCP source. I can't find any docs around this error.

For other users that have started pulling O365 activity logs using Stream instead of the Splunk app, have you noticed a discrepancy in the event counts? We consistently receive more events through the Splunk app than we do through the Stream source. The setup is exactly the same: tenant, app Id, content, and interval. We've even done comparisons into the data and have found events missing through Stream that were retrieved by the Splunk app. So it seems like Stream is either dropping some events or just not pulling all of them.We do have a case open with Cribl already but I'm wondering if we're the only ones seeing this. Has anyone else noticed compared the volume and noticed the same?

Hi All,I am new to Cribl and trying to set notification via webhook for destination have back pressure or destination is unhealthy. But i am getting below error:"the notifications service is not available because a configuration file doesn't exist or the settings are invalid."

We installed the Cribl on our Linux server, and by default, it is user=cribl group=cribl.Our organization requires us to us a Service account to run configuration inside it.Question :Can we change the user and group ownership of the Linux directory of Cribl?And would it affect anything ?From user=cribl group=cribl.To user=svcacc_cribl group=svcacc_cribl

I am currently setting an environmental variable in a Script collector using a JS Expression as follows: TEST_VAR = C.Secret("test1").valueThis works fine for some secrets, but not for others. In the latter case, TEST_VAR is equal to 'C.Secret("test1").value' instead of the actual value.I get the impression that the secrets for which it does not work have not yet been distributed to the workers in the group. Is this correct? Or is there something else wrong here.

With Cribl Stream what is the best practice for receiving windows logs?