Community Discussions

Why do I see some worker processes having a higher usage than others for Kafka sources or destinations?

I am seeing this error when trying to access the worker group configuration from the UI:“The Config Helper service is not available because a configuration file doesn’t exist or the settings are invalid. Please fix it and restart Cribl server”:

In red below, the license still shows an old email address. This person is no longer part of our company and we want to replace the email address:

Hi all, i'm currently trying to configure Microsoft Sentinel as my Destination.I'm stuck configuring the Authentication. I don't really know what the Client ID is and the hover text "JavaScript expression used to compute the application (client) ID for the Azure application. Can be a constant." isn't very helpful. I have the Application ID as a mix of letters and numbers, but I can't insert it in this form? Can somebody help? Thanks :)

Hey allI am using a REST API to collect logs using GCP OAuth token. As the authorization token expires every hour, is there any way I can automate refreshing the access tokens in Cribl

Hi Mates, I'm using cribl S3 collector to collect the logs from AWS S3 bucket. S3 bucket contains logs of Akamai datastream which has the log format as below.eg: s3://BUCKETNAME/APPNAME/ENV/ak-913478-1701139960-008071-ds.gzak- akamai file prefix913478 - random string170xxxxxxxx - EPOCH timestamp008071 - random stringds- file suffixWhen I schedule the S3 collector to run every 15 minutes, it collects all the log files from buckets every time. Looking for a suggestion, how do I collect only the new files from S3 not all the files every time.

Hi team, I need help with File Monitor source strange behaviour. Configured FM source (Cribl Stream single instance version 4.0.3) for one log file containing logs from shell backup script running once every day at 7PM. Backup script produces 6 lines/events during its operation. No advanced option used here, manual discovery with allowlist specifying log file name and Collect from end checked. Quick connect through simple pipeline adding metadata (index, sourcetype) and sending events to Splunk single instance destination. This setup works fine most of the time, 6 events read and sent to Splunk every day correctly. But from time to time it happens that at scheduled time not complete content of the log file is read and sent. The remaining lines from one day are then sent and ingested to Splunk together with the events from next run. It happens irregularly. I can not find any single reason for this, any idea? Original message in Cribl Community Slack Message:https://cribl-co

Why an engine id is not included into a configuration for decrypting SNMP traps?

I deployed Cribl on some Windows Servers. They are routing events to an on-premise Cribl Stream instance. Everything was working fine, until collection just stopped. It appears that the Edge agents aren't connecting to "https://cdn.cribl.io/telemetry". These servers are behind a web proxy. The proxy settings are applied properly in Windows.If I pull up a browser on the server, and surf to https://cdn.cribl.io/telemetry, I get the expected output --- “cribl /// living the stream!”I have tried:Manually configuring the proxy settings in Windows itself rather than auto-config.Defining actual environment variables for “HTTP_PROXY” and “HTTPS_PROXY” within Windows Server.Tried flipping the Cribl Edge “default_fleet” toggle

We are following this document, https://docs.cribl.io/stream/usecase-rest-ms-graph/ We have tested bringing in Azure User and Device data in the past with no issue. We are having a problem figuring out the best Collect URL to use. We've tried many different ones. We've tested our URLs to Microsoft's Graph Explorer and we are not receiving the expected output in Cribl. We have also set all the permissions that a Splunk App would require, https://docs.splunk.com/Documentation/AddOns/released/MSO365/ConfigureappinAzureAD

How would maintenance impact data processing, etc. (i.e. for scale-up procedure)?AMIs must be upgraded every X months; how does that impact the application (Stream)?Have you seen any auto-scaling ? How does scaling down impact the application? Will data be lost or will it complete its processing before shutting down?Concerned about what happens to data in flow when instances are downLooking for details on public cloud use casesFor the infrastructure scaling sideAfter disaster recovery, how do they switch back to the original state?

How do i calculate the amount of workers i need if i am using AWS Fargate arm processors? I am looking at 7tb/day input and output to 2 destination which is a total of 21tb/day throughput.Thanks

I'm quite new to Cribl, so please forgive me if I've got some easy and/or dumb questions!The first thing the Palo Alto Pack pipelines all do is extract the host from an RFC3164 formatted syslog message with a BSD time format.Eval: host = _raw.match(/[A-Z][a-z]{2}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2}\s([^\s]+)\s/)[1] || hostOurs are sending in RFC5424 w/ Octet count framing (not syslog'y enough to know if that's RFC6587 or not), so the host extraction fails and ends up sending to splunk as the IP.With the desire to use the Palo Pack, what is the best way to handle this?Right now I created a pre-processing pipeline and attached it to my source. All it does is extract the host from the header. That way I don't have to handle it in every pipeline inside in the pack.Also, is there a reason you cannot connect a pre-processing pipeline to a source that's inside a pack? In this case, it would be nice to be able to package my

Dear all,Coming up with yet another question as I couldn't find similar from the history of available requests. I have created the HTTP RAW Source and able to push the data and process it in the pipeline. However, I have observed that when I POST the valid JSON format(all the attributes with double quotes), Cribl http source is parsing them into individual fields after the source process, and if the posted content is not JSON parsable, then the content is available under the key "_raw".As per my requirement, I would like to keep the input JSON object as is, since I was trying to validate the whole json with the schema stored in the Knowledge. If it's not getting parsed then I was able to run my use case as expected. In other case where the JSON is parsed into individual fields, I couldn't find any internal key which holds all these variable to format them back to JSON object to validate against the schema.Hence, reaching out for any sugge

I'm attempting to extract values from a JSON string field. However, it seems, that none of the below methods are working for referencing or obtaining the value using a JSON path or dot notation type of approach. I've even tried the "extract_json" function, but to no avail.KQL in Azure has the bag_unpack function, but I noticed this is not supported in Cribl Search.Below are my queries and results.| extend json=parse_json(_raw)| extend test1=_raw._raw.kubernetes.container_name| extend test2=json._raw.kubernetes.container_name| extend test3=json['_raw']['kubernetes']['container_name']| extend test4=extract_json("$._raw.kubernetes.container_name", json, typeof(string))| limit 10

Hello all,I have recently started working with Cribl and my question may seem to be easier one, however, I couldn’t find an option to do this in right way and hence reaching out for help.In one of my use case I am getting a json from source and my requirement is to check whether the attributes in the object are matching with some of the existing Regexes in Crible knowledge section. I was expecting something like below,For ex: To access Schema, we use C.Schema(‘Schema Name’).Validate()In the same way can we do C.Regexes(‘Name of the regex’).test()Please advise on this with any suggestions. Thanks in advance!

Is it possible for a member company to purchase a single license for the entire business group and share the license with the rest of the companies?

Can cribl stream carry out ldap queries to Microsoft AD and then save the AD information in a lookup table or something similar? I want to automate fetching AD information from AD servers and then feeding it to a SIEM like Splunk or Elastic.

Has anyone experienced system metrics crippling the 4200 port? What are some best practices or documentation to combat this and just setting configurations for system metrics in general?

Just upgraded a Stream cluster with 4.03. When performing the Commit to update the workers got "block timeout reached" message. Can be related to the amount of changes (195)? Is there any settings to change the timeout limit for updates?

What's the best way to reference a Worker Group Level Lookup File inside a pack? Would it be: $CRIBL_HOME/groups/<groupname>/data/lookups/LookupFileName.csv ?

I used a standalone test box to get my REST collector working like I wanted.  authentication, discover and collect worked great.  When I moved it into my distributed environment, it errors out but does pull in some events.  The discovery and collect phases are distrubuted across my 5 workers.  It seems like the authentication token may not be working properly when it runs on more than one worker?  I am guessing of course.  I get errors like 503 Service Temporaily Unavailable and 500 Internal Server Error.  This does not happen when I run it from my single standalone test server.  I may need some assistance trying to figure this one out.

I am getting this error in stream when trying to configure an IP address for a TCP source. I can't find any docs around this error.

I am running Edge Nodes on our EC2 instances. When reviewing internal fields from our File Collector sources, I noticed that there is alot of really useful data that our SREs use to investigate which regions & servers are experiencing issues during incidents. These include: - AMI image - Region - Availability Zone - Architecture - Public IP - Private IPI'd like to extract these internal fields as event fields that get sent to our destinations. What is the simplest way to do that?

For other users that have started pulling O365 activity logs using Stream instead of the Splunk app, have you noticed a discrepancy in the event counts? We consistently receive more events through the Splunk app than we do through the Stream source. The setup is exactly the same: tenant, app Id, content, and interval. We've even done comparisons into the data and have found events missing through Stream that were retrieved by the Splunk app. So it seems like Stream is either dropping some events or just not pulling all of them.We do have a case open with Cribl already but I'm wondering if we're the only ones seeing this. Has anyone else noticed compared the volume and noticed the same?