Community Discussions

Attention all data lovers! Cribl is thrilled to announce the latest and greatest version of platform, Cribl v4.1! This update is so hot, it’ll make your data sizzle!So, what’s new you ask? There is a lot. Here are the first few items that we are talking about for each product. There are too many new features to talk about in an announcement. Stream (release notes):Persistent Queues Predictability: When you configure Persistent Queues on Sources, Always on is the new default mode for greater reliability.Connections and Load Throttling: Along with the new PQ options, we've provided new controls to prevent connections from overloading Workers. At Group Settings > Worker Processes, you can now limit how many connections each Worker will accept upon startup or restart. You can also determine how long after startup to impose that limit

This post will help you setup a trust between your Cribl Cloud instance and your AWS account. It follows the steps documented on the docs page for Cross-Account data collection.This Rapid Adoption CloudFormation template can help automate the trust between your Cribl Cloud stack and your AWS Account. In order to keep the IAM policy limited to only resources that Cribl Cloud should have access to, the template will ask for specific S3 buckets and related SQS's.

We had an outage over the January 21st weekend. Please let Tony Reinke (Slack - Tony Reinke (Cribl) or Email - treinke@cribl.io) know if you are still running in to any issues. Thanks.

How do I dynamically create multi value fields in Cribl?For example, I would like to take the following and send it as a multi value event in Splunk: cs19=00:40:03:05:79:e1;00:40:03:05:79:d1;00:40:03:05:7a:25;00:40:03:05:7a:89;00:40:03:05:79:f1;00:40:03:05:7a:4d;00:40:03:05:79:cd;00:40:03:05:79:dd;00:40:03:05:7a:51;00:40:03:05:79:fd;00:40:03:05:78:c9;00:40:03:05:78:f1;00:40:03:05:79:0d;00:40:03:05:79:2d;00:0a:f7:fb:77:58;64:00:6a:7c:65:48;64:00:6a:7c:6d:26;64:00:6a:7c:66:d5;64:00:6a:7c:5f:66;14:18:77:6b:31:01;00:40:03:05:78:cd;64:00:6a:7c:65:b4;00:40:03:05:79:05;64:00:6a:7c:66:f2;00:40:03:05:7a:91;00:40:03:05:7a:69;00:40:03:05:79:75;00:40:03:05:79:25;00:40:03:05:7a:09;00:40:03:05:79:35;64:00:6a:7c:6d:ce;64:00:6a:7c:6b:79;14:18:77:6b:63:81For example, this sample event can come in with anywhere from 1 to 50 different MAC addresses.

When I run an event through Cribl Stream on premises and then send to Cribl Cloud, I end up with duplicate cribl_pipe events. How can I remove the duplicate?

I want to use the API to export a Pack from one worker group and merge it to another. How do I do that?

When deploying Stream in containers local configuration changes will not persist. Is there a setting to fix this?

After being processed by Stream, is there a way have those source log files either be moved, renamed or deleted?

I have this really simple function enabled to configure current time as timestamp, but it doesn‚'t seem to work. it's still taking timestamp from event.

Is there a way to measure the size of a field and then only keep XX bytes of it? I have some huge events coming in from Corelight and it’s all from one field within the event.

When trying to attach ./scope to PID in Docker, I receive the error, you must have ptrace capabilities to attach to a process .

If Splunk Universal Forwarders are configured with compressed = true in the outputs.conf, can the Cribl Stream SplunkTCP source receive from them properly?

Is it possible to stream logs and events to Cribl Stream, send a copy of that event to S3 (or similar) for archive, and then process/parse/enrich the logs before sending them to another destination? The use cases I’m thinking of are as follows:(Primary Use Case) If a security incident occurs and we need to present evidence in court, how do we protect the forensic integrity of an event/log/etc. if the event is being changed in flight before going to an archive or analysis tool?If we needed to replay a series of logs for some reason or another, it might be helpful to do so from the source events, unmodified by Cribl.The downstream, long-term storage solution may also serve as a repository of data for other teams to use. Providing unmodified events for those teams may be preferred in that case.Thanks!

I am new to the distributed world, so please bear with me. I have a distributed setup with 2 worker groups(default & 2ndgroup). I have assigned nodes to each of the above 2 groups. In the leader ui when I navigate into the "2ndgroup" and make changes to my pipelines and commit and distribute the changes, it gets pushed to my git repo(remote) & the worker nodes.When I further navigate into the worker nodes and do a live capture of events and make changes to the pipelines, how do I push these changes to the git remote and further when I view it in the leader node under group view reflect the changes that were done at the node level? May be I am missing something obvious, can somebody shed some light on what I am not getting right in this flow?

Is there a way to set up a Destination as a POST request (w/authentication)? I tried using Code and modifying the code for DNS lookup on this page (https://cribl.io/blog/extending-cribl-building-custom-functions/). I got errors about using Promise.

I’m changing the ip on a couple of my workers - anything that I need to change or be aware of? My assumption is that Stream/Edge will just pickup the change and it will appear on the leader with those changes. Nothing else is changing.

Is there a URL or API endpoint I can use to check the status of a Stream Worker or Leader? If so, what HTTP Status codes does the endpoint return?

Hi all. I have Cribl replacing syslog-ng, but in order to listen on 514 I needed to run it as root. Is there a work-around to drop privileges or switch to user cribl after start-up?

Is there a way that we can search data in Devo using the API?

I am getting the following error in Splunk when sending an event from Stream:04-20-2023 04:21:09.000 +0000 INFO IndexWriter [7972 indexerPipe] - idx=goat: found event timestamped=1681964469123 is out of range. Using 1680307200.Here’s what _time looks like, in Stream, when viewing the data:{“_raw”: “”,“_time”: 1681964469123,“cribl_breaker”: “”,“index”: “”,“source”: “”,“sourcetype”: “”,“cribl_pipe”: “”}

I have a REST collector discovering events but not collecting them when scheduled. It works Ad-hoc, though.Any idea what could be causing this? I do not see errors in the logs.

What's the best way to move a sample data capture into a pack? Seems that exporting it isn't consistent.

This discussion has been moved.

I'm here to give you the 411 on new features of v.4.1.1 with updates to Stream, Edge, Search, and AppScope. Remember, this is just the new feature list, so, it is kind of a big deal. Stream (release notes):This release provides the following improvements: Introducing Stream Projects (Beta). This feature enables Cribl admins to create isolated spaces for teams and users to share and manage their data.New native MSK (Amazon Managed Streaming for Kafka) Source and Destination support accessing Kafka topics via IAM roles.The Mask Function now supports toggling individual Masking Rules on and off, to enable iterative development and debugging. It also now supports the SHA512 hash algorithm.The Redis Fu

A week from today!  Sunil Kimar Gudivada will work through REST API integration for ServiceNow and Smartsheet.You know you don't want to miss that!