Stream

For other users that have started pulling O365 activity logs using Stream instead of the Splunk app, have you noticed a discrepancy in the event counts? We consistently receive more events through the Splunk app than we do through the Stream source. The setup is exactly the same: tenant, app Id, content, and interval. We've even done comparisons into the data and have found events missing through Stream that were retrieved by the Splunk app. So it seems like Stream is either dropping some events or just not pulling all of them.We do have a case open with Cribl already but I'm wondering if we're the only ones seeing this. Has anyone else noticed compared the volume and noticed the same?

Hi All,I am new to Cribl and trying to set notification via webhook for destination have back pressure or destination is unhealthy. But i am getting below error:"the notifications service is not available because a configuration file doesn't exist or the settings are invalid."

We installed the Cribl on our Linux server, and by default, it is user=cribl group=cribl.Our organization requires us to us a Service account to run configuration inside it.Question :Can we change the user and group ownership of the Linux directory of Cribl?And would it affect anything ?From user=cribl group=cribl.To user=svcacc_cribl group=svcacc_cribl

I am currently setting an environmental variable in a Script collector using a JS Expression as follows: TEST_VAR = C.Secret("test1").valueThis works fine for some secrets, but not for others. In the latter case, TEST_VAR is equal to 'C.Secret("test1").value' instead of the actual value.I get the impression that the secrets for which it does not work have not yet been distributed to the workers in the group. Is this correct? Or is there something else wrong here.

With Cribl Stream what is the best practice for receiving windows logs?

We're using AWS's c5d instance types which include NVMe instance storage for CRIBL_VOLUME_DIR, but we're also paying for a small EBS volume to host the configs in /opt/cribl.Is there a compelling reason to not just put CRIBL_HOME in the NVMe storage as well and have it pull down the config on startup?I came to this question because I was thinking about a scenario where we might shut down instances during low traffic times and how to handle being sure the CRIBL_VOLUME_DIR on the Ephemeral storage was formatted and mounted. It made me think, "If CRIBL_HOME was also there, then I wouldn't have to worry about CRIBL starting up without CRIBL_VOLUME_DIR being formatted and mounted".

I have an Event with no Splunk Metadata value within the events and i am trying to add new fields such as host, index,source and sourcetype . custom fields and i tried using an eval but no luck, how do i go about this ?

TLDR : JSON Collected from API, parsed through Pipeline, should have three JSON files in S3, but have two where one file have 2 JSON object appended, need help to find out the reasonHi All,At first, I want to say, no this is not an event breaker issue, as much as my understanding. Let me explain.I am trying to get some data from a REST Api endpoint using REST Collector. I am using discovery with Item list, and in the list, there are three values. In preview, three different events are also getting registered (sometimes, sometimes nothing is coming to the preview also).Now I am parsing this response using pipeline and putting the data in a S3 bucket using the router. I followed the documentation to configure the Destination as well as the pipeline and route.Now, I am setting up the schedular to run the REST Collector. Then, after each collection event, when I am checking my s3, there instead of having three separate JSON files for three separate events, there are two JSON files, one wit

Is the Cribl supporting SNMPv3?

Hi everyone, I have a use case atm where I have a lookup that contains a column of "allowed fields" for various sourcetypes, that is, the value for a given sourcetype is a common separted list of field names. At the end of my pipeline, I basically want to import this field list and throw away all fields in the data that are not contained in the list (because I'm finished with the normalization). Is there an easy way to do this? The serialize function has essentially the same functionality, when you import the parser you get a keep fields list. But how to do this without the parser? Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/CPYBPK65V/p1682331204917309

Why is my Splunk destination showing unhealthy when the individual workers are showing a green healthy status?

How do I change a worker's temp directory from "/tmp"?

When I use a parser function to parse my json _raw field, I see that my original data changes and it breaks my dashboards:Why does it change?

How to connect my Bitbucket repository with Cribl?Question: When trying to push to a remote git repository created in Bitbucket under a project, I am getting the following error:This is working for other repositories created under a specific user, but not under a project in bitbucket.

Hi.I need to continuously read events from log file which is UTF-16 encoded. Is any way to convert collected data to UTF-8?Before that I used File System Collector which could use "Custom Command" option. Using "/usr/bin/iconv -f UTF16BE -t UTF8" solves the problem. But there is no such option for "File Monitor" source.

Hello All,I'm new to Cribl and basically a Splunk Admin & developer. Been working on Cribl migration project for a while.My requirement is to collect data from a S3 bucket Via Cribl and apply some cool stuffs than send it to Splunk for indexing.Now, I have established the connection with my S3 bucket from Cribl stream. I'm looking for some assistance to optimize & send data to Splunk.Can anyone guide me the steps & tricks for same?TIA.

I'm working with our security team who's wanting me to send events to his AWS ALB syslog destination which has multiple syslog nodes behind it.He's getting events, but the cribl UI is consistantly showing the destination as experiencing issues. It's going through a loop of successfully establishing a connection, 1 minute later the sender disconnects it, 1 minute later it restablishesI do not experience this same thing when I use a non-loadbalanced syslog destination.Does anyone have thoughts on how to resolve or even troubleshoot this? Would using TLS help?

Hello,I noticed yesterday that the cribl_route system field is always added to all events even if deactivated in the system fields post processing settings of my destination (Splunk).I am running a Stream cloud instance since more than a year and never had this issue before.Anyone else has the issue ?

Hello,We are exploring running a self-hosted Cribl deployment on Kubernetes (EKS+Fargate) using the provided Cribl Helm charts. We've got a leader and worker pod running. We are now looking at persistent queuing.My question is: what's the recommended configuration for ensuring no data loss in the case of a worker pod terminating with events still in the persistent queue?We've been exploring the use of extraVolumeMounts and persistent volumes backed backed by EFS. We found this technique in this blog post (https://cribl.io/blog/persistent-queueing-using-the-logstream-workergroup-chart/) but we see it is not recommended in the the docs (https://docs.cribl.io/stream/deploy-kubernetes#using-persistent-storage-for-persistent-queueing). We were wondering if there are better solutions that allow for a more automated recovery of events sitting in the persistent queue of the lost pod?

From the Data Routes, Apache has a yellow triangle. The Pipeline is Apache_Access.I tried other pipelines and it did not make a difference. Any pointer for me is appreciated.

What are the steps to create worker group via CLI?

How many worker groups can a Cribl Stream leader support?

I have a big deployment with hundreds of worker nodes. Is there an easy way to download a full list of worker nodes in json format?

When using syslog-ng server my syslog data is 'pinning' to a single stream worker process when sending syslog over TCP. Am sending too much data for that single WP to handle?

Hi all,I want to do a drop-in replacement for an existing syslog-ng + Splunk HF system. Currently, it receives a variety of logs via syslog, syslog-ng writes them to disk (using different paths per sourcetype), and then the Splunk HF reads them, applies index-time transforms based on sourcetype, and forwards to the indexing tier. I can see a variety of ways to approach this, so I’m looking for a best-practice recommendation. How would you set this up so that some event sources can be processed with Cribl, but others can be left to the Splunk HF/Indexer? In other words, I’d like to make use of things like the PAN pack, but also be able to have it just work the same as before with some outlier sources, which means passing to the HF to process props/transforms on the data.