Community Discussions

TLDR : JSON Collected from API, parsed through Pipeline, should have three JSON files in S3, but have two where one file have 2 JSON object appended, need help to find out the reasonHi All,At first, I want to say, no this is not an event breaker issue, as much as my understanding. Let me explain.I am trying to get some data from a REST Api endpoint using REST Collector. I am using discovery with Item list, and in the list, there are three values. In preview, three different events are also getting registered (sometimes, sometimes nothing is coming to the preview also).Now I am parsing this response using pipeline and putting the data in a S3 bucket using the router. I followed the documentation to configure the Destination as well as the pipeline and route.Now, I am setting up the schedular to run the REST Collector. Then, after each collection event, when I am checking my s3, there instead of having three separate JSON files for thre

Is the Cribl supporting SNMPv3?

Hi everyone, I have a use case atm where I have a lookup that contains a column of "allowed fields" for various sourcetypes, that is, the value for a given sourcetype is a common separted list of field names. At the end of my pipeline, I basically want to import this field list and throw away all fields in the data that are not contained in the list (because I'm finished with the normalization). Is there an easy way to do this? The serialize function has essentially the same functionality, when you import the parser you get a keep fields list. But how to do this without the parser? Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/CPYBPK65V/p1682331204917309

Why is my Splunk destination showing unhealthy when the individual workers are showing a green healthy status?

How do I change a worker's temp directory from "/tmp"?

I am trying to add a new Edge node to my Cribl Cloud instance. I click on the Add/Update Edge Node and get the script. The docker instance installs but when it goes to run the Edge instance, it errors out.Server:Ubuntu 22.04.3 LTS (GNU/Linux 5.2.0 x86_64)Install:docker run -d --privileged -e "CRIBL_DIST_MASTER_URL=tls://{token}@{instance}.cribl.cloud:4200?group=default_fleet" -e "CRIBL_DIST_MODE=managed-edge" -e "CRIBL_EDGE=1" -p 9420:9420 -v "/:/hostfs:ro" --restart unless-stopped --name "cribl-edge" cribl/cribl:latestUnable to find image 'cribl/cribl:latest' locallylatest: Pulling from cribl/cribledaedc954fb5: Pull completef83a34600de3: Pull complete94a7ecdae25d: Pull complete995bcc02c181: Pull completeb0adc0627ec2: Pull completeDigest: sha256:8879de72b7cd95715e938d7828dd6adb0e5869fe893cfe1612cd520e0746bf46Status: Downloaded ne

When I use a parser function to parse my json _raw field, I see that my original data changes and it breaks my dashboards:Why does it change?

How to connect my Bitbucket repository with Cribl?Question: When trying to push to a remote git repository created in Bitbucket under a project, I am getting the following error:This is working for other repositories created under a specific user, but not under a project in bitbucket.

Hi.I need to continuously read events from log file which is UTF-16 encoded. Is any way to convert collected data to UTF-8?Before that I used File System Collector which could use "Custom Command" option. Using "/usr/bin/iconv -f UTF16BE -t UTF8" solves the problem. But there is no such option for "File Monitor" source.

Hello, we are triying to setup a Splunk Search collector, against an on-prem splunk with a self signed certificate. Is it possible to disable certificate validation?

https://docs.cribl.io/stream/usecase-replay-s3/#which-format Starting in version 4.3, Cribl Stream supports replaying data that has been exported as Parquet, using either the S3 Collector or the Filesystem Collector.Meanwhile, the Azure Blob Storage and Google Cloud Storage Collectors support ingesting data in Parquet format, but do not support replay.I am glad to see that Parquet is now supported for S3 replay.  Is replay actually a feature in itself or just a technique implemented via a Source with different handling?How can Azure Blob and Google Cloud Storage collectors support ingesting but not support replay?

There's something about filters in an eval function (and probably other functions as well) that I either don't understand or that don't work as advertised. I have some events, all of them already have a field state. In some of these events, the value is a straight number. In some events, the field value contains an epoch time and a value, delimited by a pipe, e.g. 1696254765000|5.333333333333333. To get that value, I though I'd just use an eval and filter on events that have a pipe, like seen in the screenshot. However, that seems to remove the field for all events that do not match the filter (events that have just a number in state). I created a state_copy field to illustrate this: the top event only has state_copy, it does not have a pipe in the field value, and the second event has

Hi all,Is there a cribl equivalent to Splunk’s rex mode=sed? I’ve tried replace_regex but I kept hitting a brick wall.I’m trying to take a timestamp with an indeterminate number of spaces and replace those spaces with a single space in a pipeline (using ‘eval’ function)Data looks like this:timestamp: “2023-10-03     09:29:00”Currently I’ve done the really hacky fix of extracting both the date and time and smooshing them together (eval date + ’ ' time)Thanks!

Hi All, I am having some issues when enabling a Smart Persistent Queue for a Splunk TCP source (TLS disabled). When the persistent queue is enabled the thruput drops by 2 orders of magnitude (from ~20k eps to ~200eps). The Cribl worker nodes are running on VMs, could this be due to the storage not being fast enough? If this is the reason why would the thruput drop so dramatically? Does enabling persistent queues on a source use much CPU?I don’t have any other source persistent queues enabled currently, but do have some on destinations that are working fine.

For a Raw HTTP source, if I put that behind a load balancer, what would I use for a health check?

Using "Webhook" Destination: Can we transform events to be used as HTTP requests?Background - we are trying to send alerts from the Cribl Internal (Logs) source.  We use a pipeline to reduce these events down to only Cribl errors and any Cribl Internal Logs that we care about.  However, we need a way to send this data in the correct format to something like  Slack "Incoming Webhooks."2 Problems with Sending Events as HTTP Requests Using "Webhook" Destination:Slack has a specific format and keys for the JSON body data -  https://api.slack.com/messaging/webhooks#advanced_message_formatting There are certain keys that may only appear in the JSON that Slack receives like "text".This is also problematic when we can't control if cribl_pipe is appended to the top-level fieldsThe JSON data must be indi

Does anyone have an example of a working configuration to receive Palo Alto logs over HTTPS?

Hello All,I'm new to Cribl and basically a Splunk Admin & developer. Been working on Cribl migration project for a while.My requirement is to collect data from a S3 bucket Via Cribl and apply some cool stuffs than send it to Splunk for indexing.Now, I have established the connection with my S3 bucket from Cribl stream. I'm looking for some assistance to optimize & send data to Splunk.Can anyone guide me the steps & tricks for same?TIA.

Anyone have a good set up for querying Azure AD via say graph to access specific user object parameters? Looking to replace our current method, which brings in only about 6-8 fields out of 20 something available. Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/C02A9EQV61J/p1680717534405409

I'm working with our security team who's wanting me to send events to his AWS ALB syslog destination which has multiple syslog nodes behind it.He's getting events, but the cribl UI is consistantly showing the destination as experiencing issues. It's going through a loop of successfully establishing a connection, 1 minute later the sender disconnects it, 1 minute later it restablishesI do not experience this same thing when I use a non-loadbalanced syslog destination.Does anyone have thoughts on how to resolve or even troubleshoot this? Would using TLS help?

Hello,I noticed yesterday that the cribl_route system field is always added to all events even if deactivated in the system fields post processing settings of my destination (Splunk).I am running a Stream cloud instance since more than a year and never had this issue before.Anyone else has the issue ?

Hello,We are exploring running a self-hosted Cribl deployment on Kubernetes (EKS+Fargate) using the provided Cribl Helm charts. We've got a leader and worker pod running. We are now looking at persistent queuing.My question is: what's the recommended configuration for ensuring no data loss in the case of a worker pod terminating with events still in the persistent queue?We've been exploring the use of extraVolumeMounts and persistent volumes backed backed by EFS. We found this technique in this blog post (https://cribl.io/blog/persistent-queueing-using-the-logstream-workergroup-chart/) but we see it is not recommended in the the docs (https://docs.cribl.io/stream/deploy-kubernetes#usin

An export of code is done to a json fromat only while on servers is used a yml fromat.IMHO, it should be possible to do import/export also in yml format. :)

Hi I am trying to setup Cribl Stream leader on Fargate with EFS as a persistent storage for the leader using Cribl_Volume_Dir environment variable. The container keeps running with unhealthy health check. Anyone has a working task definition for me to refer to?

From the Data Routes, Apache has a yellow triangle. The Pipeline is Apache_Access.I tried other pipelines and it did not make a difference. Any pointer for me is appreciated.