General

does anyone know how to 'unflatten' fields in Cribl Stream? For example: source.ip is a flattened field that I need to be modified as json: '{ "source": { "ip": "whatever" } }' Original message in Cribl Community Slack Message:https://cribl-community.slack.com/archives/C02A9EQV61J/p1698617495203379

Hello, we are triying to setup a Splunk Search collector, against an on-prem splunk with a self signed certificate. Is it possible to disable certificate validation?

https://docs.cribl.io/stream/usecase-replay-s3/#which-format Starting in version 4.3, Cribl Stream supports replaying data that has been exported as Parquet, using either the S3 Collector or the Filesystem Collector.Meanwhile, the Azure Blob Storage and Google Cloud Storage Collectors support ingesting data in Parquet format, but do not support replay.I am glad to see that Parquet is now supported for S3 replay.  Is replay actually a feature in itself or just a technique implemented via a Source with different handling?How can Azure Blob and Google Cloud Storage collectors support ingesting but not support replay?

There's something about filters in an eval function (and probably other functions as well) that I either don't understand or that don't work as advertised. I have some events, all of them already have a field state. In some of these events, the value is a straight number. In some events, the field value contains an epoch time and a value, delimited by a pipe, e.g. 1696254765000|5.333333333333333. To get that value, I though I'd just use an eval and filter on events that have a pipe, like seen in the screenshot. However, that seems to remove the field for all events that do not match the filter (events that have just a number in state). I created a state_copy field to illustrate this: the top event only has state_copy, it does not have a pipe in the field value, and the second event has

Hi all,Is there a cribl equivalent to Splunk’s rex mode=sed? I’ve tried replace_regex but I kept hitting a brick wall.I’m trying to take a timestamp with an indeterminate number of spaces and replace those spaces with a single space in a pipeline (using ‘eval’ function)Data looks like this:timestamp: “2023-10-03     09:29:00”Currently I’ve done the really hacky fix of extracting both the date and time and smooshing them together (eval date + ’ ' time)Thanks!

Hi All, I am having some issues when enabling a Smart Persistent Queue for a Splunk TCP source (TLS disabled). When the persistent queue is enabled the thruput drops by 2 orders of magnitude (from ~20k eps to ~200eps). The Cribl worker nodes are running on VMs, could this be due to the storage not being fast enough? If this is the reason why would the thruput drop so dramatically? Does enabling persistent queues on a source use much CPU?I don’t have any other source persistent queues enabled currently, but do have some on destinations that are working fine.

For a Raw HTTP source, if I put that behind a load balancer, what would I use for a health check?

Using "Webhook" Destination: Can we transform events to be used as HTTP requests?Background - we are trying to send alerts from the Cribl Internal (Logs) source.  We use a pipeline to reduce these events down to only Cribl errors and any Cribl Internal Logs that we care about.  However, we need a way to send this data in the correct format to something like  Slack "Incoming Webhooks."2 Problems with Sending Events as HTTP Requests Using "Webhook" Destination:Slack has a specific format and keys for the JSON body data -  https://api.slack.com/messaging/webhooks#advanced_message_formatting There are certain keys that may only appear in the JSON that Slack receives like "text".This is also problematic when we can't control if cribl_pipe is appended to the top-level fieldsThe JSON data must be indi

Does anyone have an example of a working configuration to receive Palo Alto logs over HTTPS?

Anyone have a good set up for querying Azure AD via say graph to access specific user object parameters? Looking to replace our current method, which brings in only about 6-8 fields out of 20 something available. Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/C02A9EQV61J/p1680717534405409

Hi I am trying to setup Cribl Stream leader on Fargate with EFS as a persistent storage for the leader using Cribl_Volume_Dir environment variable. The container keeps running with unhealthy health check. Anyone has a working task definition for me to refer to?

Hi, I am trying to ingest sysmon logs via the Elastic api. But i do not see any live data but instead i get all dropped counts. Can anyone help?

Quick question. If I have a field called like "srcIP" with a value of 1.2.3.4, and sent it to a system like Splunk or Elasticsearch, would it see the srcIP field as a string or as an IP field? Because I would want to ensure that there is a way for the values to be the right type and be converted if a change is needed.

Our Splunk Cloud environment requires IP whitelists in order to access or ingest data. If we have a Cribl cloud org created, is there a way to get the IPs that would be required to whitelist in Splunk Cloud?

In a cribl cloud deployment, what certificate do we use for TLS for syslog?

Any reason my leader doesn't want to grab 4.03?

I am finally getting around to converting some of my crontabs to Cribl Stream API sources. I am looking to have a dynamic date/time variable in the URL. The date would need to be the date from yesterday. Example: Today's Date/Time: 2023-08-11T08:00:00.00Z Needed Date/Time: 2023-08-10T08:00:00.00Z Using %3C to do before date. I need the `dateLastValue` field to change. Example URL:https://web.site.url/api/v3/discussions?dateLastValue=%3C2023-08-10T08%3A00%3A00.00Z&status=true&field1=false&field2=true&page=1&limit=30Need something like: https://web.site.url/api/v3/discussions?dateLastValue=%3C{date-1_day}T08%3A00%3A00.00Z&status=true&field1=false&field2=true&page=1&limit=30

Is there any documentation available for implementing local storage archival and replay functions?

how would i configure encryption from a splunk uf to cribl cloud?

I was asked by the customer if it is possible to configure Cribl to sign the commits to github? They use remote git for backing up the configurations.

We are seeing this issue quite often, the leader UI shows the the rotating wheel whenever a particular worker group is selected and it never loads the pipelines or routes. This issue happens only for one worker group and others load fine. A restart of the leader fixes the problem temporarily. I noticed the following error on leader but not clear on what it is indicating. The CPU usage of the leader is below average.

quick question (I think) - if you need wildcards to reference existing field names in an eval, how do you reference those field names in your eval? e.g. do an eval operation against fieldA, fieldB, and fieldC using a single operation? could be any fieldX

Hello,While configuring cribl edge source of type file monitor and setting discovery mode to manual, it keeps on telling me permission denied(attached) and the *.log files are not monitored, although the user running cribl have access to this directory and the underneath *.log files. Attached the configuration of the source.Any idea what could be causing this issue?

If I am reading the docs correctly, when converting from json to CSV, I have to define all the fields. Since all of the fields are really already known, is there a way to output all the fields first without having to define all of them? We are working on converting some Azure logs, and there are LOTs of properties.field(s). Hoping there is another way without us having to define all of these in the "Fields to serialize" field within the Serialize function.

is there a way to remove the metric event but still send it to splunk not as a metric (if i hover over it says metric event) i was trying to move _value to value and _metric to metric__name