Stream

Hi, we currently receive multiple data types (ad, dns, dhcp) in splunk via one source/port and split them with multiple input apps into multiple indices within the correct data.How can we do that in Cribl? With multiple routes/pipelines to forward the data into the correct indices?Thanks

How do I keep the punct field when sending from a Splunk UF through stream

In Docker, is there a way to add workers to groups other than "default" upon deployment?

For the S3 collector if I have events with a _time field from 3/1/2022 in a file created today, so it is in a path of /dir1/<today’s_date>/dir2/. Will the earliest/latest values filter those events out if I select -24h? I do not want to filter those out, but do want to be able to collect older events that get put into newly created paths.

We are working on setting up High Availability for Leader Nodes and it mentions that we need to have a directory NFS mounted to both the Primary and the Backup Leader Node.Should this directory be within $CRIBL_HOME directory or elsewhere?

Our destination has PQ enabled. When the destination is accepting data again we are seeing error messages in the logs that the there's blocking when it is set to drop. Why?{ "level": "warn", "message": "begin backpressure [blocking]", "reason": { "message": "failed to push to persistent queue due to backpressure: in backpressure mode", "stack": "PQBackpressureError: failed to push to persistent queue due to backpressure: in backpressure mode\n at /data/cribl/opt/bin/cribl.js:14:12494168", "reason": "output is experiencing increased load", "name": "PQBackpressureError", "response": { "shouldBackpressure": true, "reason": "in backpressure mode", "forceBlock": true } }}

Does Stream attach the worker ID to each event in the internal fields like cribl_pipe does for the pipelines? If so, what is the field name? If not, is there a way to make stream add the worker ID to the event?

Would like to be able to convert SNMP traps to JSON and then perform subsequent routing and enrichment leveraging the JSON content. Thanks.

I am working with an external API to collect data. This works beautifully except they throttle requests per second to 1.Is there a way in Stream where I can enforce 1 request per second at collection phase? Not per second/per worker unfortunately. Hard limit of 1 request per second which results in 429 errors for most of the pulls.I am on version 3.5.4.

I’m attempting to send Cribl Stream notifications for one of my sources via HEC to Splunk. I’ve configured the Webhook target with the url to my Splunk instance’s HEC endpoint with the token in the url. However, I see Status Codes of 400 and the following message in Stream: text:Query string authorization is not enabled. How do I properly configure the notification target to send via HEC to Splunk?

Trying to do a playback using Azure Blob. We are sending logs to this Azure Blob for our DataLake. Logs are being written to the Blob and are stored by year/month/day.Set up the Azure Blob as a source using the same connection strings. When running a preview test we get the following error.This request is not authorized to perform this operation using this permission.code:AuthorizationPermissionMismatch

What are the default connection limits for a worker? Is there a maximum number of connections a worker can handle?

We're seeing the following message and events are no longer flowing through the Worker Group.Message: Failed to send anonymized telemetry metadata. Data flow has been stopped.Failed to send anonymized telemetry metadata. License is blocking inputs. Please contact sales@cribl.io or check that firewall rules are not blocking Cribl telemetry metadata collection.

This is gonna sound insane buuuuut…we have a team that has some evtx files that sit on a NAS. Using edge on an independent windows machine, is it possible to monitor the files on the NAS location to collect those files thru the Windows Event Logs source in Edge? Or is it only possible to collect the local windows files for that machine specifically?

Noticed one of multiple collectors wasn’t working (I probably broke it last time I was fooling around). If I go to Data/Sources, “Collectors” at the top can be clicked on but the resulting page is empty. If I select any other input on that page, it goes to that page and from there I can see there are 12 Collectors configured, but if I click Collectors there I just get an “API error” and get dumped back at the main Data Sources page.

How can I use the Stream REST API to create new routes? The API doc reference doesn’t show a POST but I do see a PATCH. Is PATCH the operation I should use? Can someone provide an example payload by chance?

I’m using the /system/logs/search endpoint to filter for custom events within the access.log file.Filters work properly via log search in the UIMy filter experience with the API is still clumsy and time consumingIs there a way to quickly generate a custom search query that can be used with the log search endpoint from the UI?

When running a preview of a Pack with two lookup functions pointing to 1 million records files, I get the message:Preview results may be incomplete due to process timeoutIf I disable one of the functions - it does not matter which one - the preview works fine.Is there a lookup limit the preview can handle? The sample file used contains 57 events.Thanks.

If a distributed leader’s configuration is lost and it reverts to a standalone instance, the existing worker group configurations should remain on the workers, since they haven’t gotten any updates. What’s the easiest way to get those configurations copied over to the leader?

Hi all. For us poor slobs currently stuck with a manual process… has anyone tested a backup and restore? I am zipping /local, /data, and /default under /opt/cribl, and a quick test seemed to indicate this works for DR, but anyone know of other directories that could be needed?

I have a data stream that contains various different log types. For example, I have a Windows log, a firewall log and an antivirus log in the same stream. Each of the events from the source is tagged for exampletype: windowstype: firewalltype: antivirusWhat I want to do is split these logs by type. Essentially I would like to do:If the type equals windows then drop everything else and run it through a pipeline / route to transform my data with a windows parser and ship my dataIf the type equals firewall then drop everything else and run it through a pipeline / route to transform my data with a firewall parser and ship my dataIf the type equals antivirus then drop everything else and run it through a pipeline / route to transform my data with a antivirus parser and ship my dataI started thinking that I would use the eval function to filter based on this and group them together but unsure if this would destroy my data due to order. I also started thinking that I could do some pre-pro

We are currently in PoC. We cannot configure RBAC on a single instance. I need this to work; reading in Cribl documents, RBAC is allowed only in a distributed environment; this is not feasible. How do you go around this restriction?

Hello everyone,is there a “best practise” way to collect internal and metrics logs from the cribl leader?For worker is an existing Source called “Cribl Internal”.Thank you

Hello there,we currently have a leader with 10 workers in different groups under it.We use Indexer discovery within Splunk to connect the workers in the destinations tab to our Indexer.Can i copy the outputs.yml from one worker to another worker in a seperated group to directly have the config ready? Or do i have to go the way thorugh the GUI and make my marks.Thank you

I was sending events to a syslog-ng server with the following configsource s_test { syslog( transport(tcp) port());};I introduced Stream, and now I’m not seeing those logs on our syslog-ng server. Instead, I’m seeing this:Invalid frame header; header=‘’syslog-ng[3025832]: Syslog connection closed; fd=‘12’, client=‘AF_INET(10.1.80.158:12506)’, local=‘AF_INET(0.0.0.0:23514)’How can I fix this?