Stream

Hello community,we have a bunch of workers in different network zones with different sources (syslog,s2s,tls etc.)How would you prefer to proceed with the groups management?We got 2 options here:All workers in a single group with all sources (if you deploy a new source, all workers need a restart? Not good at all)A group for every network zone, with a worker, with dedicated sources (we think it is really uncomfortable to work with a lot of groups#, because every group has its own configuration)Thanks for feedback and suggestions

In regard to workers in a distributed environment, is there any recommendation on the Network bonding configuration? Would mode 5 balanced-tlb be a better choice than mode 1 active-backup?Thanks

our cribl_pipe field being a json array and they don’t know how to use splunk to mvexpand them, instruct them to use the flatten function instead in a post-processing pipeline.I have multiple values in the cribl_pipe field for my events. How do I correctly process those in Splunk when the events are sent downstream? I’m trying to use mvexpand but it’s not working as desired.

Hi, is it possible for the REST collector to save a checkpoint (e.g. sequential ID of last collected event) and use this as a starting point at the following execution?Something like Splunk’s DB Connect “growing column” setting.

When reviewing license usage in an Enterprise cribl.cloud instance, it’s been noticed that the license page only shows 1TB available, even though more has been provisioned to the Org.

In a passthru route, Stream is adding the Cribl Worker ip address into the host field before sending to Splunk. How that can be avoided? This fields already comes from the source with different values.

Hello,we want to copy all routes from one group to another including all the comments on the same stage. We tried the following ways:Copy and Paste via Advanced ModeCopy and Paste the routes.yml via Shell to the other groupIn Advanced Mode export and then import in the other groupIn Advanced Mode export and then import in the other group on another stageNone of this ways were copying the comments. Is there a working way to copy the routes from one group to another?Thank you in advance

I am testing out the database collector in v4.0 (cannot choose v4 in the form for creating a new topic) and running into an issue. Some of this is documented in Slack but I wanted to bring it here as well. I cannot get a connection to an MSSQL server using an Microsoft Domain account (AD). For security reasons, we do not use SQL accounts, we use service accounts created in our active directory domain. I have tried DOMAIN\user in the “User ID” option in the Connection String but it does not work. Without this feature then this collector option is not going to be used. Can someone shed some light on this?

Are there any tricks to parse messagepack data in Cribl? We want to take send large amounts of log data from fluentd, but it doesn’t have a TCP output plugin - just forward (messagepack), http, or syslog.We’re probably going to have to go with http, but the overheads involved and performance hit are quite significant, so it’d be great to use messagepack natively, and save compute on both ends.I don’t see any source type, nor any function under C that looks like it can do the job.I’d be very happy with a breaker, or a pre-processing pipeline on a raw TCP input. There just doesn’t seem to be a native way to handle messagepack.It doesn’t look like any messagepack plugins are installed natively - would it be possible to install a plugin in the workers config somehow?

Hi, we currently receive multiple data types (ad, dns, dhcp) in splunk via one source/port and split them with multiple input apps into multiple indices within the correct data.How can we do that in Cribl? With multiple routes/pipelines to forward the data into the correct indices?Thanks

How do I keep the punct field when sending from a Splunk UF through stream

In Docker, is there a way to add workers to groups other than "default" upon deployment?

For the S3 collector if I have events with a _time field from 3/1/2022 in a file created today, so it is in a path of /dir1/<today’s_date>/dir2/. Will the earliest/latest values filter those events out if I select -24h? I do not want to filter those out, but do want to be able to collect older events that get put into newly created paths.

We are working on setting up High Availability for Leader Nodes and it mentions that we need to have a directory NFS mounted to both the Primary and the Backup Leader Node.Should this directory be within $CRIBL_HOME directory or elsewhere?

Our destination has PQ enabled. When the destination is accepting data again we are seeing error messages in the logs that the there's blocking when it is set to drop. Why?{ "level": "warn", "message": "begin backpressure [blocking]", "reason": { "message": "failed to push to persistent queue due to backpressure: in backpressure mode", "stack": "PQBackpressureError: failed to push to persistent queue due to backpressure: in backpressure mode\n at /data/cribl/opt/bin/cribl.js:14:12494168", "reason": "output is experiencing increased load", "name": "PQBackpressureError", "response": { "shouldBackpressure": true, "reason": "in backpressure mode", "forceBlock": true } }}

Does Stream attach the worker ID to each event in the internal fields like cribl_pipe does for the pipelines? If so, what is the field name? If not, is there a way to make stream add the worker ID to the event?

Would like to be able to convert SNMP traps to JSON and then perform subsequent routing and enrichment leveraging the JSON content. Thanks.

I am working with an external API to collect data. This works beautifully except they throttle requests per second to 1.Is there a way in Stream where I can enforce 1 request per second at collection phase? Not per second/per worker unfortunately. Hard limit of 1 request per second which results in 429 errors for most of the pulls.I am on version 3.5.4.

I’m attempting to send Cribl Stream notifications for one of my sources via HEC to Splunk. I’ve configured the Webhook target with the url to my Splunk instance’s HEC endpoint with the token in the url. However, I see Status Codes of 400 and the following message in Stream: text:Query string authorization is not enabled. How do I properly configure the notification target to send via HEC to Splunk?

Trying to do a playback using Azure Blob. We are sending logs to this Azure Blob for our DataLake. Logs are being written to the Blob and are stored by year/month/day.Set up the Azure Blob as a source using the same connection strings. When running a preview test we get the following error.This request is not authorized to perform this operation using this permission.code:AuthorizationPermissionMismatch

What are the default connection limits for a worker? Is there a maximum number of connections a worker can handle?

We're seeing the following message and events are no longer flowing through the Worker Group.Message: Failed to send anonymized telemetry metadata. Data flow has been stopped.Failed to send anonymized telemetry metadata. License is blocking inputs. Please contact sales@cribl.io or check that firewall rules are not blocking Cribl telemetry metadata collection.

This is gonna sound insane buuuuut…we have a team that has some evtx files that sit on a NAS. Using edge on an independent windows machine, is it possible to monitor the files on the NAS location to collect those files thru the Windows Event Logs source in Edge? Or is it only possible to collect the local windows files for that machine specifically?

Noticed one of multiple collectors wasn’t working (I probably broke it last time I was fooling around). If I go to Data/Sources, “Collectors” at the top can be clicked on but the resulting page is empty. If I select any other input on that page, it goes to that page and from there I can see there are 12 Collectors configured, but if I click Collectors there I just get an “API error” and get dumped back at the main Data Sources page.

How can I use the Stream REST API to create new routes? The API doc reference doesn’t show a POST but I do see a PATCH. Is PATCH the operation I should use? Can someone provide an example payload by chance?