Skip to main content

Stream

Ask and answer questions about routing, shaping, and optimizing observability and security data.

  • 283 Topics
  • 601 Replies
283 Topics
P
padillaNew Participant
asked in Stream
How to Ignore Timestamps in S3 Collector?

For the S3 collector if I have events with a _time field from 3/1/2022 in a file created today, so it is in a path of /dir1/<today’s_date>/dir2/. Will the earliest/latest values filter those events out if I select -24h? I do not want to filter those out, but do want to be able to collect older events that get put into newly created paths.

02
J
P
padillaNew Participant
asked in Stream
NFS Mount Directory Location for HA

We are working on setting up High Availability for Leader Nodes and it mentions that we need to have a directory NFS mounted to both the Primary and the Backup Leader Node.Should this directory be within $CRIBL_HOME directory or elsewhere?

71
M
P
padillaNew Participant
asked in Stream
Backpressure/Blocking when PQ Drained and Destination Healthy

Our destination has PQ enabled. When the destination is accepting data again we are seeing error messages in the logs that the there's blocking when it is set to drop. Why?{ "level": "warn", "message": "begin backpressure [blocking]", "reason": { "message": "failed to push to persistent queue due to backpressure: in backpressure mode", "stack": "PQBackpressureError: failed to push to persistent queue due to backpressure: in backpressure mode\n at /data/cribl/opt/bin/cribl.js:14:12494168", "reason": "output is experiencing increased load", "name": "PQBackpressureError", "response": { "shouldBackpressure": true, "reason": "in backpressure mode", "forceBlock": true } }}

11
M
C
Chris RogersEmployee
asked in Stream
Finding which pipeline and worker touched an event

Does Stream attach the worker ID to each event in the internal fields like cribl_pipe does for the pipelines? If so, what is the field name? If not, is there a way to make stream add the worker ID to the event?

12
K
P
plummcrazyNew Participant
asked in Stream
How can an SNMP trap be converted to JSON in Cribl Stream?

Would like to be able to convert SNMP traps to JSON and then perform subsequent routing and enrichment leveraging the JSON content. Thanks.

02
G
A
amillerInspiring
asked in Stream
REST Collector Requests per second?

I am working with an external API to collect data. This works beautifully except they throttle requests per second to 1.Is there a way in Stream where I can enforce 1 request per second at collection phase? Not per second/per worker unfortunately. Hard limit of 1 request per second which results in 429 errors for most of the pulls.I am on version 3.5.4.

12
J
J
Jeremy PrescottEmployee
asked in Stream
How to configure Splunk HEC as a Notification Webhook Target

I’m attempting to send Cribl Stream notifications for one of my sources via HEC to Splunk. I’ve configured the Webhook target with the url to my Splunk instance’s HEC endpoint with the token in the url. However, I see Status Codes of 400 and the following message in Stream: text:Query string authorization is not enabled. How do I properly configure the notification target to send via HEC to Splunk?

01
J
M
Michael BrunnmeierNew Participant
asked in Stream
Permissions in Azure Blob for playback

Trying to do a playback using Azure Blob. We are sending logs to this Azure Blob for our DataLake. Logs are being written to the Blob and are stored by year/month/day.Set up the Azure Blob as a source using the same connection strings. When running a preview test we get the following error.This request is not authorized to perform this operation using this permission.code:AuthorizationPermissionMismatch

11
J
C
Chris RogersEmployee
asked in Stream
Worker Connection Limits

What are the default connection limits for a worker? Is there a maximum number of connections a worker can handle?

01
B
C
Chris RogersEmployee
asked in Stream
Using a Load Balancer to Check Stream Health

Is there a URL or API endpoint I can use to check the status of a Stream Worker or Leader? If so, what HTTP Status codes does the endpoint return?

00
E
M
Martin PradoEmployee
asked in Stream
Failed to send anonymized telemetry metadata. Data flow has been stopped

We're seeing the following message and events are no longer flowing through the Worker Group.Message: Failed to send anonymized telemetry metadata. Data flow has been stopped.Failed to send anonymized telemetry metadata. License is blocking inputs. Please contact sales@cribl.io or check that firewall rules are not blocking Cribl telemetry metadata collection.

31
K
J
jaime_raderNew Participant
asked in Stream
Evtx file ingestion

This is gonna sound insane buuuuut…we have a team that has some evtx files that sit on a NAS. Using edge on an independent windows machine, is it possible to monitor the files on the NAS location to collect those files thru the Windows Event Logs source in Edge? Or is it only possible to collect the local windows files for that machine specifically?

02
J
B
Brandon McCombsEmployee
asked in Stream
API error but only with the Collectors page

Noticed one of multiple collectors wasn’t working (I probably broke it last time I was fooling around). If I go to Data/Sources, “Collectors” at the top can be clicked on but the resulting page is empty. If I select any other input on that page, it goes to that page and from there I can see there are 12 Collectors configured, but if I click Collectors there I just get an “API error” and get dumped back at the main Data Sources page.

11
B
B
Brandon McCombsEmployee
asked in Stream
What is the process for ensuring Stream/Edge works with new IPs on my workers?

I’m changing the ip on a couple of my workers - anything that I need to change or be aware of? My assumption is that Stream/Edge will just pickup the change and it will appear on the leader with those changes. Nothing else is changing.

00
J
B
Brandon McCombsEmployee
asked in Stream
Using the API to create new routes

How can I use the Stream REST API to create new routes? The API doc reference doesn’t show a POST but I do see a PATCH. Is PATCH the operation I should use? Can someone provide an example payload by chance?

01
R
J
Jeremiah RossParticipating Frequently
asked in Stream
How to use the UI to create search filters for the /system/logs/search API endpoint

I’m using the /system/logs/search endpoint to filter for custom events within the access.log file.Filters work properly via log search in the UIMy filter experience with the API is still clumsy and time consumingIs there a way to quickly generate a custom search query that can be used with the log search endpoint from the UI?

01
J
M
mauriciothomsenNew Participant
asked in Stream
Preview results may be incomplete due to process timeout

When running a preview of a Pack with two lookup functions pointing to 1 million records files, I get the message:Preview results may be incomplete due to process timeoutIf I disable one of the functions - it does not matter which one - the preview works fine.Is there a lookup limit the preview can handle? The sample file used contains 57 events.Thanks.

12
M
E
Eugene KatzEmployee
asked in Stream
Recovering worker group configs from worker nodes

If a distributed leader’s configuration is lost and it reverts to a standalone instance, the existing worker group configurations should remain on the workers, since they haven’t gotten any updates. What’s the easiest way to get those configurations copied over to the leader?

01
G
S
Steve BridgeKnown Participant
asked in Stream
Manual backups for Stream stand-alone

Hi all. For us poor slobs currently stuck with a manual process… has anyone tested a backup and restore? I am zipping /local, /data, and /default under /opt/cribl, and a quick test seemed to indicate this works for DR, but anyone know of other directories that could be needed?

03
B
D
Draco3New Participant
asked in Stream
Breaking a single stream input into separate pipelines

I have a data stream that contains various different log types. For example, I have a Windows log, a firewall log and an antivirus log in the same stream. Each of the events from the source is tagged for exampletype: windowstype: firewalltype: antivirusWhat I want to do is split these logs by type. Essentially I would like to do:If the type equals windows then drop everything else and run it through a pipeline / route to transform my data with a windows parser and ship my dataIf the type equals firewall then drop everything else and run it through a pipeline / route to transform my data with a firewall parser and ship my dataIf the type equals antivirus then drop everything else and run it through a pipeline / route to transform my data with a antivirus parser and ship my dataI started thinking that I would use the eval function to filter based on this and group them together but unsure if this would destroy my data due to order. I also started thinking that I could do some pre-pro

01
B
C
cmianzaNew Participant
asked in Stream
I Cannot configure RBAC on a single instance

We are currently in PoC. We cannot configure RBAC on a single instance. I need this to work; reading in Cribl documents, RBAC is allowed only in a distributed environment; this is not feasible. How do you go around this restriction?

13
B
S
Sven BreierParticipating Frequently
asked in Stream
Cribl Leader Logs to Splunk

Hello everyone,is there a “best practise” way to collect internal and metrics logs from the cribl leader?For worker is an existing Source called “Cribl Internal”.Thank you

25
S
S
Sven BreierParticipating Frequently
asked in Stream
Copy outputs.yml to differnt worker

Hello there,we currently have a leader with 10 workers in different groups under it.We use Indexer discovery within Splunk to connect the workers in the destinations tab to our Indexer.Can i copy the outputs.yml from one worker to another worker in a seperated group to directly have the config ready? Or do i have to go the way thorugh the GUI and make my marks.Thank you

02
S
J
Johan WogerInspiring
asked in Stream
Syslog-ng invalid frame header issue

I was sending events to a syslog-ng server with the following configsource s_test { syslog( transport(tcp) port());};I introduced Stream, and now I’m not seeing those logs on our syslog-ng server. Instead, I’m seeing this:Invalid frame header; header=‘’syslog-ng[3025832]: Syslog connection closed; fd=‘12’, client=‘AF_INET(10.1.80.158:12506)’, local=‘AF_INET(0.0.0.0:23514)’How can I fix this?

61
P
K
Kyle McCririeEmployee
asked in Stream
How can I send Heroku logs to Cribl Stream/Edge?

I am interested in sending Heroku logs to Stream. Has anyone ever done this before?

01
P

Badge winners

Show all badges
Terms & ConditionsCookie settings