Skip to main content

Stream

Ask and answer questions about routing, shaping, and optimizing observability and security data.

  • 282 Topics
  • 601 Replies
282 Topics
J
Jason RehmNew Participant
asked in Stream
syslog output to ALB showing TCP endpoint errors

I'm working with our security team who's wanting me to send events to his AWS ALB syslog destination which has multiple syslog nodes behind it.He's getting events, but the cribl UI is consistantly showing the destination as experiencing issues. It's going through a loop of successfully establishing a connection, 1 minute later the sender disconnects it, 1 minute later it restablishesI do not experience this same thing when I use a non-loadbalanced syslog destination.Does anyone have thoughts on how to resolve or even troubleshoot this? Would using TLS help?

05
B
L
Laurent ZannettacciNew Participant
asked in Stream
Clibl_route field added but not configured in the system fields destination settings

Hello,I noticed yesterday that the cribl_route system field is always added to all events even if deactivated in the system fields post processing settings of my destination (Splunk).I am running a Stream cloud instance since more than a year and never had this issue before.Anyone else has the issue ?

01
T
G
Grant ForresterNew Participant
asked in Stream
Recommended Config for Persistent Queuing with Kubernetes Deployment?

Hello,We are exploring running a self-hosted Cribl deployment on Kubernetes (EKS+Fargate) using the provided Cribl Helm charts. We've got a leader and worker pod running. We are now looking at persistent queuing.My question is: what's the recommended configuration for ensuring no data loss in the case of a worker pod terminating with events still in the persistent queue?We've been exploring the use of extraVolumeMounts and persistent volumes backed backed by EFS. We found this technique in this blog post (https://cribl.io/blog/persistent-queueing-using-the-logstream-workergroup-chart/) but we see it is not recommended in the the docs (https://docs.cribl.io/stream/deploy-kubernetes#using-persistent-storage-for-persistent-queueing). We were wondering if there are better solutions that allow for a more automated recovery of events sitting in the persistent queue of the lost pod?

01
T
P
Pawel KwiatkowskiInspiring
asked in Stream
An import/export of code to a yml format

An export of code is done to a json fromat only while on servers is used a yml fromat.IMHO, it should be possible to do import/export also in yml format. :)

01
T
W
Warner ChengNew Participant
asked in Stream
Hi Cribl Streamer,

From the Data Routes, Apache has a yellow triangle. The Pipeline is Apache_Access.I tried other pipelines and it did not make a difference. Any pointer for me is appreciated.

03
W
W
Warner ChengNew Participant
asked in Stream
What are the steps to create worker group via CLI?

What are the steps to create worker group via CLI?

03
W
S
Shane DanielsEmployee
asked in Stream
How many worker groups can a Cribl Stream leader support?

How many worker groups can a Cribl Stream leader support?

01
B
J
Johan WogerInspiring
asked in Stream
How can I download a list of worker nodes in JSON format?

I have a big deployment with hundreds of worker nodes. Is there an easy way to download a full list of worker nodes in json format?

03
B
J
John PondromParticipating Frequently
asked in Stream
How can I prevent my syslog-ng server from pinning to a single stream worker process

When using syslog-ng server my syslog data is 'pinning' to a single stream worker process when sending syslog over TCP. Am sending too much data for that single WP to handle?

04
M
S
Steve BridgeKnown Participant
asked in Stream
Processing Syslog messages and sending to Splunk

Hi all,I want to do a drop-in replacement for an existing syslog-ng + Splunk HF system. Currently, it receives a variety of logs via syslog, syslog-ng writes them to disk (using different paths per sourcetype), and then the Splunk HF reads them, applies index-time transforms based on sourcetype, and forwards to the indexing tier. I can see a variety of ways to approach this, so I’m looking for a best-practice recommendation. How would you set this up so that some event sources can be processed with Cribl, but others can be left to the Splunk HF/Indexer? In other words, I’d like to make use of things like the PAN pack, but also be able to have it just work the same as before with some outlier sources, which means passing to the HF to process props/transforms on the data.

05
M
M
Michael HockeNew Participant
asked in Stream
How long can a worker node run without a leader node?

I am in charge of documenting our disaster recovery and high-availability efforts. One of the questions that came up is whether the worker nodes will cease to function (or reduce functionality) when the leader node disappears for a prolonged time. If so, how much time do we have until we must resurrect the leader?

03
B
K
Kyle McCririeEmployee
asked in Stream
How do I change a Cribl workers Host name?

In the Workers tab, the second column, "Host", shows the Cribl worker node name. Currently, I have some hosts with a hostname, and other hosts with hostname + domain name.How can I change the hostname to be consistent here?

01
R
T
Tony REmployee
asked in Stream
Is there a simple way to do if-elif-elif in an eval statement? Support 3 or 4 tests

Support 3 or 4 tests

01
D
J
Jordan PerksParticipating Frequently
asked in Stream
Does Stream support receiving Syslog that uses Octet-Counting Framing?

Does Stream support receiving Syslog that uses Octet-Counting Framing? For example, from the default configuration in Corelight?

02
M
P
Pawel KwiatkowskiInspiring
asked in Stream
[FEATURE] A filter in a group

Hi,IMHO, a group of functions or routes should contains a filter also.What you think?

02
T
M
Michael DonnellyInspiring
asked in Stream
Modifying data format, on a per-destination basis

A customer asked this: We have multiple syslog senders routing their messages to Cribl. Each of the senders has an event format that's specific to the source. And, we are routing those events to multiple destinations which also require their own format, such as CEF format for our security tool, JSON format for our logging tool.What's the best way to handle the processing of these events, so they're correctly processed and formatted correctly for the destination?Here is the example scenario:We have devices sending via syslog 514 ie (f5s, Cisco ASA, Cisco routers, cisco switches, storage gear, Palos, etc.) All the syslog messages would be different based on the vendor. Some (like the PAN) have a single device sending multiple different formats.We're doing filtering, enrichment, reformatting on the events. We're prioritizing this effort based on volumes.Once processed, events are sent to different destinations. Lets say we have 3: one requires JSON, one CEF, and one requires a single

04
M
L
Log GoblinNew Participant
asked in Stream
How to authenticate to Sentinel destination

Hello,Currently experiencing an issue when trying to configure the authentication section of the sentinel destination.Have been following the Cribl guides but it is not clear on what should be entered into the Login URL, OAuth Secret and Client ID.I am guessing the Login URL can be found from the App Registration in Azure and clicking on Endpoints?Client ID of the App Registration is not accepted in the Cribl UI.Any suggestions?Thanks :)

01
T
N
Nick Kwiecien Nick KwiecienKnown Participant
asked in Stream
How do you reference a scheduled collector inside a route?

How do you reference a scheduled collector inside a route?

01
R
T
Tony REmployee
asked in Stream
Is it safe to remove Time from _raw?

Can I remove the time stamp from Raw events and keep only the remaining message for raw?

01
D
D
Draco3New Participant
asked in Stream
Separate array into individual events

I have a JSON file that I am ingesting into CRIBL. I have created a pipeline with the following functions “JSON Unroll” which is then followed by the “Parser” function with the following settings (Operation mode = extract; Type=JSON Object, Source Field = _raw).My single event now has a new field created called “records:” with what appears to be multiple arrays for each record (there are in excess of 250) with thisrecords:{}0:a account_id: xxxxxxxxxxa account_name: xxxxxxxxa owner: xxxxxxxxxa cost: xxxxxxxxxxa group: xxxxxxxxxxxxa reporter: xxxxxxxxxxxxx{}1:a account_id: yyyyyyyyyyya account_name: yyyyyyyyyyya owner: yyyyyyyyyyya cost: yyyyyyyyyyya group: yyyyyyyyyyya reporter: yyyyyyyyyyy{}2:a account_id: zzzzzzzzzzzzza account_name: zzzzzzzzzzzzza owner: zzzzzzzzzzzzza cost: zzzzzzzzzzzzza group: zzzzzzzzzzzzza reporter: zzzzzzzzzzzzzetc…I am running CRIBL version 3.5.1. I am trying to separate each of these records into an individual event. So in my example above {}0 with its 6 f

02
A
S
Shawn CannonKnown Participant
asked in Stream
What the best way to convert a sample data file into a datagen?

What the best way to convert a sample data file into a datagen?

01
R
P
Pawel KwiatkowskiInspiring
asked in Stream
Support for SNMPv3

In the https://docs.cribl.io/stream/sources-snmp-traps/ I don't see description regarding to the SNMPv3. So I have question if the Cribl support the SNMPv3 integrations?

03
B
V
VINOTH RAJ T NNew Participant
asked in Stream
Is there an option in Cribl Stream to store the data in local storage?

Hi guys,I am new to Cribl and I was wondering whether there is an option in Cribl to store the data in a local storage.So, if there is a way, a clear step by step procedure with the prerequisites would be really helpful.Also, a little clarity on using the "replay function" along with this local storage would be nice.Thanks in advance.

00
X
T
Tony REmployee
asked in Stream
Provider only is able to send webhooks to 80/443/8443

gonna sound funny but a Popular vendor in our telecom world does support sending webhook notifications for alarms/alerts (but the problem it only sends to 80 / 443 / 8443 (no custom port) i dont think cribl cloud supports these ports (but is there a online proxy like ngrok or something that i can send/relay to cribl cloud (i dont wanna cloud ---> punch hole through FW to allow 443/8443 and then back to criblCloud at least in this simple POC approachOriginal Thread on https://cribl-community.slack.com/archives/C022FME74PM/p1695069411755299

07
J
V
Valeriu SargheNew Participant
asked in Stream
Unable to use multiple certificates and passphrases for Splunk TCP Source

Hello,In a Cribl Distributed Deployment, with one Worker Group, and 2 Worker Nodes in this Worker Group, with a Source Splunk TCP enabled, and TLS enabled from Leader Node, It is not possible to use a unique certificate and passphrase for each Worker Node.It is possible, to have each Worker Node in a different Worker Group. But that means all the settings must be the same (Sources, Destination, Routes, Pipelines). That means in the end that the use of Leader Node is useless.Is that correct?Can we improve that? Or make an enhancement?Thank you!Best regards,Vali

06
J

Badge winners

Show all badges
Terms & ConditionsCookie settings