Skip to main content

Stream

Ask and answer questions about routing, shaping, and optimizing observability and security data.

  • 282 Topics
  • 601 Replies
282 Topics
P
Pawel KwiatkowskiInspiring
asked in Stream
SNMP source - SNMPv3 engine id

Why an engine id is not included into a configuration for decrypting SNMP traps?

01
K
J
Joel YueNew Participant
asked in Stream
Cribl on AWS Fargate

How do i calculate the amount of workers i need if i am using AWS Fargate arm processors? I am looking at 7tb/day input and output to 2 destination which is a total of 21tb/day throughput.Thanks

01
J
J
JasonRNew Participant
asked in Stream
Palo Alto pack host headers

I'm quite new to Cribl, so please forgive me if I've got some easy and/or dumb questions!The first thing the Palo Alto Pack pipelines all do is extract the host from an RFC3164 formatted syslog message with a BSD time format.Eval: host = _raw.match(/[A-Z][a-z]{2}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2}\s([^\s]+)\s/)[1] || hostOurs are sending in RFC5424 w/ Octet count framing (not syslog'y enough to know if that's RFC6587 or not), so the host extraction fails and ends up sending to splunk as the IP.With the desire to use the Palo Pack, what is the best way to handle this?Right now I created a pre-processing pipeline and attached it to my source. All it does is extract the host from the header. That way I don't have to handle it in every pipeline inside in the pack.Also, is there a reason you cannot connect a pre-processing pipeline to a source that's inside a pack? In this case, it would be nice to be able to package my paloalto_syslog_pre-processing pipeline inside our customized PaloAlto

03
J
C
Christian SchmidtNew Participant
asked in Stream
What value would Stream provide to someone who has logstash for routing/transforming?

So, this question has been bothering me for quite some time now. While I am a big fan of Cribl and I really enjoy working with their products and showing/explaining them to others I still wonder every now and then what value Stream would provide to a customer, who already has a well-maintained and functioning logstash for routing/transforming data.If I think about it the following points come to my mind, but if someone here has more/different reasons I would be glad to hear them!Stream's replay functionScalabilityLogstash works as a single instance, Cribl can be clustered to infinityStream is easier to maintainNo grepping around in config files for that one transformation command you need to changePipelines are easier to understand/maintain than logstash filesLess complex to get started with for new users/adminsVisualization of data flows"Debuggability" Being able to look into arriving/leaving data from within the tool without having to restart anything or using tcpdump is incredibly h

00
R
J
Joel YueNew Participant
asked in Stream
Ldap queries for Active Directory Info

Can cribl stream carry out ldap queries to Microsoft AD and then save the AD information in a lookup table or something similar? I want to automate fetching AD information from AD servers and then feeding it to a SIEM like Splunk or Elastic.

03
S
M
Mauricio ThomsenNew Participant
asked in Stream
Block timeout reached message when trying commit

Just upgraded a Stream cluster with 4.03. When performing the Commit to update the workers got "block timeout reached" message. Can be related to the amount of changes (195)? Is there any settings to change the timeout limit for updates?

02
M
S
Shawn CannonKnown Participant
asked in Stream
Assistance with a REST collector

I used a standalone test box to get my REST collector working like I wanted.  authentication, discover and collect worked great.  When I moved it into my distributed environment, it errors out but does pull in some events.  The discovery and collect phases are distrubuted across my 5 workers.  It seems like the authentication token may not be working properly when it runs on more than one worker?  I am guessing of course.  I get errors like 503 Service Temporaily Unavailable and 500 Internal Server Error.  This does not happen when I run it from my single standalone test server.  I may need some assistance trying to figure this one out.

02
J
R
Ronnie TirellaKnown Participant
asked in Stream
bind EADDRNOTAVAIL

I am getting this error in stream when trying to configure an IP address for a TCP source. I can't find any docs around this error.

020
R
P
peteNew Participant
asked in Stream
Office 365 Activity log counts less than Splunk app

For other users that have started pulling O365 activity logs using Stream instead of the Splunk app, have you noticed a discrepancy in the event counts? We consistently receive more events through the Splunk app than we do through the Stream source. The setup is exactly the same: tenant, app Id, content, and interval. We've even done comparisons into the data and have found events missing through Stream that were retrieved by the Splunk app. So it seems like Stream is either dropping some events or just not pulling all of them.We do have a case open with Cribl already but I'm wondering if we're the only ones seeing this. Has anyone else noticed compared the volume and noticed the same?

01
K
P
pallavi guptaNew Participant
asked in Stream
Cribl stream "the notifications service is not available because a configuration file doesn't exist

Hi All,I am new to Cribl and trying to set notification via webhook for destination have back pressure or destination is unhealthy. But i am getting below error:"the notifications service is not available because a configuration file doesn't exist or the settings are invalid."

02
J
J
Jade NgohoNew Participant
asked in Stream
Can we change the file ownership of Cribl Linux directory

We installed the Cribl on our Linux server, and by default, it is user=cribl group=cribl.Our organization requires us to us a Service account to run configuration inside it.Question :Can we change the user and group ownership of the Linux directory of Cribl?And would it affect anything ?From user=cribl group=cribl.To user=svcacc_cribl group=svcacc_cribl

02
J
D
deon.joubertNew Participant
asked in Stream
How are secrets distributed to worker in a group

I am currently setting an environmental variable in a Script collector using a JS Expression as follows: TEST_VAR = C.Secret("test1").valueThis works fine for some secrets, but not for others. In the latter case, TEST_VAR is equal to 'C.Secret("test1").value' instead of the actual value.I get the impression that the secrets for which it does not work have not yet been distributed to the workers in the group. Is this correct? Or is there something else wrong here.

01
K
R
Ronnie TirellaKnown Participant
asked in Stream
Ideal way to handle windows logs

With Cribl Stream what is the best practice for receiving windows logs?

04
R
J
Jason RehmNew Participant
asked in Stream
Ephemeral storage for entire worker config & PQ?

We're using AWS's c5d instance types which include NVMe instance storage for CRIBL_VOLUME_DIR, but we're also paying for a small EBS volume to host the configs in /opt/cribl.Is there a compelling reason to not just put CRIBL_HOME in the NVMe storage as well and have it pull down the config on startup?I came to this question because I was thinking about a scenario where we might shut down instances during low traffic times and how to handle being sure the CRIBL_VOLUME_DIR on the Ephemeral storage was formatted and mounted. It made me think, "If CRIBL_HOME was also there, then I wouldn't have to worry about CRIBL starting up without CRIBL_VOLUME_DIR being formatted and mounted".

01
J
T
Tike AweNew Participant
asked in Stream
Adding Splunk Metadata to events

I have an Event with no Splunk Metadata value within the events and i am trying to add new fields such as host, index,source and sourcetype . custom fields and i tried using an eval but no luck, how do i go about this ?

02
P
S
Swapnamay HalderNew Participant
asked in Stream
Data collected using REST Collector getting appended in a single file, how to resolve?

TLDR : JSON Collected from API, parsed through Pipeline, should have three JSON files in S3, but have two where one file have 2 JSON object appended, need help to find out the reasonHi All,At first, I want to say, no this is not an event breaker issue, as much as my understanding. Let me explain.I am trying to get some data from a REST Api endpoint using REST Collector. I am using discovery with Item list, and in the list, there are three values. In preview, three different events are also getting registered (sometimes, sometimes nothing is coming to the preview also).Now I am parsing this response using pipeline and putting the data in a S3 bucket using the router. I followed the documentation to configure the Destination as well as the pipeline and route.Now, I am setting up the schedular to run the REST Collector. Then, after each collection event, when I am checking my s3, there instead of having three separate JSON files for three separate events, there are two JSON files, one wit

06
S
P
Pawel KwiatkowskiInspiring
asked in Stream
Is the Cribl supporting SNMPv3?

Is the Cribl supporting SNMPv3?

06
P
T
Tony REmployee
asked in Stream
Import this field list and throw away all fields in the data that are not contained in the list

Hi everyone, I have a use case atm where I have a lookup that contains a column of "allowed fields" for various sourcetypes, that is, the value for a given sourcetype is a common separted list of field names. At the end of my pipeline, I basically want to import this field list and throw away all fields in the data that are not contained in the list (because I'm finished with the normalization). Is there an easy way to do this? The serialize function has essentially the same functionality, when you import the parser you get a keep fields list. But how to do this without the parser? Original message in Cribl Community Slack Message: https://cribl-community.slack.com/archives/CPYBPK65V/p1682331204917309

01
R
B
Brandon McCombsEmployee
asked in Stream
Splunk destination health not matching individual worker node health in destination

Why is my Splunk destination showing unhealthy when the individual workers are showing a green healthy status?

03
B
J
John PadillaNew Participant
asked in Stream
How do I change a worker's temp directory from "/tmp"?

How do I change a worker's temp directory from "/tmp"?

02
J
B
Brian YearwoodInspiring
asked in Stream
Why is my parsed data different from my original _raw data?

When I use a parser function to parse my json _raw field, I see that my original data changes and it breaks my dashboards:Why does it change?

01
R
O
Oliver HoppeEmployee
asked in Stream
How can I connect my Bitbucket repository with Cribl?

How to connect my Bitbucket repository with Cribl?Question: When trying to push to a remote git repository created in Bitbucket under a project, I am getting the following error:This is working for other repositories created under a specific user, but not under a project in bitbucket.

02
O
J
Jessica EdensNew Participant
asked in Stream
Locked out of Stream, how do I reset my password?

Hi,I recently installed Cribl and today when I tried to log back in I couldn't, I'm locked out! I have CLI access so I was hoping there was a way to reset my password through the CLI?Thank you!

00
A
I
Ihor SerohinNew Participant
asked in Stream
File Monitor UTF-16BE

Hi.I need to continuously read events from log file which is UTF-16 encoded. Is any way to convert collected data to UTF-8?Before that I used File System Collector which could use "Custom Command" option. Using "/usr/bin/iconv -f UTF16BE -t UTF8" solves the problem. But there is no such option for "File Monitor" source.

01
K
D
Dinesh RajaNew Participant
asked in Stream
Collect and Send S3 logs via Cribl to Splunk

Hello All,I'm new to Cribl and basically a Splunk Admin & developer. Been working on Cribl migration project for a while.My requirement is to collect data from a S3 bucket Via Cribl and apply some cool stuffs than send it to Splunk for indexing.Now, I have established the connection with my S3 bucket from Cribl stream. I'm looking for some assistance to optimize & send data to Splunk.Can anyone guide me the steps & tricks for same?TIA.

01
T

Badge winners

Show all badges
Terms & ConditionsCookie settings