Stream

Hello Cribl gurus - could you square https://cribl.io/blog/how-to-architect-your-logstream-to-logstream-data-flows/ with https://docs.cribl.io/stream/destinations-cribl-tcp/#reqs & https://docs.cribl.io/stream/destinations-cribl-http#reqs? We have an on-prem distributed deployment and a Cloud hybrid deployment, and we would like to forward data from one source from the on-prem workers to a hybrid Cloud deployment to continue our Cribl PS engagement, but it’s unclear if this is possible and if so how to achieve it. To be clear, our on-prem and cloud instances/deployments’ have different tokens. Please advise what options, if any, we have to connect the 2 instances together in terms of the on-prem source of interest.

Trying to implement a new data source but getting the error "Error 404-Not Found"Also getting an error saying "Failed to send anonymized telemetry metadata. Data flow has been stopped."

Hello,After upgrading my forwarders to the latest version of 9.1, I am no longer able to send to Cribl Stream. I am using the outputs.conf referenced here to send to my on-prem Stream instances:[tcpout]disabled = falsedefaultGroup = cribl[tcpout:cribl]server = STREAM_IP:PORTsendCookedData = trueforceTimebasedAutoLB = falsenegotiateProtocolLevel = 0However, no data is making it through. When I look in the Log section of my Source, I do see this error message:Any idea what I can do for this issue?

I have changed the auth token on my leader's distributed settings page and the workers in my worker group can no longer connect. what is the proper way to update the worker node's settings to reflect the new auth key?

Best way to extract fields from this data where the the field and value are separated by a : and the group is delimited by a , ?{"action": "update", "body": "New token value generated", "name": "user1", "timestamp": "2023-08-24T19:33:22+00:00", "userID": "1"}

Currently I'm ingesting Windows Classic and Windows Sysmon XML using Splunk UF, and apply corresponding packs to convert to JSON and some custom modifications. Until few days everything worked, but now we are receiving empty or broken messages.I'm running Splunk UFs 9.0.4/Splunk Cloud 9.0.2209.4 and Cribl 4.2.2.Config on UFs, as recommended in the docs. On Cribl Worker s2s is v4.[tcpout] disabled = false defaultGroup = cribl-worker-1[tcpout:cribl-worker-1] sendCookedData = true server=x.x.x.x:9997I noticed that Cribl defaults to fallback event breaker. No error/warnings in Cribl logs.

I can't get any of the data destinations to work. I have tried this with ElasticSearch, AWS S3 and Splunk. The message I get when I try a Test payload is "Error: 400-Bad Request Output … does not exist!"The logs don't have any information either.I'm running stream in a docker container. I got the same results on Cribl cloud instance as well.

Can Cribl Stream read AWS S3 tags? I know there are multiple places where you can add tags to parts of the stream, such as in the Source or in a Pipeline, but I want my stream to be able to read the tags that are assigned via S3 in order to filter them. If that's not possible, would I be able to call a Lambda function and use that to manage the tags instead?

We don't want our Cloud org to be removed, can that action be undone? We don't want to lose our configurations or any of the data being processed.

Hello, I’m trying to creation a S3 destination and trying to figure out how I can add index value to the file name.  How do I pass the index field from the source to the S3 destination file name?

hello, i’m trying to create a sql server db connection, but got a self signed certificate error. Is there an option that I can use to bypass this issue?  I’m using the following format from the documentation: Server=localhost; Database=test;User Id=SA; Password=Testing123![; Option=Value]Original Author: Tae HaLink: https://cribl-community.slack.com/archives/CPYBPK65V/p1690207040810899?thread_ts=1690207040.810899&cid=CPYBPK65V

Dear Cribl Community,Is there a way to paginate the DISCOVER (HTTP Request) stage results when using the REST API collector. The COLLECT stage has pagination parameters, but I can't figure out how to paginate discover results. Thanks!Regards,Justas

When a worker nodes fail (down for whatever reason), what happens to the data that the specific worker had received for processing ( like, in-memory , PQ data ) ? Worker nodes receive data in distributed manner for processing , how does it ensures that data delivered to any third party destination (e.g. splunk ) is the chronological order of the events ?e.g. in case of hardware issues / slowness if a worker node is experiencing lag in processing data ., can it be expected that data send to any output destination has latency ?

Is there a configurable setting within Stream to alert or notify when a particular source stops sending events for a given period of time? Thanks

Hi does the Cribl leader or worker need to have access to the internet to download the packs?Original Question: https://cribl-community.slack.com/archives/C021UP7ETM3/p1688612591923099

I have bean configured source Windows Event Forwarder with certificate authorization. When my workstations sending events to cribl i had error "request error - processing error"

I've been up and down all through the documentation and even NXLogs at https://docs.nxlog.co/refman/current/im/wseventing.html but I still can't seem to get the logs in. I can send just find to a Windows WEC without a certs so I know it's cert related.Does anyone have a guide or tips when working with Windows Events in a Domain and Certs?

E.g. I would like use subshell like $(date). Is it possible?

Just curious. I find the lack of support and community to make it harder to work with Cribl compared to other services.

When a Google chronicle destination is configured, below errors are commonly seen? How to fix them?"hit request concurrency limit""Request payload size exceeds the limit: 1048576 bytes"

Hi,is it possible to define how much space should be left free for a filesystem destination?Kind regards

Currently, I have the Sentinel Agent installed on the windows machines. Would the best solution be to install edge on the all the window machines and send to stream or forward all the logs to a central windows machine and then send from there with Edge?

{ "_raw": "{"Id":4634,"Version":0,"Qualifiers":null,"Level":0,"Task":12545,"Opcode":0,"Keywords":-9214364837600034816,"RecordId":24675211,"ProviderName":"Microsoft-Windows-Security-Auditing","ProviderId":"54849625-5478-4994-a5ba-3e3b0328c30d","LogName":"Security","ProcessId":816,"ThreadId":2972,"MachineName":"vmwopsadm02-dev.hq.xxx.com","UserId":null,"TimeCreated":"\/Date(1686604325474)\/","ActivityId":null,"RelatedActivityId":null,"ContainerLog":"ForwardedEvents","MatchedQueryIds":[],"Bookmark":{},"LevelDisplayName":"Information","OpcodeDisplayName":"Info","TaskDisplayName":"Logoff","KeywordsDisplayNames":["Audit Success"],"Properties":[{"Value":"S-1-5-21-3567637-1906459281-1427260136-1830845"},{"Value":"VMWOPSADM03-DEV$"},{"Value":"xxx"},{"Value":"0xbc526b"},{"Value":"3"}],"Message":"An account was logged off.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3567637-1906459281-1427260136-1830845\r\n\tAccount Name:\t\tVMWOPSADM03-DEV$\r\n\tAccount Domain:\t\txxx\r\n\tLogon ID:\t

hello, I’m working on a dummy task. I have a lookup of 3 fields one of them is “card number” and it’s masked as shown in the attachment. I then have sample files that I need to enrich them with this lookup using the “card number” field as the joining field. The sample files have the “card number” field but umasked with normal digits. How can i enrich the sample files with the lookup when i don’t know the masking expression of the lookup field ?Can anyone help ?attached is the lookup file

I'm trying to figure out why every event from my pfsense firewall is being cloned as it passes through the pipeline. When I look at Preview Full from the Pipeline, I see the first event in my sample data file being processed as desired through the pipeline, then the second event is an unprocessed clone of the first. The third event is processed correctly by the pipeline, and the fourth event is an unprocessed clone, etc. When enabling the Advanced Setting to Show Internal Fields, I see a field called "__cloneCount". The value of this field in the correctly processed events is, "__cloneCount: 0", and the value for the unprocessed, cloned events is, "__cloneCount: 1". I've tried adding a Drop function to the end of the pipeline to drop events where, "__cloneCount == 1", but this has no effect. I've also tried to use Chain to call another pipeline to Drop the clone events, but this also has no effect. I though about adding a post-processing pipeline to the destination, but I'm s