Products

Is there any guide available for deploying Cribl standalone on a Azure Kubernetes Service?Original Question: https://cribl-community.slack.com/archives/CPYBPK65V/p1688623502905129

Hi does the Cribl leader or worker need to have access to the internet to download the packs?Original Question: https://cribl-community.slack.com/archives/C021UP7ETM3/p1688612591923099

I have bean configured source Windows Event Forwarder with certificate authorization. When my workstations sending events to cribl i had error "request error - processing error"

I've been up and down all through the documentation and even NXLogs at https://docs.nxlog.co/refman/current/im/wseventing.html but I still can't seem to get the logs in. I can send just find to a Windows WEC without a certs so I know it's cert related.Does anyone have a guide or tips when working with Windows Events in a Domain and Certs?

E.g. I would like use subshell like $(date). Is it possible?

Just curious. I find the lack of support and community to make it harder to work with Cribl compared to other services.

When a Google chronicle destination is configured, below errors are commonly seen? How to fix them?"hit request concurrency limit""Request payload size exceeds the limit: 1048576 bytes"

Hi,is it possible to define how much space should be left free for a filesystem destination?Kind regards

Currently, I have the Sentinel Agent installed on the windows machines. Would the best solution be to install edge on the all the window machines and send to stream or forward all the logs to a central windows machine and then send from there with Edge?

As a part of a cribl log processing pipeline, I want to add a function to remove any JSON keys that have empty strings or empty objects. I have a serialization function, such as this: - id: serde filter: "true" disabled: null conf: mode: reserialize type: json srcField: text fieldFilterExpr: value !=="" fields: [] remove: - .*but I don't want to reserialize, as it converts a JSON object during my pipeline into a JSON string. Is there another built-in function I can use to achieve this? Example input:{"my-log": {}, "my-other-log": "", "my-third-log": false} Expected output:{"my-third-log": false}

{ "_raw": "{"Id":4634,"Version":0,"Qualifiers":null,"Level":0,"Task":12545,"Opcode":0,"Keywords":-9214364837600034816,"RecordId":24675211,"ProviderName":"Microsoft-Windows-Security-Auditing","ProviderId":"54849625-5478-4994-a5ba-3e3b0328c30d","LogName":"Security","ProcessId":816,"ThreadId":2972,"MachineName":"vmwopsadm02-dev.hq.xxx.com","UserId":null,"TimeCreated":"\/Date(1686604325474)\/","ActivityId":null,"RelatedActivityId":null,"ContainerLog":"ForwardedEvents","MatchedQueryIds":[],"Bookmark":{},"LevelDisplayName":"Information","OpcodeDisplayName":"Info","TaskDisplayName":"Logoff","KeywordsDisplayNames":["Audit Success"],"Properties":[{"Value":"S-1-5-21-3567637-1906459281-1427260136-1830845"},{"Value":"VMWOPSADM03-DEV$"},{"Value":"xxx"},{"Value":"0xbc526b"},{"Value":"3"}],"Message":"An account was logged off.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3567637-1906459281-1427260136-1830845\r\n\tAccount Name:\t\tVMWOPSADM03-DEV$\r\n\tAccount Domain:\t\txxx\r\n\tLogon ID:\t\t0xBC526

hello, I’m working on a dummy task. I have a lookup of 3 fields one of them is “card number” and it’s masked as shown in the attachment. I then have sample files that I need to enrich them with this lookup using the “card number” field as the joining field. The sample files have the “card number” field but umasked with normal digits. How can i enrich the sample files with the lookup when i don’t know the masking expression of the lookup field ?Can anyone help ?attached is the lookup file

I'm trying to figure out why every event from my pfsense firewall is being cloned as it passes through the pipeline. When I look at Preview Full from the Pipeline, I see the first event in my sample data file being processed as desired through the pipeline, then the second event is an unprocessed clone of the first. The third event is processed correctly by the pipeline, and the fourth event is an unprocessed clone, etc. When enabling the Advanced Setting to Show Internal Fields, I see a field called "__cloneCount". The value of this field in the correctly processed events is, "__cloneCount: 0", and the value for the unprocessed, cloned events is, "__cloneCount: 1". I've tried adding a Drop function to the end of the pipeline to drop events where, "__cloneCount == 1", but this has no effect. I've also tried to use Chain to call another pipeline to Drop the clone events, but this also has no effect. I though about adding a post-processing pipeline to the destination, but I'm sending to

Total Cribl noob. I've been asked to use Cribl to download a file at regular intervals, but I can't see how that is done. The source is not anything in our environment - it's an external entity. If I were to do it manually in a scheduled python job, it would look like this.oldest = (datetime.utcnow() - timedelta(hours=2)).strftime("%Y%m%d%H")file_name = f"{oldest}.tar.bz2" link = f"https://site.com/feeds/files/hourly/{oldest}"r = requests.get(link, headers={"x-apikey": apikey})file = BytesIO(r.content)I've been asked to download the file with Cribl, send it to s3, then extract it and ingest the contents into elastic. The thought is they expect more data sources in the future as I would probably consider it overkill, but I said I would give it a try and see. But not sure if it does that sort of thing. Thanks for any advice you can offer.

Hello - I am testing out cribl stream/edge - first time user. Using version 4.1.2.I've deployed a stream server, a single worker and a Edge node leader.I've installed the edge node agents using the "Add/Update Edge Node" automated script. I've done this on 5 servers in my environment to test how this would work. So far, I can see logs coming into Edge/Stream and into my siem with one exception. My domain controller is not capturing/sending any data. I have followed this: https://docs.cribl.io/stream/usecase-edge-stream as my guide for the setup. The other 4 servers are working as expected but the domain controller is not. I dug and found at the end of an article https://docs.cribl.io/edge/sources-windows-event-logs which states to check permissions on a registry key and make sure it has read rights; I checked that and still no data. I have checked the logs found under programdata\cribl\ and nothing jumps out.cribl is now accounting for 40GBs of space on that dc in the span of 3

I would like decode data from buffer 'data'.I was trying use Buffer.from but with partial success.Any ideas how should be done properly?

Have anybody tried to replay Splunk frozen buckets from AWS S3

I would like to move 15-20 REST collectors from one worker group to another.  For routes/pipelines, I just copy and then switch groups and paste but the copy function isn't available in REST collectors.   Any suggestions on how to move a REST collector?

I would like remove a field from event like using eval

Occasionally when looking at the Status of my Syslog Source, which has a configured TCP Port of 51111 : Error: Initialization error: bind EADDRINUSE 0.0.0.0:51111I don't have any other Sources or Destinations on the machine configured to leverage that port. I don't see there error on application startup either.

Which Config file stores the configured git branch? I don't see it in cribl.yml

I have a S3 Source configured, to collect Parquet Files, but I end up getting the following error:{ "time": "2023-03-31T00:00:00.000Z", "cid": "w1", "channel": "input:my_parquet_source", "level": "error", "message": "failed to process file", "file": "s3://super-awesome-s3bucket/folder/year=2023/month=03/day=30/hour=00/part-0abcde12-456f-78a9-b01c-de23f4a56b0c.c000.parquet", "receiveCount": 1, "messageId": "0abcde12-456f-78a9-b01c-de23f4a56b0c", "error": { "message": "Error while reading file.", "stack": "Error: malloc of size 65536 failed" }}How can I address this?

When configuring the WEF source, Stream shows the source as working, but the Windows Event Forwarding plugin event logs show: Error Event ID: 107 A subscription policy contains invalid configuration.How do I overcome this error?

I am trying to create multiple metrics events from a single log event. In my log event, I have six metrics and seven dimensions. I would like each metric to output to Prometheus as its own event with all seven dimensions. How can a achieve this?

o365 message trace input: changed from basic auth to oauth and the input fails now with this error{ "time": "2023-04-10T23:00:12.531Z", "cid": "w1", "channel": "TaskExecutor", "level": "error", "message": "failed to execute task", "jobId": "1681167600.54.scheduled.CD_o365_msg_traces:MessageTrace", "taskId": "collect.1", "host": "hostname", "reason": { "message": "Non-whitespace before {[.\nLine: 3\nColumn: 1\nChar: 60", "stack": "Error: Non-whitespace before {[.\nLine: 3\nColumn: 1\nChar: 60\n at v (/opt/cribl_w/bin/cribl.js:14:9579357)\n at p._ [as write] (/opt/cribl_w/bin/cribl.js:14:9580223)\n at y.write (/opt/cribl_w/bin/cribl.js:14:17538090)\n at S._transform (/opt/cribl_w/bin/cribl.js:14:17539431)\n at S.Transform._write (node:internal/streams/transform:205:23)\n at writeOrBuffer (node:internal/streams/writable:391:12)\n at _write (node:internal/streams/writable:332:10)\n at S.Writable.write (node:internal/streams/writable:336:10)\n at IncomingMessage.ondata (