Community Discussions

hello, I’m working on a dummy task. I have a lookup of 3 fields one of them is “card number” and it’s masked as shown in the attachment. I then have sample files that I need to enrich them with this lookup using the “card number” field as the joining field. The sample files have the “card number” field but umasked with normal digits. How can i enrich the sample files with the lookup when i don’t know the masking expression of the lookup field ?Can anyone help ?attached is the lookup file

I'm trying to figure out why every event from my pfsense firewall is being cloned as it passes through the pipeline. When I look at Preview Full from the Pipeline, I see the first event in my sample data file being processed as desired through the pipeline, then the second event is an unprocessed clone of the first. The third event is processed correctly by the pipeline, and the fourth event is an unprocessed clone, etc. When enabling the Advanced Setting to Show Internal Fields, I see a field called "__cloneCount". The value of this field in the correctly processed events is, "__cloneCount: 0", and the value for the unprocessed, cloned events is, "__cloneCount: 1". I've tried adding a Drop function to the end of the pipeline to drop events where, "__cloneCount == 1", but this has no effect. I've also tried to use Chain to call another pipeline to Drop the clone events, but this also has no effect. I though about adding a post-processing pipeline to the destination, but I'm sending to

Total Cribl noob. I've been asked to use Cribl to download a file at regular intervals, but I can't see how that is done. The source is not anything in our environment - it's an external entity. If I were to do it manually in a scheduled python job, it would look like this.oldest = (datetime.utcnow() - timedelta(hours=2)).strftime("%Y%m%d%H")file_name = f"{oldest}.tar.bz2" link = f"https://site.com/feeds/files/hourly/{oldest}"r = requests.get(link, headers={"x-apikey": apikey})file = BytesIO(r.content)I've been asked to download the file with Cribl, send it to s3, then extract it and ingest the contents into elastic. The thought is they expect more data sources in the future as I would probably consider it overkill, but I said I would give it a try and see. But not sure if it does that sort of thing. Thanks for any advice you can offer.

Hello - I am testing out cribl stream/edge - first time user. Using version 4.1.2.I've deployed a stream server, a single worker and a Edge node leader.I've installed the edge node agents using the "Add/Update Edge Node" automated script. I've done this on 5 servers in my environment to test how this would work. So far, I can see logs coming into Edge/Stream and into my siem with one exception. My domain controller is not capturing/sending any data. I have followed this: https://docs.cribl.io/stream/usecase-edge-stream as my guide for the setup. The other 4 servers are working as expected but the domain controller is not. I dug and found at the end of an article https://docs.cribl.io/edge/sources-windows-event-logs which states to check permissions on a registry key and make sure it has read rights; I checked that and still no data. I have checked the logs found under programdata\cribl\ and nothing jumps out.cribl is now accounting for 40GBs of space on that dc in the span of 3

I would like decode data from buffer 'data'.I was trying use Buffer.from but with partial success.Any ideas how should be done properly?

Have anybody tried to replay Splunk frozen buckets from AWS S3

I would like to move 15-20 REST collectors from one worker group to another.  For routes/pipelines, I just copy and then switch groups and paste but the copy function isn't available in REST collectors.   Any suggestions on how to move a REST collector?

I would like remove a field from event like using eval

Occasionally when looking at the Status of my Syslog Source, which has a configured TCP Port of 51111 : Error: Initialization error: bind EADDRINUSE 0.0.0.0:51111I don't have any other Sources or Destinations on the machine configured to leverage that port. I don't see there error on application startup either.

Which Config file stores the configured git branch? I don't see it in cribl.yml

I have a S3 Source configured, to collect Parquet Files, but I end up getting the following error:{ "time": "2023-03-31T00:00:00.000Z", "cid": "w1", "channel": "input:my_parquet_source", "level": "error", "message": "failed to process file", "file": "s3://super-awesome-s3bucket/folder/year=2023/month=03/day=30/hour=00/part-0abcde12-456f-78a9-b01c-de23f4a56b0c.c000.parquet", "receiveCount": 1, "messageId": "0abcde12-456f-78a9-b01c-de23f4a56b0c", "error": { "message": "Error while reading file.", "stack": "Error: malloc of size 65536 failed" }}How can I address this?

When configuring the WEF source, Stream shows the source as working, but the Windows Event Forwarding plugin event logs show: Error Event ID: 107 A subscription policy contains invalid configuration.How do I overcome this error?

I am trying to create multiple metrics events from a single log event. In my log event, I have six metrics and seven dimensions. I would like each metric to output to Prometheus as its own event with all seven dimensions. How can a achieve this?

o365 message trace input: changed from basic auth to oauth and the input fails now with this error{ "time": "2023-04-10T23:00:12.531Z", "cid": "w1", "channel": "TaskExecutor", "level": "error", "message": "failed to execute task", "jobId": "1681167600.54.scheduled.CD_o365_msg_traces:MessageTrace", "taskId": "collect.1", "host": "hostname", "reason": { "message": "Non-whitespace before {[.\nLine: 3\nColumn: 1\nChar: 60", "stack": "Error: Non-whitespace before {[.\nLine: 3\nColumn: 1\nChar: 60\n at v (/opt/cribl_w/bin/cribl.js:14:9579357)\n at p._ [as write] (/opt/cribl_w/bin/cribl.js:14:9580223)\n at y.write (/opt/cribl_w/bin/cribl.js:14:17538090)\n at S._transform (/opt/cribl_w/bin/cribl.js:14:17539431)\n at S.Transform._write (node:internal/streams/transform:205:23)\n at writeOrBuffer (node:internal/streams/writable:391:12)\n at _write (node:internal/streams/writable:332:10)\n at S.Writable.write (node:internal/streams/writable:336:10)\n at IncomingMessage.ondata (

My leader UI will randomly go down for roughly 10 minutes and when we look at the logs there are no errors that would cause the UI to go down for a period of time. Is there anything that conflicts with the UI’s processes that would cause the UI to go down for a period of time? We would like any guidance on how we would go about figuring out why the UI is none responsive.

Hi,We’re running Cribl in AWS, using ECS to manage docker containers on EC2 instances. We’re trying to tune our instance size and auto-scaling, and wondering if anyone else has any past experience on the matter.For example, should we use memory reservations, and if so what are the best settings for soft and hard limits?If we want to scale when under memory pressure, how do we get this information out?I.e. to know when we’re under memory pressure we need to see heap usage of the worker threads, not just memory usage for the box (since the threads pre-allocate heap), what’s the best way to get this?If anyone has anyone had any luck - good or bad - with this, I’d be keen to hear about your experiences!

I opted to have “Authenticate client (mutual auth)” turned on, which is under Distributed Settings > TLS Settings on the leader, and I also read to make sure to put your certs outside $CRIBL_HOME due to the issue of every time a new config version would be pushed it would overwrite the directory and would remove the certs on the workers. So I placed the certs outside $CRIBL_HOME, and on each worker I had distributed and API TLS (each worker has their own cert) enabled and pointing to the same cert. When I deployed a new config version to the workers, the distributed TLS cert would stay applied, but the API TLS cert would remove itself. This issue doesn’t allow me to access each worker independently via there DNS name using TLS/HTTPS. Is there a way to use the same TLS cert for both API and distributed? So that when a new config is deployed they both stay applied.

Hi, I am not seeing data come in on a Syslog Source I am setting up. But I do see the syslog data coming into the Worker with a TCP Dump. The worker OS is RHEL 9.

does cirbl support MDT cisco yet Yang/netconf collection Original message in Cribl Community Slack Message:https://cribl-community.slack.com/archives/C02A9EQV61J/p1680730994732489

Hello everyone,I’m looking for more Packs to work with into Cribl stream.I know the Pack Dispencary and Packs from Github. Are there more usefull sites to look through?Or are any more Packs planned for e.x. reducing logs for Checkpoint.Thank you and best regards

Hi,I've added some fields (lookups, etc) from a syslog source and trying to push it into the Log Insight API via a webhook.But, I'm struggling understand how to get it into the format Log Insight wants.I'll say up front, JSON and Javascript is pretty new to me so be gentle 🙂The JSON format Log Insight expects is…{“messages”: [{“fields”: [{“name”: “Field1”, “content”: “Field1_value”},{“name”: “Field2”, “content”: “Field2_value”},{“name”: “Field_xxx”, “content”: “Field_Last_xxx”}],“text”: “original message”,“timestamp”: timestamp}]}I've tried the Serialize to JSON, but it doesn't seem to have a way to arrange the fields for it to output in a custom way.I've seen a post about using Object.fromEntries, but I'm still not able to get working. Not sure if it's even the right way to go.Converting an array into a simplified JSON object: Does anyone have a slick way of converting an array where each element is an array of length 2, that contains the key at index 0 and value at index 1, into a

I was a bit careless and i pointed my stream deployment to a remote repo that had a bunch of old commits. Now my leader pulled down those old changes and they were deployed to my worker nodes. I reverted back from the remote repo but still have no access to my old commits which should still be living on the leaders local git. How do i fix this?

Hey, if I have an event that is coming into Crible Stream, with format of “<?xml version="1.0" encoding="UTF-8"?>” how would I convert the event to JSON?I see the use of C.Text.parseXml(_raw, false), but that doesnt seem to parse it properly for me. Maybe I’m doing it wrong, but it doesnt seem to get parsed. How would I do that? Tools such as NiFi can convert from XML to JSON easily and quickly, but this seems a bit harder.

I am running a distributed environment, and when I use the API Reference to query an endpoint it returns Status Code 404. How can I resolve this?

I have created an event breaker rule that works in the knowledge area of cribl stream. But when I run a filesystem collection job to pick up the same file I used to create the event breaker, it does not work.Event breaker rules in