Products

Which Config file stores the configured git branch? I don't see it in cribl.yml

I have a S3 Source configured, to collect Parquet Files, but I end up getting the following error:{ "time": "2023-03-31T00:00:00.000Z", "cid": "w1", "channel": "input:my_parquet_source", "level": "error", "message": "failed to process file", "file": "s3://super-awesome-s3bucket/folder/year=2023/month=03/day=30/hour=00/part-0abcde12-456f-78a9-b01c-de23f4a56b0c.c000.parquet", "receiveCount": 1, "messageId": "0abcde12-456f-78a9-b01c-de23f4a56b0c", "error": { "message": "Error while reading file.", "stack": "Error: malloc of size 65536 failed" }}How can I address this?

When configuring the WEF source, Stream shows the source as working, but the Windows Event Forwarding plugin event logs show: Error Event ID: 107 A subscription policy contains invalid configuration.How do I overcome this error?

I am trying to create multiple metrics events from a single log event. In my log event, I have six metrics and seven dimensions. I would like each metric to output to Prometheus as its own event with all seven dimensions. How can a achieve this?

o365 message trace input: changed from basic auth to oauth and the input fails now with this error{ "time": "2023-04-10T23:00:12.531Z", "cid": "w1", "channel": "TaskExecutor", "level": "error", "message": "failed to execute task", "jobId": "1681167600.54.scheduled.CD_o365_msg_traces:MessageTrace", "taskId": "collect.1", "host": "hostname", "reason": { "message": "Non-whitespace before {[.\nLine: 3\nColumn: 1\nChar: 60", "stack": "Error: Non-whitespace before {[.\nLine: 3\nColumn: 1\nChar: 60\n at v (/opt/cribl_w/bin/cribl.js:14:9579357)\n at p._ [as write] (/opt/cribl_w/bin/cribl.js:14:9580223)\n at y.write (/opt/cribl_w/bin/cribl.js:14:17538090)\n at S._transform (/opt/cribl_w/bin/cribl.js:14:17539431)\n at S.Transform._write (node:internal/streams/transform:205:23)\n at writeOrBuffer (node:internal/streams/writable:391:12)\n at _write (node:internal/streams/writable:332:10)\n

My leader UI will randomly go down for roughly 10 minutes and when we look at the logs there are no errors that would cause the UI to go down for a period of time. Is there anything that conflicts with the UI’s processes that would cause the UI to go down for a period of time? We would like any guidance on how we would go about figuring out why the UI is none responsive.

Hi,We’re running Cribl in AWS, using ECS to manage docker containers on EC2 instances. We’re trying to tune our instance size and auto-scaling, and wondering if anyone else has any past experience on the matter.For example, should we use memory reservations, and if so what are the best settings for soft and hard limits?If we want to scale when under memory pressure, how do we get this information out?I.e. to know when we’re under memory pressure we need to see heap usage of the worker threads, not just memory usage for the box (since the threads pre-allocate heap), what’s the best way to get this?If anyone has anyone had any luck - good or bad - with this, I’d be keen to hear about your experiences!

I opted to have “Authenticate client (mutual auth)” turned on, which is under Distributed Settings > TLS Settings on the leader, and I also read to make sure to put your certs outside $CRIBL_HOME due to the issue of every time a new config version would be pushed it would overwrite the directory and would remove the certs on the workers. So I placed the certs outside $CRIBL_HOME, and on each worker I had distributed and API TLS (each worker has their own cert) enabled and pointing to the same cert. When I deployed a new config version to the workers, the distributed TLS cert would stay applied, but the API TLS cert would remove itself. This issue doesn’t allow me to access each worker independently via there DNS name using TLS/HTTPS. Is there a way to use the same TLS cert for both API and distributed? So that when a new config is deployed they both stay applied.

Hi, I am not seeing data come in on a Syslog Source I am setting up. But I do see the syslog data coming into the Worker with a TCP Dump. The worker OS is RHEL 9.

does cirbl support MDT cisco yet Yang/netconf collection Original message in Cribl Community Slack Message:https://cribl-community.slack.com/archives/C02A9EQV61J/p1680730994732489

Hi,I've added some fields (lookups, etc) from a syslog source and trying to push it into the Log Insight API via a webhook.But, I'm struggling understand how to get it into the format Log Insight wants.I'll say up front, JSON and Javascript is pretty new to me so be gentle 🙂The JSON format Log Insight expects is…{“messages”: [{“fields”: [{“name”: “Field1”, “content”: “Field1_value”},{“name”: “Field2”, “content”: “Field2_value”},{“name”: “Field_xxx”, “content”: “Field_Last_xxx”}],“text”: “original message”,“timestamp”: timestamp}]}I've tried the Serialize to JSON, but it doesn't seem to have a way to arrange the fields for it to output in a custom way.I've seen a post about using Object.fromEntries, but I'm still not able to get working. Not sure if it's even the right way to go.Converting an array into a simplified JSON object:

I was a bit careless and i pointed my stream deployment to a remote repo that had a bunch of old commits. Now my leader pulled down those old changes and they were deployed to my worker nodes. I reverted back from the remote repo but still have no access to my old commits which should still be living on the leaders local git. How do i fix this?

Hey, if I have an event that is coming into Crible Stream, with format of “<?xml version="1.0" encoding="UTF-8"?>” how would I convert the event to JSON?I see the use of C.Text.parseXml(_raw, false), but that doesnt seem to parse it properly for me. Maybe I’m doing it wrong, but it doesnt seem to get parsed. How would I do that? Tools such as NiFi can convert from XML to JSON easily and quickly, but this seems a bit harder.

I am running a distributed environment, and when I use the API Reference to query an endpoint it returns Status Code 404. How can I resolve this?

I have created an event breaker rule that works in the knowledge area of cribl stream. But when I run a filesystem collection job to pick up the same file I used to create the event breaker, it does not work.Event breaker rules in

Hello, this is probably simple, but I can’t get it to work. I just want to remove IPV6 encapsulation from the host field of events arriving at Cribl via syslog. So, existing host field is ::ffff:10.1.1.1, I want it to be just 10.1.1.1. For some reason I can’t get this to work, Splunk shows a null value for host when I try to EVAL, MASK, REGEX, etc. Anyone have a way to do this?

I am trying to send the count or average of certain fields instead of the whole events for a specific time period.

I’ve cloned user, and am now trying to add more GET polices. However, looks like ‘GET’ /system/something, ‘POST’ etc can’t be added as NewPolicy - only GroupFull etc available.Can this be done in the UI?or only using POST /system/roles?or do I need to post new Policy , POST /system/policies?

I am seeing some irregularities with collecting large files from a filesystem.We batch process files anywhere from 100MB to 100GB in size. I am currently noticing an issue with larger files.To troubleshoot I created a collector that reads the data in, and directly writes it back to disk. No other ETL is done on the data.Result of 1 collectionResult of collecting the same file from the same location using the same event breakers and going to the same destination.My current environment is 1 leader, 1 worker, and the file is being picked up from an NFS mount.As you can see from the screenshots, only 3-4 million of the events are being collected of the ~24 million events in the file. The destination is writing to disk about 5-6 GB from the original ~38GB.There are no errors that I see in the job log, and I can't find any setting regarding worker process limits or job limits that would affect this.

Given a key-value formatted source, routed through a pipeline where data is parsed into JSON then manipulatued, how do I write the new data back out in KV format? Using the filesystem Destination, I’m getting JSON after my pipeline regardless of whether I use Raw or JSON output. I’m assuming there’s a function I’m missing that does this. The closest thing I can find is Serialize, but that doesn’t support KV format.

When I look at Monitoring > Logs, I can’t find logs from scheduled collection jobs using channel.includes(input_name)

Hello,i need some input.We want to have a separate dataflow between different stages within cribl.So we receive data in our prod environment and want to have an extra dataflow to QS stage, to test parsing changes etc.One way we know is to configure a second output wihtin “Output Router”Are there any other ways? Maybe within routes or pipelines?Thank you for your input

FYI, I was unable to sign up to the site via the normal account sign up.Never got any confirmation emails, tried multiple email accounts (they are were all hosted with google if that has a bearing on anything)Had to use “OpenID Connect” and sign up with Google

Has anyone connected up an AWS Opensearch Service instance to Cribl? I created a local backend user in my domain and gave it what I thought were good permissions but I still get a 401 error when I test the connection. Can anyone share what they did to get this to work?

I’m writing a custom function and I’m able to add it to a pipeline. However, when I try to process an event, I get an error.Here’s the content of my index.js:exports.name = ‘MyCustomFunction’;exports.group = ‘Custom Functions’;exports.version = 0.1;exports.init = (opts) => {const conf = opts.conf || {};};exports.process = (event) => {return event; // noop};I’m unable to find where, if any, error messages are written to the logs. I looked at the logs under Monitoring (Logs tab) and docker logs with no success.Are errors of this type written anywhere?