Stream

After being processed by Stream, is there a way have those source log files either be moved, renamed or deleted?

I have this really simple function enabled to configure current time as timestamp, but it doesn‚'t seem to work. it's still taking timestamp from event.

Is it possible to stream logs and events to Cribl Stream, send a copy of that event to S3 (or similar) for archive, and then process/parse/enrich the logs before sending them to another destination? The use cases I’m thinking of are as follows:(Primary Use Case) If a security incident occurs and we need to present evidence in court, how do we protect the forensic integrity of an event/log/etc. if the event is being changed in flight before going to an archive or analysis tool?If we needed to replay a series of logs for some reason or another, it might be helpful to do so from the source events, unmodified by Cribl.The downstream, long-term storage solution may also serve as a repository of data for other teams to use. Providing unmodified events for those teams may be preferred in that case.Thanks!

I’m changing the ip on a couple of my workers - anything that I need to change or be aware of? My assumption is that Stream/Edge will just pickup the change and it will appear on the leader with those changes. Nothing else is changing.

Is there a URL or API endpoint I can use to check the status of a Stream Worker or Leader? If so, what HTTP Status codes does the endpoint return?

Hi all. I have Cribl replacing syslog-ng, but in order to listen on 514 I needed to run it as root. Is there a work-around to drop privileges or switch to user cribl after start-up?

Is there a way that we can search data in Devo using the API?

I am getting the following error in Splunk when sending an event from Stream:04-20-2023 04:21:09.000 +0000 INFO IndexWriter [7972 indexerPipe] - idx=goat: found event timestamped=1681964469123 is out of range. Using 1680307200.Here’s what _time looks like, in Stream, when viewing the data:{“_raw”: “”,“_time”: 1681964469123,“cribl_breaker”: “”,“index”: “”,“source”: “”,“sourcetype”: “”,“cribl_pipe”: “”}

I have a REST collector discovering events but not collecting them when scheduled. It works Ad-hoc, though.Any idea what could be causing this? I do not see errors in the logs.

What's the best way to move a sample data capture into a pack? Seems that exporting it isn't consistent.

I have a stream instance hosted locally and have several machines hosted externally. Is the ideal solution to install edge on all the external machines and then forward to an edge on one of the machines with a VPS connection that can then forward everything to stream?

As we know, we have the default route where there goes any data that was not taken by previous routes.Since that route has a filter true how can I find what data is going to that route? Capture is not helping of course because it has filtered trueAny other method except for negating all filters from previous routes?Original Question: https://cribl-community.slack.com/archives/CPYBPK65V/p1688625589301539

Double checking on this, what is the expected behavior of Cribl if the Redis cluster is unavailable for some time? To my understanding events skip the Redis function (after default 30seconds) and processes through the rest of the pipeline. Is that how it is?

How do I write a database query for a DB Collector that uses state-tracking for MS SQL?

Hi,I recently installed Cribl and today when I tried to log back in I couldn't, I'm locked out! I have CLI access so I was hoping there was a way to reset my password through the CLI?Thank you!

So, this question has been bothering me for quite some time now. While I am a big fan of Cribl and I really enjoy working with their products and showing/explaining them to others I still wonder every now and then what value Stream would provide to a customer, who already has a well-maintained and functioning logstash for routing/transforming data.If I think about it the following points come to my mind, but if someone here has more/different reasons I would be glad to hear them!Stream's replay functionScalabilityLogstash works as a single instance, Cribl can be clustered to infinityStream is easier to maintainNo grepping around in config files for that one transformation command you need to changePipelines are easier to understand/maintain than logstash filesLess complex to get started with for new users/adminsVisualization of data flows"Debuggability" Being able to

When we check monitoring/sources we see metrics for the average thruput for both Events Per Second and Bytes Per Second for the sourcesHowever when we check monitoring/routes we see metrics for the average thruput for only Events Per Second and and NOT Bytes Per Second for the sources

Here at Cribl we are committed to making our customers' lives easier. One of the requests we had was how we can quickly ship new versions of our products into the Amazon Marketplace for customers who cannot use our SaaS offering Cribl.Cloud. Today we are announcing the availability of Cribl Amazon Machine Images or AMI’s for both x86_64 and ARM64  platforms. These two AMI’s can be used as part of a Free Tier for up-to 1TB/data per day. Customers can also elect to bring their own license and convert these images to paid Enterprise or Standard versions. This would also work for deploying hybrid Cribl worker nodes in our customer’s own AWS environment.The CloudFormation templates will still be available via our GitHub repo https://github.com/criblio/cribl-aws-cloudformation-templates .  Navigate to the AWS Marketplace and select the AMI archi

GOAT Mind, I am using a packet broker to aggregate my syslog data, then push it to Cribl Stream, which is running on Red Hat 9. I have had to spoof the MAC to the NIC and give it the accepting IP address that the Packet Broker is expecting to receive the data. Next I have had to modify kernel parameters so the data can be seen at the NIC.I can see the data at the NIC via tcpdump and ngrep. However, nether of the Crible Stream Sources Syslog or UDP are able to pick up the data, whether I use the specific IP for the NIC or leave it wide open (0.0.0.0).I have gone as far as SELinux running in permissive mode and disabled the firewalld, all with no luck with the source picking up data.Has anyone run into this issue and if so, how did they solve it?Scott

IMO, the Cribl logging need improvements.While in logs can see errors it is hard find which a pipeline, function causing it.My suggestion is to add infromation to each ERROR line about a pipeline and fuction releated to an error.

Hi All, i am facing an issue in Cribl data script, Actually the job is completed based on the logs, but it stuck saying 2 tasks inflight,We are using script option to pull the data, We have handled all the scenarios to prevent the issues, but after sometime, the job is getting stuck, As we can see in the above screenshot, totally 55 tasks( 1 discovery + 54 collector), and it says, 53 completed. But when I downloaded the logs where condition is ""message": "collect task completed","  based on this log count, inclusive of 0 to 53 totally 54. and all of them is completed but why the job didnt stopped.

I am confused about the new Cribl permissions system, I am in the process of configuring some roles for a local user, I am very happy with the fine grained controls available, especially allowing me to control individual endpoint access, the new permissions system appears to be much more coarse and losing a lot of functionality available in roles? Seems to be a "dumbing down" of the permissions system that takes away a lot of control?Hope I am misunderstanding this.Thanks for your help

The default format of exporting pipelines is the JSON. As the YML is used in config files, IMO, also exporting to this format should be possible.

Hi Team,I am a Cribl Certified Engineer and have been working on a use case which is described below, but I haven't achieved the expected output.The use case involves an S3/Blob Storage where we have one bucket containing 2-3 folders. The source should be a specific folder, and from that folder, I need to ingest data to the required destination.In S3/Blob, I can see that regex can be used for matching file names to download and process, with the default being .*. Can I change it to /foldername/.*? Will this work? If yes, can I have a whitepaper or some documentation on this?