Community Discussions

Hi Guys,I have setup a test environment on my laptop using VBOX and with a Centos7 edge leader node and worker using ports 9000(ui) and 4200 for internal comms.I’ve installed the worker node using curl. Seems there is an issue with the worker node connecting to the leader as shown below. From the worker node, telnet to the leader nodes port 9000 and 4200 connect fine. But not sure what additional stuff to check based on the below messages:{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"info","message":"attempting to connect","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"will retry to connect","nextConnectTime":1653267018346}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"connecting","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.264Z","cid":"api","channel":"input:D

We have a collector script (python script) that can take up to 10 min to pull down all the data (the script will cache it) and then “print” it out so that Cribl can then ingest it. Ends up being equivalent to a 650MB CSV file. The first ad hoc full run seemed to have timed out even though I had 0 set for job timeout. Debug shows a lot of “failed to pop task” messages while it is running. Any ideas?

So I have a script that will collect data from an API. The data is returns is CSV formatted, the first line will be the list of fields. How do you tell Stream to use that first line as the headers?

HiAn interesting topic came up yesterday which I didn’t have a good answer too, around how many events are in flight in Stream at any point in time.It was a discussion around potential for data loss, and if something adverse were to happen the worker, what the impact would be. I imagine there is up to one event being processed per worker thread at any point in time, but is there a buffer of events before the worker thread picks it up? If so, how much, and if not, does the thread just block the network socket until it’s done processing?Supplementary to that - what happens when a destination goes down? Persistent queues aside, how much will Cribl buffer before applying back-pressure to the pipeline?And probably scenarios I haven’t considered. How much data is it theoretically possible to have in flight at any point in time, and thus risk of loss in case of catastrophe?

One of the neatest features of the product is the live preview. This works great for relatively small events.If I have a massive event with a pretty big intensive pipeline to boot, things time out and it really is a PITA to debug my code. How do people work through such a work flow?I usually increase the timeout from 10 seconds to 60 seconds and "play" the pipeline, but I wanted to know if people had some other nifty hacks.Thank you!b1scu1t

Hi, not sure if this is a FREE license thing or something I have wrong with docker. Here is my docker compose.version: 3.5services: master: image: ${CRIBL_IMAGE:-cribl/cribl:latest} environment: - CRIBL_DIST_MODE=master - CRIBL_DIST_MASTER_URL=tcp://criblmaster@0.0.0.0:4200 - CRIBL_VOLUME_DIR=/share/homes/chris/cribl:/opt/cribl/config-volume ports: - 19000:9000 hostname: cribl_leader volumes: - /share/homes/chris/cribl:/opt/cribl/config-volume workers: image: ${CRIBL_IMAGE:-cribl/cribl:latest} depends_on: - master hostname: cribl_worker environment: - CRIBL_DIST_MODE=worker - CRIBL_DIST_MASTER_URL=tcp://criblmaster@0.0.0.0:4200 ports: - 9000Every-time I modify the compose and redeploy, I have to re-input config (user / password, registration, etc). The homes DIR is on the host and has open permissions. I do see data being written in there. Is there a setting I may be missing? Thank you Chris

As a Stream user, if I deploy Edge, do I need to have a leader for each of them? Or do they work from the same leader that I use for Stream today?

when i try to upgrade splunk app via search head deployer i get an error that it fails to apply bundle because cribl is busyError while deploying apps to first member, aborting apps deployment to all members: Error while updating app=cribl on target=https://10.150.22.183:8089: Non-200/201 status _code=500; {"messages":[{"type":"ERROR","text":"\n In handler localapps: Error installing application: Failed to copy: /opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa 723/cribl/bin to /opt/splunk/etc/apps/cribl/bin. 2 errors occurred. Description for first 2: [{operation:\"copying source to destination\", error:\"Text file busy\", src:\" /opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa723/cribl/bin/cribl\", dest:\"/opt/splunk/etc/apps/cribl/bin/cribl\"}, {operation:\"transfering contents from source to d estination\", error:\"Text file busy\", src:\"/opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa723/cribl/bin\", dest:\"/opt/splunk/etc/apps/cribl/bin\"}]"}]}

I am in charge of documenting our disaster recovery and high-availability efforts. One of the questions that came up is whether the worker nodes will cease to function (or reduce functionality) when the leader node disappears for a prolonged time. If so, how much time do we have until we must resurrect the leader?

When adding a GeoIP function to a pipeline, where do I upload .mmdb files (or where are they stored if I grabbed a pack that includes one)?

I wrote a pretty elaborate pipeline that extracts a bunch of metrics from a pretty hideously ugly event and further creates a multi metric that is further shipped to a metric index in splunk thanks to @david & @bdalpe.I noticed today when I was making some minor edits to this pipeline it would time out with the default 10 second, but when I made it 60 seconds, it ran to success. When I promote this pipeline into production, how can I make sure none of my pipelines timed out and dropped the data? When I query Splunk I do see the metrics, but I just want to make sure I am not losing stuff.Thanks!

I am kinda being nudged into a move from a Standalone to a distributed setup. So have to start playing in the lab with this before I plan it out in production. Does any cribl’er have any write ups or discussions here where such a thing has been talked over that I can review before embarking on this adventure.My end goal is to have a Cribl Stream Distributed + A handful of Edge clients to start with(and a ton of Splunk Universal Forwarders sending out stuff to the worker node).

I’m curious how the GCS source keeps track of files that it has already consumed and what prevents Stream from consuming the same file more than once? Our use case involves a BigQuery export of Gmail activity logs to a GCS bucket. We’d like to transfer those events to an S3 data lake (I know, moving from one cloud storage to another; we have reasons, but I digress). Ideally, we’d like to consume the events from each file in the GCS bucket and upon success, delete the file from GCS, but at the least, we want to make sure we don’t ingest the same file(s) over and over again.Thanks!

Any good debugging steps for the O365 activity log source. We set it up correctly could validate the tokens but even with a poll intervall of 1 Min I am not getting any data nor any logs. Would appreciate any debugging steps to help me understand the mess I created.

GOATS,I run multiple REST collectors where the APIs I am collecting from have hard limits on the number of events returned which requires me to paginate.Doing this creates multiple collection tasks. Part of my main pipeline runs a custom REST function to create an object in the API based on a field in the event and returns data to enrich my event. If the values exists in the API I get an "this already exists" error. I normally just aggregate this field, make my API call, then join and unroll the events on the enrichments.This breaks when the REST collector creates multiple collection tasks as it seems that I cannot aggregate all of my events into one single aggregate event even when all the events are within my time window.How can I aggregate my events from a single job that creates multiple collection tasks into a single event?Thank you!

Is there a pattern that I can use to suppress Splunk logs based on data that is part of the _raw field? So far I have come up with a pipeline that uses Regex Extract, Suppress, and Eval function. I am extracting the key:value pair that I need from the _raw field, utilize it in my suppression function, and then remove it from the logs before sending the event to Splunk. While this seems to be working, I wonder if there is a better way to achieve this especially if we want to add additional key:value pairs to suppress on in the future? I was looking into the Parser function that allows me to extract all key:value pairs from the _raw field but I could not find a way to remove all those from the event before sending it to Splunk.

Can i do this in the product today or would it have to be an alert developed outside of Cribl Stream?

I’ve been experimenting with “Next Processing Queue” in the Advanced Settings for a SPlunk HEC Destination, and I can’t seem to tell any difference when I change the value. I’ve used the default value (indexQueue) as well as parsingQueue, but I don’t see any difference once the data gets to Splunk.What exactly is this setting supposed to do?

We are enabling persistent queuing (PQ) on our destinations and notices that the Max File size is defaulted to 1MB. With the amount of data we are processing that will lead to A LOT of small files so we were thinking of increasing that to a Max File Size of 100MB. However, before we do that - we wanted to know if there was any potential impact to Cribl if the PQ files are to large ?When Cribl is reading these files to send to the destination, does it read the entire file into memory ?

When i check the monitoring page for sources, we still see data, and we see logs for the deleted collector. The logs button is greyed out though.

Is there a good way to enable HEC destinations where the HEC input on the Splunk side has Indexer acknowledgment enabled?

I have a field that includes IP and Port (i.e. 0.0.0.0:1234) and I want to split them into separate fields for further analysis. What’s the best way to do that?

I want to make sure I understand any additional costs.

Does GitOps only make changes via the UI read-only or are changes via API read-only also?

How much data can I use before I need to upgrade from the free version?