Products

Hi, I have problem with deploy config from Leader to Worker. I have distributed deployment with 2 Workers and 1 worker group (testing, free license used). I configured secure connection between Leader and Workers according to https://docs.cribl.io/stream/securing-communications 1. For the first look everything is OK - both Workers are visible on Manage Worker Nodes on Leader as alive, I can connect to Workers GUI (Remote UI Access is on), config version is OK etc.Problem is when I want to deploy config changes to Workers. On Manage Worker Nodes I see yellow exclamation mark and neverending rotating circle in Config version column and config changes are not propagated to Workers.In log on Leader I see this:{"time":"2022-06-17T13:01:21.350Z","cid":"api","channel":"CriblMaster","level":"info","message":"sending config update request","group":"default","version":"8627dcf","logStreamEnv":"master","worker":"ff5ea4c4-51eb-4aa4-a650-2e6c8251bb21"}{"time":"2022-06-17T13:01:21.363Z","cid":"api",

HiWe have setup event hub to send data stream to cribl and our now trying to extract the json into individual events. We have tried the event breaker JSON Array method and JSON unroll method. Neither seems to work. Has anyone tried this successfully or have pointers?Thank you

Are the available Redis commands in the dropdown list or should I go by the complete list at the URL provided when I hover over the ‘?’?I’m trying to set multiple values for a hash and I’m having trouble grabbing the key-value pairs from my event to send. I’m not sure if it’s a limitation or I’m doing something incorrectly. I’m using ‘__e’ to grab the entire event (Args section) and specifying a key. The Command section allows me to put in hmset.

Can I use self-signed certificate with a REST collector? Does Cribl perform any TLS certificate validity checks?

If I need to accept an SNMP trap from a source, and send it to multiple trap destinations, will Cribl spoof the source IP and send the trap “un-altered” to the final destinations? Meaning will it preserve the source IP address from the original trap IP header?

I’ve made the switch from a single instance to a distributed setup (1 leader and 2 worker nodes). My old server, which receives logs from a variety of sources, is now acting as a worker. What happens if this worker goes down, because all of the sources are sending logs to one single worker IP? What should I do to guarantee appropriate high availability?

Not a question but rather something useful. I have data that is in key:value format, using key=value in the extract format did not work. I was able to extract every key and value with this one regex extract.(?<_NAME_0>.*):\s(?<_VALUE_0>.*)

i have a use case where we have a nfs share that applications writes logs too i would like to monitor these while being written to (file monitor?) but distribute the load across workers. is that possible?

Hi guys,I’m testing cribl stream for the 1st time by getting a splunk UF to forward to a Worker on the same centos7 host (ip: 10.0.2.9).When checking Live Data, I only see events from /opt/splunkforwarder/var/log/splunk/metrics.logI was expecting to see events from /var/log/audit/audit.log and /var/log/secure which are listed as being monitored files:/opt/splunkforwarder/bin/splunk list monitorYour session is invalid. Please login.Splunk username: adminPassword:Monitored Directories:$SPLUNK_HOME/var/log/splunk/opt/splunkforwarder/var/log/splunk/audit.log/opt/splunkforwarder/var/log/splunk/btool.log/opt/splunkforwarder/var/log/splunk/conf.log/opt/splunkforwarder/var/log/splunk/configuration_change.log/opt/splunkforwarder/var/log/splunk/dfm_stderr.log/opt/splunkforwarder/var/log/splunk/dfm_stdout.log/opt/splunkforwarder/var/log/splunk/first_install.log/opt/splunkforwarder/var/log/splunk/health.log/opt/splunkforwarder/var/log/splunk/license_usage.log/opt/splunkforwarder/var/log/sp

What does this message indicate on the Splunk TCP source:message:Failed to parse s2s payload{}error:αmessage:Cannot advance offset 2023514992 past end of event 74αstack:Error: Cannot advance offset 2023514992 past end of event 74at p.advance (/opt/cribl/bin/cribl.js:14:12271888)at h._readEvents (/opt/cribl/bin/cribl.js:14:12274323)at h._transform (/opt/cribl/bin/cribl.js:14:12275791)at h.Transform._read (_stream_transform.js:205:10)at h.Transform._write (_stream_transform.js:193:12)at writeOrBuffer (_stream_writable.js:352:12)at h.Writable.write (_stream_writable.js:303:10)at Socket.ondata (_stream_readable.js:719:22)at Socket.emit (events.js:315:20)at Socket.EventEmitter.emit (domain.js:486:12)Show less

Hi Guys,I have setup a test environment on my laptop using VBOX and with a Centos7 edge leader node and worker using ports 9000(ui) and 4200 for internal comms.I’ve installed the worker node using curl. Seems there is an issue with the worker node connecting to the leader as shown below. From the worker node, telnet to the leader nodes port 9000 and 4200 connect fine. But not sure what additional stuff to check based on the below messages:{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"info","message":"attempting to connect","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"will retry to connect","nextConnectTime":1653267018346}{"time":"2022-05-23T00:50:14.256Z","cid":"api","channel":"output:DistWorker","level":"debug","message":"connecting","host":"10.0.2.4","port":4200,"tls":false}{"time":"2022-05-23T00:50:14.264Z","cid":"api","channel":"input:D

We have a collector script (python script) that can take up to 10 min to pull down all the data (the script will cache it) and then “print” it out so that Cribl can then ingest it. Ends up being equivalent to a 650MB CSV file. The first ad hoc full run seemed to have timed out even though I had 0 set for job timeout. Debug shows a lot of “failed to pop task” messages while it is running. Any ideas?

So I have a script that will collect data from an API. The data is returns is CSV formatted, the first line will be the list of fields. How do you tell Stream to use that first line as the headers?

HiAn interesting topic came up yesterday which I didn’t have a good answer too, around how many events are in flight in Stream at any point in time.It was a discussion around potential for data loss, and if something adverse were to happen the worker, what the impact would be. I imagine there is up to one event being processed per worker thread at any point in time, but is there a buffer of events before the worker thread picks it up? If so, how much, and if not, does the thread just block the network socket until it’s done processing?Supplementary to that - what happens when a destination goes down? Persistent queues aside, how much will Cribl buffer before applying back-pressure to the pipeline?And probably scenarios I haven’t considered. How much data is it theoretically possible to have in flight at any point in time, and thus risk of loss in case of catastrophe?

One of the neatest features of the product is the live preview. This works great for relatively small events.If I have a massive event with a pretty big intensive pipeline to boot, things time out and it really is a PITA to debug my code. How do people work through such a work flow?I usually increase the timeout from 10 seconds to 60 seconds and "play" the pipeline, but I wanted to know if people had some other nifty hacks.Thank you!b1scu1t

Hi, not sure if this is a FREE license thing or something I have wrong with docker. Here is my docker compose.version: 3.5services: master: image: ${CRIBL_IMAGE:-cribl/cribl:latest} environment: - CRIBL_DIST_MODE=master - CRIBL_DIST_MASTER_URL=tcp://criblmaster@0.0.0.0:4200 - CRIBL_VOLUME_DIR=/share/homes/chris/cribl:/opt/cribl/config-volume ports: - 19000:9000 hostname: cribl_leader volumes: - /share/homes/chris/cribl:/opt/cribl/config-volume workers: image: ${CRIBL_IMAGE:-cribl/cribl:latest} depends_on: - master hostname: cribl_worker environment: - CRIBL_DIST_MODE=worker - CRIBL_DIST_MASTER_URL=tcp://criblmaster@0.0.0.0:4200 ports: - 9000Every-time I modify the compose and redeploy, I have to re-input config (user / password, registration, etc). The homes DIR is on the host and has open permissions. I do see data being written in there. Is there a setting I may be missing? Thank you Chris

As a Stream user, if I deploy Edge, do I need to have a leader for each of them? Or do they work from the same leader that I use for Stream today?

when i try to upgrade splunk app via search head deployer i get an error that it fails to apply bundle because cribl is busyError while deploying apps to first member, aborting apps deployment to all members: Error while updating app=cribl on target=https://10.150.22.183:8089: Non-200/201 status _code=500; {"messages":[{"type":"ERROR","text":"\n In handler localapps: Error installing application: Failed to copy: /opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa 723/cribl/bin to /opt/splunk/etc/apps/cribl/bin. 2 errors occurred. Description for first 2: [{operation:\"copying source to destination\", error:\"Text file busy\", src:\" /opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa723/cribl/bin/cribl\", dest:\"/opt/splunk/etc/apps/cribl/bin/cribl\"}, {operation:\"transfering contents from source to d estination\", error:\"Text file busy\", src:\"/opt/splunk/var/run/splunk/bundle_tmp/eb494662a4bfa723/cribl/bin\", dest:\"/opt/splunk/etc/apps/cribl/bin\"}]"}]}

I am in charge of documenting our disaster recovery and high-availability efforts. One of the questions that came up is whether the worker nodes will cease to function (or reduce functionality) when the leader node disappears for a prolonged time. If so, how much time do we have until we must resurrect the leader?

When adding a GeoIP function to a pipeline, where do I upload .mmdb files (or where are they stored if I grabbed a pack that includes one)?

I wrote a pretty elaborate pipeline that extracts a bunch of metrics from a pretty hideously ugly event and further creates a multi metric that is further shipped to a metric index in splunk thanks to @david & @bdalpe.I noticed today when I was making some minor edits to this pipeline it would time out with the default 10 second, but when I made it 60 seconds, it ran to success. When I promote this pipeline into production, how can I make sure none of my pipelines timed out and dropped the data? When I query Splunk I do see the metrics, but I just want to make sure I am not losing stuff.Thanks!

I am kinda being nudged into a move from a Standalone to a distributed setup. So have to start playing in the lab with this before I plan it out in production. Does any cribl’er have any write ups or discussions here where such a thing has been talked over that I can review before embarking on this adventure.My end goal is to have a Cribl Stream Distributed + A handful of Edge clients to start with(and a ton of Splunk Universal Forwarders sending out stuff to the worker node).

I’m curious how the GCS source keeps track of files that it has already consumed and what prevents Stream from consuming the same file more than once? Our use case involves a BigQuery export of Gmail activity logs to a GCS bucket. We’d like to transfer those events to an S3 data lake (I know, moving from one cloud storage to another; we have reasons, but I digress). Ideally, we’d like to consume the events from each file in the GCS bucket and upon success, delete the file from GCS, but at the least, we want to make sure we don’t ingest the same file(s) over and over again.Thanks!

Any good debugging steps for the O365 activity log source. We set it up correctly could validate the tokens but even with a poll intervall of 1 Min I am not getting any data nor any logs. Would appreciate any debugging steps to help me understand the mess I created.

GOATS,I run multiple REST collectors where the APIs I am collecting from have hard limits on the number of events returned which requires me to paginate.Doing this creates multiple collection tasks. Part of my main pipeline runs a custom REST function to create an object in the API based on a field in the event and returns data to enrich my event. If the values exists in the API I get an "this already exists" error. I normally just aggregate this field, make my API call, then join and unroll the events on the enrichments.This breaks when the REST collector creates multiple collection tasks as it seems that I cannot aggregate all of my events into one single aggregate event even when all the events are within my time window.How can I aggregate my events from a single job that creates multiple collection tasks into a single event?Thank you!