Stream

Trying to do a playback using Azure Blob. We are sending logs to this Azure Blob for our DataLake. Logs are being written to the Blob and are stored by year/month/day.Set up the Azure Blob as a source using the same connection strings. When running a preview test we get the following error.This request is not authorized to perform this operation using this permission.code:AuthorizationPermissionMismatch

What are the default connection limits for a worker? Is there a maximum number of connections a worker can handle?

We're seeing the following message and events are no longer flowing through the Worker Group.Message: Failed to send anonymized telemetry metadata. Data flow has been stopped.Failed to send anonymized telemetry metadata. License is blocking inputs. Please contact sales@cribl.io or check that firewall rules are not blocking Cribl telemetry metadata collection.

This is gonna sound insane buuuuut…we have a team that has some evtx files that sit on a NAS. Using edge on an independent windows machine, is it possible to monitor the files on the NAS location to collect those files thru the Windows Event Logs source in Edge? Or is it only possible to collect the local windows files for that machine specifically?

Noticed one of multiple collectors wasn’t working (I probably broke it last time I was fooling around). If I go to Data/Sources, “Collectors” at the top can be clicked on but the resulting page is empty. If I select any other input on that page, it goes to that page and from there I can see there are 12 Collectors configured, but if I click Collectors there I just get an “API error” and get dumped back at the main Data Sources page.

How can I use the Stream REST API to create new routes? The API doc reference doesn’t show a POST but I do see a PATCH. Is PATCH the operation I should use? Can someone provide an example payload by chance?

I’m using the /system/logs/search endpoint to filter for custom events within the access.log file.Filters work properly via log search in the UIMy filter experience with the API is still clumsy and time consumingIs there a way to quickly generate a custom search query that can be used with the log search endpoint from the UI?

When running a preview of a Pack with two lookup functions pointing to 1 million records files, I get the message:Preview results may be incomplete due to process timeoutIf I disable one of the functions - it does not matter which one - the preview works fine.Is there a lookup limit the preview can handle? The sample file used contains 57 events.Thanks.

If a distributed leader’s configuration is lost and it reverts to a standalone instance, the existing worker group configurations should remain on the workers, since they haven’t gotten any updates. What’s the easiest way to get those configurations copied over to the leader?

Hi all. For us poor slobs currently stuck with a manual process… has anyone tested a backup and restore? I am zipping /local, /data, and /default under /opt/cribl, and a quick test seemed to indicate this works for DR, but anyone know of other directories that could be needed?

I have a data stream that contains various different log types. For example, I have a Windows log, a firewall log and an antivirus log in the same stream. Each of the events from the source is tagged for exampletype: windowstype: firewalltype: antivirusWhat I want to do is split these logs by type. Essentially I would like to do:If the type equals windows then drop everything else and run it through a pipeline / route to transform my data with a windows parser and ship my dataIf the type equals firewall then drop everything else and run it through a pipeline / route to transform my data with a firewall parser and ship my dataIf the type equals antivirus then drop everything else and run it through a pipeline / route to transform my data with a antivirus parser and ship my dataI started thinking that I would use the eval function to filter based on this and group them together but unsure if t

We are currently in PoC. We cannot configure RBAC on a single instance. I need this to work; reading in Cribl documents, RBAC is allowed only in a distributed environment; this is not feasible. How do you go around this restriction?

Hello everyone,is there a “best practise” way to collect internal and metrics logs from the cribl leader?For worker is an existing Source called “Cribl Internal”.Thank you

Hello there,we currently have a leader with 10 workers in different groups under it.We use Indexer discovery within Splunk to connect the workers in the destinations tab to our Indexer.Can i copy the outputs.yml from one worker to another worker in a seperated group to directly have the config ready? Or do i have to go the way thorugh the GUI and make my marks.Thank you

I was sending events to a syslog-ng server with the following configsource s_test { syslog( transport(tcp) port());};I introduced Stream, and now I’m not seeing those logs on our syslog-ng server. Instead, I’m seeing this:Invalid frame header; header=‘’syslog-ng[3025832]: Syslog connection closed; fd=‘12’, client=‘AF_INET(10.1.80.158:12506)’, local=‘AF_INET(0.0.0.0:23514)’How can I fix this?

I am interested in sending Heroku logs to Stream. Has anyone ever done this before?

Hello,We have turned on the cribl internal metrics source and are receiving all the metrics data in Splunk via HEC. The only metric we are not receiving is cribl.logstream.sourcetype.in_bytes. We are receiving out_bytes.Does anyone know the reason for this?Thank you

An F5 load-balancer was setup and should be sending syslog traffic to some Cribl Stream workers using round-robin. Currently there is a single source of events, for testing purposes. The events are almost exclusively going to a single worker, which is unexpected.The expectation is that the events would be load-balanced and that there would be a fair distribution across the worker nodes. According to F5 team, the connection gets load-balanced, so as more sources added, traffic will start showing on the other workers. That is to be expected but since UDP syslog is being used the events from a single source should already getting load balanced.

Does the unroll function support any other methods than split() ?Please give me your examples 🙂

Hi,I’ve configured an index template in Elastic for a specific log source and set it to use a data stream instead of the indices.Whenever Cribl starts sending data, the data stream is created, but no data is added to is. I presume this has something to do with the _bulk API, since data streams only support the op_type create.Does anyone have a working configuration for logging towards data streams in Elastic?

HiWe have setup event hub to send data stream to cribl and our now trying to extract the json into individual events. We have tried the event breaker JSON Array method and JSON unroll method. Neither seems to work. Has anyone tried this successfully or have pointers?Thank you

Hi guys,I’m testing cribl stream for the 1st time by getting a splunk UF to forward to a Worker on the same centos7 host (ip: 10.0.2.9).When checking Live Data, I only see events from /opt/splunkforwarder/var/log/splunk/metrics.logI was expecting to see events from /var/log/audit/audit.log and /var/log/secure which are listed as being monitored files:/opt/splunkforwarder/bin/splunk list monitorYour session is invalid. Please login.Splunk username: adminPassword:Monitored Directories:$SPLUNK_HOME/var/log/splunk/opt/splunkforwarder/var/log/splunk/audit.log/opt/splunkforwarder/var/log/splunk/btool.log/opt/splunkforwarder/var/log/splunk/conf.log/opt/splunkforwarder/var/log/splunk/configuration_change.log/opt/splunkforwarder/var/log/splunk/dfm_stderr.log/opt/splunkforwarder/var/log/splunk/dfm_stdout.log/o

HiAn interesting topic came up yesterday which I didn’t have a good answer too, around how many events are in flight in Stream at any point in time.It was a discussion around potential for data loss, and if something adverse were to happen the worker, what the impact would be. I imagine there is up to one event being processed per worker thread at any point in time, but is there a buffer of events before the worker thread picks it up? If so, how much, and if not, does the thread just block the network socket until it’s done processing?Supplementary to that - what happens when a destination goes down? Persistent queues aside, how much will Cribl buffer before applying back-pressure to the pipeline?And probably scenarios I haven’t considered. How much data is it theoretically possible to have in flight at any point in time, and thus risk of loss in case of catastrophe?

I am in charge of documenting our disaster recovery and high-availability efforts. One of the questions that came up is whether the worker nodes will cease to function (or reduce functionality) when the leader node disappears for a prolonged time. If so, how much time do we have until we must resurrect the leader?

I’m curious how the GCS source keeps track of files that it has already consumed and what prevents Stream from consuming the same file more than once? Our use case involves a BigQuery export of Gmail activity logs to a GCS bucket. We’d like to transfer those events to an S3 data lake (I know, moving from one cloud storage to another; we have reasons, but I digress). Ideally, we’d like to consume the events from each file in the GCS bucket and upon success, delete the file from GCS, but at the least, we want to make sure we don’t ingest the same file(s) over and over again.Thanks!