Stream

I’m using the /system/logs/search endpoint to filter for custom events within the access.log file.Filters work properly via log search in the UIMy filter experience with the API is still clumsy and time consumingIs there a way to quickly generate a custom search query that can be used with the log search endpoint from the UI?

When running a preview of a Pack with two lookup functions pointing to 1 million records files, I get the message:Preview results may be incomplete due to process timeoutIf I disable one of the functions - it does not matter which one - the preview works fine.Is there a lookup limit the preview can handle? The sample file used contains 57 events.Thanks.

If a distributed leader’s configuration is lost and it reverts to a standalone instance, the existing worker group configurations should remain on the workers, since they haven’t gotten any updates. What’s the easiest way to get those configurations copied over to the leader?

Hi all. For us poor slobs currently stuck with a manual process… has anyone tested a backup and restore? I am zipping /local, /data, and /default under /opt/cribl, and a quick test seemed to indicate this works for DR, but anyone know of other directories that could be needed?

I have a data stream that contains various different log types. For example, I have a Windows log, a firewall log and an antivirus log in the same stream. Each of the events from the source is tagged for exampletype: windowstype: firewalltype: antivirusWhat I want to do is split these logs by type. Essentially I would like to do:If the type equals windows then drop everything else and run it through a pipeline / route to transform my data with a windows parser and ship my dataIf the type equals firewall then drop everything else and run it through a pipeline / route to transform my data with a firewall parser and ship my dataIf the type equals antivirus then drop everything else and run it through a pipeline / route to transform my data with a antivirus parser and ship my dataI started thinking that I would use the eval function to filter based on this and group them together but unsure if t

We are currently in PoC. We cannot configure RBAC on a single instance. I need this to work; reading in Cribl documents, RBAC is allowed only in a distributed environment; this is not feasible. How do you go around this restriction?

Hello everyone,is there a “best practise” way to collect internal and metrics logs from the cribl leader?For worker is an existing Source called “Cribl Internal”.Thank you

Hello there,we currently have a leader with 10 workers in different groups under it.We use Indexer discovery within Splunk to connect the workers in the destinations tab to our Indexer.Can i copy the outputs.yml from one worker to another worker in a seperated group to directly have the config ready? Or do i have to go the way thorugh the GUI and make my marks.Thank you

I was sending events to a syslog-ng server with the following configsource s_test { syslog( transport(tcp) port());};I introduced Stream, and now I’m not seeing those logs on our syslog-ng server. Instead, I’m seeing this:Invalid frame header; header=‘’syslog-ng[3025832]: Syslog connection closed; fd=‘12’, client=‘AF_INET(10.1.80.158:12506)’, local=‘AF_INET(0.0.0.0:23514)’How can I fix this?

I am interested in sending Heroku logs to Stream. Has anyone ever done this before?

Hello,We have turned on the cribl internal metrics source and are receiving all the metrics data in Splunk via HEC. The only metric we are not receiving is cribl.logstream.sourcetype.in_bytes. We are receiving out_bytes.Does anyone know the reason for this?Thank you

An F5 load-balancer was setup and should be sending syslog traffic to some Cribl Stream workers using round-robin. Currently there is a single source of events, for testing purposes. The events are almost exclusively going to a single worker, which is unexpected.The expectation is that the events would be load-balanced and that there would be a fair distribution across the worker nodes. According to F5 team, the connection gets load-balanced, so as more sources added, traffic will start showing on the other workers. That is to be expected but since UDP syslog is being used the events from a single source should already getting load balanced.

Does the unroll function support any other methods than split() ?Please give me your examples 🙂

Hi,I’ve configured an index template in Elastic for a specific log source and set it to use a data stream instead of the indices.Whenever Cribl starts sending data, the data stream is created, but no data is added to is. I presume this has something to do with the _bulk API, since data streams only support the op_type create.Does anyone have a working configuration for logging towards data streams in Elastic?

HiWe have setup event hub to send data stream to cribl and our now trying to extract the json into individual events. We have tried the event breaker JSON Array method and JSON unroll method. Neither seems to work. Has anyone tried this successfully or have pointers?Thank you

Hi guys,I’m testing cribl stream for the 1st time by getting a splunk UF to forward to a Worker on the same centos7 host (ip: 10.0.2.9).When checking Live Data, I only see events from /opt/splunkforwarder/var/log/splunk/metrics.logI was expecting to see events from /var/log/audit/audit.log and /var/log/secure which are listed as being monitored files:/opt/splunkforwarder/bin/splunk list monitorYour session is invalid. Please login.Splunk username: adminPassword:Monitored Directories:$SPLUNK_HOME/var/log/splunk/opt/splunkforwarder/var/log/splunk/audit.log/opt/splunkforwarder/var/log/splunk/btool.log/opt/splunkforwarder/var/log/splunk/conf.log/opt/splunkforwarder/var/log/splunk/configuration_change.log/opt/splunkforwarder/var/log/splunk/dfm_stderr.log/opt/splunkforwarder/var/log/splunk/dfm_stdout.log/o

HiAn interesting topic came up yesterday which I didn’t have a good answer too, around how many events are in flight in Stream at any point in time.It was a discussion around potential for data loss, and if something adverse were to happen the worker, what the impact would be. I imagine there is up to one event being processed per worker thread at any point in time, but is there a buffer of events before the worker thread picks it up? If so, how much, and if not, does the thread just block the network socket until it’s done processing?Supplementary to that - what happens when a destination goes down? Persistent queues aside, how much will Cribl buffer before applying back-pressure to the pipeline?And probably scenarios I haven’t considered. How much data is it theoretically possible to have in flight at any point in time, and thus risk of loss in case of catastrophe?

I am in charge of documenting our disaster recovery and high-availability efforts. One of the questions that came up is whether the worker nodes will cease to function (or reduce functionality) when the leader node disappears for a prolonged time. If so, how much time do we have until we must resurrect the leader?

I’m curious how the GCS source keeps track of files that it has already consumed and what prevents Stream from consuming the same file more than once? Our use case involves a BigQuery export of Gmail activity logs to a GCS bucket. We’d like to transfer those events to an S3 data lake (I know, moving from one cloud storage to another; we have reasons, but I digress). Ideally, we’d like to consume the events from each file in the GCS bucket and upon success, delete the file from GCS, but at the least, we want to make sure we don’t ingest the same file(s) over and over again.Thanks!

Can i do this in the product today or would it have to be an alert developed outside of Cribl Stream?

I have a field that includes IP and Port (i.e. 0.0.0.0:1234) and I want to split them into separate fields for further analysis. What’s the best way to do that?

The Filesystem destination allows setting a prefix and suffix for the filename. However, I don’t see a place to set the actual base filename, or to just not have one. I don’t want the 6 random character base filename. How can this be removed? Also, I’d prefer not to have file staging, since I am not compressing files, is there a way to just write raw output to disk in real-time? Using the staging dir creates latency.

Had a situation where someone added a bunch of files for an upgrade that was committed on the leader node but not deployed. I want to revert back but the UI is broken for this. Can I do a revert back from the cli to force it to a good version?

If I add an additional node only for traffic passthrough, does the additional hop affect the total amount of traffic that counts against the Cribl license? How is Stream → Stream or Edge–>Stream traffic marked and tracked for billing purposes? ie what needs to be set to make sure it doesn’t double count?

What permissions are needed for Cribl Stream to pull an S3 bucket? I’m creating a role for it to use.