Community Discussions

I am trying to send the count or average of certain fields instead of the whole events for a specific time period.

I’ve cloned user, and am now trying to add more GET polices. However, looks like ‘GET’ /system/something, ‘POST’ etc can’t be added as NewPolicy - only GroupFull etc available.Can this be done in the UI?or only using POST /system/roles?or do I need to post new Policy , POST /system/policies?

I am seeing some irregularities with collecting large files from a filesystem.We batch process files anywhere from 100MB to 100GB in size. I am currently noticing an issue with larger files.To troubleshoot I created a collector that reads the data in, and directly writes it back to disk. No other ETL is done on the data.Result of 1 collectionResult of collecting the same file from the same location using the same event breakers and going to the same destination.My current environment is 1 leader, 1 worker, and the file is being picked up from an NFS mount.As you can see from the screenshots, only 3-4 million of the events are being collected of the ~24 million events in the file. The destination is writing to disk about 5-6 GB from the original ~38GB.There are no errors that I see in the job log, and I can't find any setting regarding worker process limits or job limits that would affect this.

Given a key-value formatted source, routed through a pipeline where data is parsed into JSON then manipulatued, how do I write the new data back out in KV format? Using the filesystem Destination, I’m getting JSON after my pipeline regardless of whether I use Raw or JSON output. I’m assuming there’s a function I’m missing that does this. The closest thing I can find is Serialize, but that doesn’t support KV format.

When I look at Monitoring > Logs, I can’t find logs from scheduled collection jobs using channel.includes(input_name)

Hello,i need some input.We want to have a separate dataflow between different stages within cribl.So we receive data in our prod environment and want to have an extra dataflow to QS stage, to test parsing changes etc.One way we know is to configure a second output wihtin “Output Router”Are there any other ways? Maybe within routes or pipelines?Thank you for your input

FYI, I was unable to sign up to the site via the normal account sign up.Never got any confirmation emails, tried multiple email accounts (they are were all hosted with google if that has a bearing on anything)Had to use “OpenID Connect” and sign up with Google

Has anyone connected up an AWS Opensearch Service instance to Cribl? I created a local backend user in my domain and gave it what I thought were good permissions but I still get a 401 error when I test the connection. Can anyone share what they did to get this to work?

I’m writing a custom function and I’m able to add it to a pipeline. However, when I try to process an event, I get an error.Here’s the content of my index.js:exports.name = ‘MyCustomFunction’;exports.group = ‘Custom Functions’;exports.version = 0.1;exports.init = (opts) => {const conf = opts.conf || {};};exports.process = (event) => {return event; // noop};I’m unable to find where, if any, error messages are written to the logs. I looked at the logs under Monitoring (Logs tab) and docker logs with no success.Are errors of this type written anywhere?

Hello,We are trying to enable the cribl metrics to send to Splunk. We are specifically looking for the following metrics:cribl.logstream.host.(in_bytes,in_events,out_bytes,out_events)cribl.logstream.index.(in_bytes,in_events,out_bytes,out_events)cribl.logstream.source.(in_bytes,in_events,out_bytes,out_events)cribl.logstream.sourcetype.(in_bytes,in_events,out_bytes,out_events)The source has been configured with Full Fidelity set to Yes. In the settings under limits, we have removed all the values in Disable Field Metrics on both the leader and worker nodes. We have restarted all the services.The data is still not visible in the Sources → Live Data view.Have we missed a step?Thank you

I am trying to capture login attempts (successful/unsuccessful) to Cribl. I can see that the cribl.log file contains logs for these (with a type of “auth”).I have enabled CriblLogs as a source.The log level for channel “auth” is set to Info.When I do a capture on the CriblLogs source I don’t see these “auth” events, I do however see (some) other events - any suggestions as to what I might be doing wrong?Thanks.

I am currently bringing in a bunch of Apache logs. The sites are divided up in /var/log/httpd (eg. /var/log/httpd/site1/access.log, /var/log/httpd/site2/access.log). It would be amazing to set the base folder, the max depth, the allow list, and the host segment of the path. Is there a way to read in the path of the file to set the host field? Currently doing this in Splunk with the “host_segment” function in the inputs.conf file.

OK so I am trying to pull from the Microsoft Graph API. To do that, I must first POST to get a bearer token. I assume I can use discover type of HTTP Request to get it. A JSON response will be returned with the field access_token containing what I need.How do I use the data from access_token as a collect header for Authorization? This is a first for me so I need some hand holding here.Thanks!

Hi, we use Cribl 4.0.3 in our environment and have currently issues within a stable connection to Splunk Agents within version 9.0.2.It seems like, that the connection is up and dropping after some minutes.Could it be a possible error with the s2s protocol?In Sources config we can use v4 (beta). Does it work properly? Splunk says in their release document that v3 will be deprecated soon.

Hi,I have question about Cribl monitoring. When I go to Monitoring → Overview in Cribl GUI, I can see data (Free Memory, CPU Load etc.) that is at most 1 day old, because time picker not allows choose longer period. Is there any way how to see older data - for example for 1 week back? And is there any way for other monitoring dashboards with the same time picker setting? Thanks for any hint.Best regardsLukas Mecir

Hello community,we have a bunch of workers in different network zones with different sources (syslog,s2s,tls etc.)How would you prefer to proceed with the groups management?We got 2 options here:All workers in a single group with all sources (if you deploy a new source, all workers need a restart? Not good at all)A group for every network zone, with a worker, with dedicated sources (we think it is really uncomfortable to work with a lot of groups#, because every group has its own configuration)Thanks for feedback and suggestions

In regard to workers in a distributed environment, is there any recommendation on the Network bonding configuration? Would mode 5 balanced-tlb be a better choice than mode 1 active-backup?Thanks

our cribl_pipe field being a json array and they don’t know how to use splunk to mvexpand them, instruct them to use the flatten function instead in a post-processing pipeline.I have multiple values in the cribl_pipe field for my events. How do I correctly process those in Splunk when the events are sent downstream? I’m trying to use mvexpand but it’s not working as desired.

Hi, is it possible for the REST collector to save a checkpoint (e.g. sequential ID of last collected event) and use this as a starting point at the following execution?Something like Splunk’s DB Connect “growing column” setting.

When reviewing license usage in an Enterprise cribl.cloud instance, it’s been noticed that the license page only shows 1TB available, even though more has been provisioned to the Org.

In a passthru route, Stream is adding the Cribl Worker ip address into the host field before sending to Splunk. How that can be avoided? This fields already comes from the source with different values.

Hello,we want to copy all routes from one group to another including all the comments on the same stage. We tried the following ways:Copy and Paste via Advanced ModeCopy and Paste the routes.yml via Shell to the other groupIn Advanced Mode export and then import in the other groupIn Advanced Mode export and then import in the other group on another stageNone of this ways were copying the comments. Is there a working way to copy the routes from one group to another?Thank you in advance

I am testing out the database collector in v4.0 (cannot choose v4 in the form for creating a new topic) and running into an issue. Some of this is documented in Slack but I wanted to bring it here as well. I cannot get a connection to an MSSQL server using an Microsoft Domain account (AD). For security reasons, we do not use SQL accounts, we use service accounts created in our active directory domain. I have tried DOMAIN\user in the “User ID” option in the Connection String but it does not work. Without this feature then this collector option is not going to be used. Can someone shed some light on this?

Are there any tricks to parse messagepack data in Cribl? We want to take send large amounts of log data from fluentd, but it doesn’t have a TCP output plugin - just forward (messagepack), http, or syslog.We’re probably going to have to go with http, but the overheads involved and performance hit are quite significant, so it’d be great to use messagepack natively, and save compute on both ends.I don’t see any source type, nor any function under C that looks like it can do the job.I’d be very happy with a breaker, or a pre-processing pipeline on a raw TCP input. There just doesn’t seem to be a native way to handle messagepack.It doesn’t look like any messagepack plugins are installed natively - would it be possible to install a plugin in the workers config somehow?

Hi, we currently receive multiple data types (ad, dns, dhcp) in splunk via one source/port and split them with multiple input apps into multiple indices within the correct data.How can we do that in Cribl? With multiple routes/pipelines to forward the data into the correct indices?Thanks