Skip to main content

Community Discussions

Do you have a question about a Cribl product? look no further

766 Topics
F
Florian UtztNew Participant
asked in Stream
REST Collector - complete variable reference

Hi everyone, I can't find a complete list of available variables to use in the REST collector configuration. I only know those that are embedded in the documentation (for example, ${earliest}). Is there a complete reference table which I may have overlooked?

02
F
F
Florian UtztNew Participant
asked in Stream
REST Collector Strategies - pull all events since last run

Hi, for most of my SaaS services, I have to use the REST collector to pull events periodically, as they do not support sending their events directly into Cribl. I wonder what the recommended strategy is to ensure that all events are getting collected. What schedule is common to run the REST collector on? every few hours or rather minutes?Most APIs support giving a "from" timestamp parameter - but what variable can I use within Cribl to reference "the last run time of a job" ?In general: Is it better to give the API precise instructions beforehand, so it will not return events already collected, or is it better to just collect duplicate events and then drop them in Cribl (I did it with a Suppression rule, but the thresholds need to be very long)Thanks a lot.

01
J
R
Robbert HinkInspiring
asked in Stream
What does an "Unsupported op-code 249" mean?

Why do we see this message when using Cribl v4.4.0 with S2S V4 and Splunk forwarder version 9.1.x:{"time": "2023-12-19T15:16:15.549Z","cid": "w0","channel": "input:in_splunk_tcp","level": "error","message": "closed connection","src": "10.10.10.10:62453","error": {"message": "Unsupported op-code 249.","stack": "Error: Unsupported op-code 249.\n at g._processData (/opt/cribl/bin/cribl.js:15:15721335)\n at g._transform (/opt/cribl/bin/cribl.js:15:15717080)\n at Transform._write (node:internal/streams/transform:175:8)\n at writeOrBuffer (node:internal/streams/writable:392:12)\n at _write (node:internal/streams/writable:333:10)\n at Writable.write (node:internal/streams/writable:337:10)\n at TLSSocket.ondata (node:internal/streams/readable:766:22)\n at TLSSocket.emit (node:events:513:28)\n at TLSSocket.emit (node:domain:489:12)\n at TLSSocket.<anonymous> (/opt/cribl/bin/cribl.js:15:12649570)\n at addChunk (node:internal/streams/readable:324:12)\n at readableAddChunk (node:internal/str

01
B
K
KalebNew Participant
asked in Stream
Data cloned on F5 and sent to Cribl workers

Posting this to get community feedback and track our progress in case anyone encounters this issue in the future.We are attempting to use the data cloning functionality on our F5 load balancer to duplicate data to our Cribl workers. We want to do this so that we can route, filter, and reduce production data in Cribl without impacting production operations. When the time comes, Cribl will be ready for the cutover without much impact at all, hopefully.Right now, the F5 is sending data to our syslog servers, which then send to Splunk. The end goal is to replace syslog with Cribl, but we want to be able to see all the data in Cribl before then so that we can operate on it. We are successfully cloning data that comes from Splunk UFs and HFs, but the cloned syslog data from the F5 is not being seen by the Cribl worker.The specific problem is this: The data is making it to the to the correct interface and port on the worker, but is not being picked up by Cribl for whatever reason.I have verif

02
J
U
User1Participating Frequently
asked in Stream
Error: Initialization error: bind EADDRNOTAVAIL

Hello, has anyone encountered this error? i'm trying to forward my firewall logs to Cribl.I added Syslog Push via QuickConnect. Address: is public IP address of the firewallUDP port: 1514On my firewall Settings:Syslog Server Address: is the Ingress IP of Cribl CloudPort: 1514Are these the correct settings?

211
J
B
Brian YearwoodInspiring
asked in Stream
What do the different fields mean in for the info events for the channel func:dns_lookup

channel: func:dns_lookupcid: w11cribl_postproc_pipe_ts: 1702407784.731hits: 1399991keys: 3ksize: 50level: debugmessage: Cache statsmisses: 9time: 2023-12-12T19:03:04.731Zvsize: 240

00
M
K
kgouldsk@live.comNew Participant
asked in Stream
Replication functionality of AD Group filtering of subscriptions

What is the recommended method of restricting groupings of clients/servers to accessing event collector subscriptions, absent support for computer group control that WEC has?I thought about creating additional listeners and via GPO pointing particular groupings of clients at particular listeners. Is there a better alternative?

02
K
J
Johan WogerInspiring
asked in Stream
what does "Cannot advance offset 2023514992 past end of event 74" mean?

Would it be possible to confirm:What does Cribl do with the events when this issue occursWhat is the maximum character limit CRIBL can parse

01
B
J
John PondromParticipating Frequently
asked in Stream
Bootstrap command has HTML character codes

Why does bootstrap command have HTML character codes?

02
J
F
Franky LaaritsEmployee
asked in Stream
Uneven worker load distribution when using Kafka

Why do I see some worker processes having a higher usage than others for Kafka sources or destinations?

00
B
O
Oliver HoppeEmployee
asked in Stream
Opening Worker Group configuration gives error

I am seeing this error when trying to access the worker group configuration from the UI:“The Config Helper service is not available because a configuration file doesn’t exist or the settings are invalid. Please fix it and restart Cribl server”:

01
R
B
Brian YearwoodInspiring
asked in Stream
License shows old and wrong email address

In red below, the license still shows an old email address. This person is no longer part of our company and we want to replace the email address:

00
R
J
Jonas LinkNew Participant
asked in Stream
Cribl to Sentinel Connection - What is the Client ID for?

Hi all, i'm currently trying to configure Microsoft Sentinel as my Destination.I'm stuck configuring the Authentication. I don't really know what the Client ID is and the hover text "JavaScript expression used to compute the application (client) ID for the Azure application. Can be a constant." isn't very helpful. I have the Application ID as a mix of letters and numbers, but I can't insert it in this form? Can somebody help? Thanks :)

01
J
T
tpevelerParticipating Frequently
asked in APIs
Is there any way I can automate refreshing the access tokens in Cribl?

Hey allI am using a REST API to collect logs using GCP OAuth token. As the authorization token expires every hour, is there any way I can automate refreshing the access tokens in Cribl

06
J
D
Dinesh RajaNew Participant
asked in Stream
Cribl S3 Collector collect All the log files from S3 bucket everytime instead only new ones

Hi Mates, I'm using cribl S3 collector to collect the logs from AWS S3 bucket. S3 bucket contains logs of Akamai datastream which has the log format as below.eg: s3://BUCKETNAME/APPNAME/ENV/ak-913478-1701139960-008071-ds.gzak- akamai file prefix913478 - random string170xxxxxxxx - EPOCH timestamp008071 - random stringds- file suffixWhen I schedule the S3 collector to run every 15 minutes, it collects all the log files from buckets every time. Looking for a suggestion, how do I collect only the new files from S3 not all the files every time.

03
P
R
Radek FilipNew Participant
asked in General
File Monitor sending data irregularly

Hi team, I need help with File Monitor source strange behaviour. Configured FM source (Cribl Stream single instance version 4.0.3) for one log file containing logs from shell backup script running once every day at 7PM. Backup script produces 6 lines/events during its operation. No advanced option used here, manual discovery with allowlist specifying log file name and Collect from end checked. Quick connect through simple pipeline adding metadata (index, sourcetype) and sending events to Splunk single instance destination. This setup works fine most of the time, 6 events read and sent to Splunk every day correctly. But from time to time it happens that at scheduled time not complete content of the log file is read and sent. The remaining lines from one day are then sent and ingested to Splunk together with the events from next run. It happens irregularly. I can not find any single reason for this, any idea? Original message in Cribl Community Slack Message:https://cribl-community.slac

01
K
P
Pawel KwiatkowskiInspiring
asked in Stream
SNMP source - SNMPv3 engine id

Why an engine id is not included into a configuration for decrypting SNMP traps?

01
K
A
Andrew FarmerNew Participant
asked in Edge
Cribl Edge on Windows - How to handle proxy?

I deployed Cribl on some Windows Servers. They are routing events to an on-premise Cribl Stream instance. Everything was working fine, until collection just stopped. It appears that the Edge agents aren't connecting to "https://cdn.cribl.io/telemetry". These servers are behind a web proxy. The proxy settings are applied properly in Windows.If I pull up a browser on the server, and surf to https://cdn.cribl.io/telemetry, I get the expected output --- “cribl /// living the stream!”I have tried:Manually configuring the proxy settings in Windows itself rather than auto-config.Defining actual environment variables for “HTTP_PROXY” and “HTTPS_PROXY” within Windows Server.Tried flipping the Cribl Edge “default_fleet” toggle for “Proxy” -> “use environment variables” on and off.I have manually restarted the services and the Edge agent itself multiple times. Nothing has worked. What is the proper combination of settings in Windows and Cribl Edge to force the agent to use the proxy to phone h

04
A
H
Hillary MasciaveNew Participant
asked in General
Has anyone tried to onboard OneDrive Logs?

We are following this document, https://docs.cribl.io/stream/usecase-rest-ms-graph/ We have tested bringing in Azure User and Device data in the past with no issue. We are having a problem figuring out the best Collect URL to use. We've tried many different ones. We've tested our URLs to Microsoft's Graph Explorer and we are not receiving the expected output in Cribl. We have also set all the permissions that a Splunk App would require, https://docs.splunk.com/Documentation/AddOns/released/MSO365/ConfigureappinAzureAD

02
H
I
itsjustjordynNew Participant
asked in General
I have several questions regarding Stream deployed in AWS...

How would maintenance impact data processing, etc. (i.e. for scale-up procedure)?AMIs must be upgraded every X months; how does that impact the application (Stream)?Have you seen any auto-scaling ? How does scaling down impact the application? Will data be lost or will it complete its processing before shutting down?Concerned about what happens to data in flow when instances are downLooking for details on public cloud use casesFor the infrastructure scaling sideAfter disaster recovery, how do they switch back to the original state?

01
J
J
Joel YueNew Participant
asked in Stream
Cribl on AWS Fargate

How do i calculate the amount of workers i need if i am using AWS Fargate arm processors? I am looking at 7tb/day input and output to 2 destination which is a total of 21tb/day throughput.Thanks

01
J
J
JasonRNew Participant
asked in Stream
Palo Alto pack host headers

I'm quite new to Cribl, so please forgive me if I've got some easy and/or dumb questions!The first thing the Palo Alto Pack pipelines all do is extract the host from an RFC3164 formatted syslog message with a BSD time format.Eval: host = _raw.match(/[A-Z][a-z]{2}\s{1,2}\d{1,2}\s\d{2}:\d{2}:\d{2}\s([^\s]+)\s/)[1] || hostOurs are sending in RFC5424 w/ Octet count framing (not syslog'y enough to know if that's RFC6587 or not), so the host extraction fails and ends up sending to splunk as the IP.With the desire to use the Palo Pack, what is the best way to handle this?Right now I created a pre-processing pipeline and attached it to my source. All it does is extract the host from the header. That way I don't have to handle it in every pipeline inside in the pack.Also, is there a reason you cannot connect a pre-processing pipeline to a source that's inside a pack? In this case, it would be nice to be able to package my paloalto_syslog_pre-processing pipeline inside our customized PaloAlto

03
J
S
Santhosh MParticipating Frequently
asked in Search
How to skip the default parsing of JSON input to the HTTP(S) Cribl source in Streams?

Dear all,Coming up with yet another question as I couldn't find similar from the history of available requests. I have created the HTTP RAW Source and able to push the data and process it in the pipeline. However, I have observed that when I POST the valid JSON format(all the attributes with double quotes), Cribl http source is parsing them into individual fields after the source process, and if the posted content is not JSON parsable, then the content is available under the key "_raw".As per my requirement, I would like to keep the input JSON object as is, since I was trying to validate the whole json with the schema stored in the Knowledge. If it's not getting parsed then I was able to run my use case as expected. In other case where the JSON is parsed into individual fields, I couldn't find any internal key which holds all these variable to format them back to JSON object to validate against the schema.Hence, reaching out for any suggestions in this. Thanks in advance!NOTE: Would be

04
S
B
benjamin.raderNew Participant
asked in Search
Using JSON Paths

I'm attempting to extract values from a JSON string field. However, it seems, that none of the below methods are working for referencing or obtaining the value using a JSON path or dot notation type of approach. I've even tried the "extract_json" function, but to no avail.KQL in Azure has the bag_unpack function, but I noticed this is not supported in Cribl Search.Below are my queries and results.| extend json=parse_json(_raw)| extend test1=_raw._raw.kubernetes.container_name| extend test2=json._raw.kubernetes.container_name| extend test3=json['_raw']['kubernetes']['container_name']| extend test4=extract_json("$._raw.kubernetes.container_name", json, typeof(string))| limit 10

02
B
C
Christian SchmidtNew Participant
asked in Stream
What value would Stream provide to someone who has logstash for routing/transforming?

So, this question has been bothering me for quite some time now. While I am a big fan of Cribl and I really enjoy working with their products and showing/explaining them to others I still wonder every now and then what value Stream would provide to a customer, who already has a well-maintained and functioning logstash for routing/transforming data.If I think about it the following points come to my mind, but if someone here has more/different reasons I would be glad to hear them!Stream's replay functionScalabilityLogstash works as a single instance, Cribl can be clustered to infinityStream is easier to maintainNo grepping around in config files for that one transformation command you need to changePipelines are easier to understand/maintain than logstash filesLess complex to get started with for new users/adminsVisualization of data flows"Debuggability" Being able to look into arriving/leaving data from within the tool without having to restart anything or using tcpdump is incredibly h

00
R

Badge winners

Show all badges
Terms & ConditionsCookie settings