Community Discussions

Hello All,I am hoping someone can perhaps guide me where I need to go. I am configuring a REST collector and I connect using ClientID and ClientSecret. I would then in return get a token.The problem I have are the fields Client Secret parameter and the Client secret value. I have added the ClientID as a parameter but I am not winning.I can configure this with Postman and I can use curl to get my token. I am not able to figure how to do this in Cribl.

Hi All! When using the azure monitor destination has anyone been able to successfully set a custom timestamp field which gets translated to TimeGenerated within Sentinel? The azure monitor HTTP data collector api documentation indicates this can be set using the "time-generated-field" request header, but we're not having any luck with that so far.

Hi folks, new cribl user here. For the Google Cloud Storage collection (or any collection for that matter), does cribl keep any sort of state regarding files it has seen before? I'm coming from logstash/Splunk apps where they do actually keep track of files that have been indexed via checkpoints, but I didn't see anything like that in cribl - wanted to make sure I'm not missing it

Hello, I'm having issue where S3 source is able to get data through routes but only via passthru. When using a diff pipeline, there destination doesn't get data. I'm not sure where lese to look as the output stat don't change but yet the pipeline stats do.

hi all,I am trying to configure the Auto Timestamp function to extract a time field that can be all over the place within the event. The timestamp itself is in epoch format (start=<time>). I setup an additional timestamp within the function, but I'm not sure what to specify for a strptime format, since it's epoch.

Hello there,I am preparing a POC (on prem) for a customer based on Edge and Stream. I just wanted to know if it is possible to manage an Edge agent from a standalone Stream instance or if is mandatory to deploy a leader node to manage both Stream and Edge instances ?

is it advised to have both a cribl agent and a splunk universal forwarder installed on a host?

it might be a weird question… But as i look at live Data (cribl Stream) (to look at whats coming in the system) i see filter expression to see and capture but is there a quick way to search the data real fast like what kind of keywords across all fields as i dont know what fields there are. like a mini search like | %keyword% or on the fly matching (i have 1000s of new data to hunt for

I've comb the docs for references to IPV6 compatibility. our organization is requiring all our systems to be IPV6 compatible. Is cribl compatible with IPV6 for both sources and destination, and do all input form in GUI allow IPV6 addresses?

I would like to move 15-20 REST collectors from one worker group to another. For routes/pipelines, I just copy and then switch groups and paste but the copy function isn't available in REST collectors. Any suggestions on how to move a REST collector?

Hi All,got a question in regards to Redis.We encounter issues, that filtering isn't working as expected. We use Redis as a whitelist, meaning that we extract a field from the raw log and using the EXISTS command to check if the key (build as a combination of strings) exists in Redis. If it does not exists in Redis the return value is `'0'` and we will drop the event.What we discovered is, that events that should be dropped are available at the destination.My questions are:What is / Is there a return value from Redis in case Redis was not able to process the query (EXISTS command)? Proabably not `'0'` , but if there is a response I somehow need to catch it.That was the only thing that came to my mind could cause the issue.Do you have some other ideas?Cheers,Mario

Hi all,How do I test a set of sample data all the way through it's processing pipeline, which includes pre-processing. I want to see the end result (before destination). I know how to test it against a single pipeline - not sure how to include pre-processing as well.

Hello Cribl community. Anyone had experience on setting up a Cribl journald data source for journal logs on remote machine? Where can I specify the remote host information? Thanks in advance.

OK I have another REST collector question. My collect section I am using POST with body. The body is JSON and it has to provide the time there (start and end). I was trying `${earliest}` and `${latest}` but it does not seem to work. I think it is expecting the date to be in a certain format. This collector is currently a bash script but I was thinking it would work as a REST collector. The bash script sets and start and end variable like this:start="$(date --date=-1days '+%F %H:%M:%S')"end="$(date --date=last-minute '+%F %H:%M:%S')"How can I do this with the REST collector? Ideas?

Hey everyone! General question to see if I'm missing something. We're about to configure a Health check that is going to be ran across dozens of endpoints and will inevitably expand in the future even more. Looking to make sure this is as painless as possible so I'm trying to figure out the "best practice" for doing this. We're essentially monitoring overall health pages exposed by some of our containerized applications. All the urls will be formatted the same (<environment>.http://site.com/status/json.php|site.com/status/json.php). Ideally we'd have a way for this to dynamically update based on some outside source, second best would be for us to maintain a lookup on cribl of all the environments. Just looking for a way that I don't need to manually create sources and rely on my dev ops teams to tell me when new environments are spun up. Any thoughts?

Is there a way to order/sort the JSON keys prior to hitting the destination? I thought prepending the key names with an underscore would bring them on the top but they show up at the bottom of the JSON structure. The ordering looks the way we want it in the pipeline view but come out differently in the actually output.

I want to duplicate a certain set of sourcetypes to a second index (for reasons). I'm typically processesing data through a pipeline and adding my index to the data and then sending off to Splunk via HEC. What's the best way to do this? Do I just need a second route, with a duplicate pipeline configured that then set a different index?

To those of you that are using QRoC/QRadar if I am sending the data to QRoC as simple syslog, on the IBM Qradar side what should I select as a log source type? I went looking through the list and didnt see anything jump out at me though I did see a few that could be considered generic. What I would like to know is what have others used?

REST collector question. I used a standalone test box to get my REST collector working like I wanted. authentication, discover and collect worked great. When I moved it into my distributed environment, it errors out but does pull in some events. The discovery and collect phases are distrubuted across my 5 workers. It seems like the authentication token may not be working properly when it runs on more than one worker? I am guessing of course. I get errors like 503 Service Temporaily Unavailable and 500 Internal Server Error. This does not happen when I run it from my single standalone test server. I may need some assistance trying to figure this one out.

TCP syslog flow to cribl is reporting that TCP window is full

how do people handle secrets that need to be used in scripts that Cribl may run?

Can I create a lookup file from data I receive from a collector? I have two fields in a JSON array that I want to use to create/update a lookup file every time the collector is run.

I'm using a code function in my pipeline for some extra custom logic, but exporting that as yml config is kinda ugly. Is there any chance I can load a .js file up and just tell the code function to run the js file? This would make local testing of my custom js a much nicer experience. I am doing this with samples for my test cases where I have the sample file locally and I load it up to the cribl server, and reference the file in my samples.yml. I'd love to do basically the same thing with a code function.

Need some help with a time issue. The raw event starts with this:03-05-2023 14:40:27 0DF0 PACKETThe time on in (before processing is correct), but post processing is incorrect.Pre-processing is 2023-03-05 (correct) but post processing is 2023-03-05

Hi all,Setting up my first HEC source within Cribl. I'm sending json from Corelight to this new HEC source. Should the HEC services endpoint within Cribl be "/services/collector/event" or "/services/event" ?