We have a source coming in that consists of many different sourcetypes all under the same sourcetype, and we want to assign sourcetypes in a pipeline. E.g. an event starts with `2023-06-15T00:00` it's going to be sourcetype A, another one starts with `INFO 06/15/23` that's sourcetype B and so on. How would I best set up the functions in the pipeline, so that when one of the assignments was made the others aren't tried (for logical and performance reasons - an event can only ever have one sourcetype)? I guess we could make the function final, but then other functions (after the sourcetype has been assigned) aren't applied either, e.g. aggreations on some of these sourcetypes. Should I create two pipelines, one just for assigning the sourcetype with final functions, and then a follow up pipeline for further processing?
Question
Optimize sourcetype assignment in pipeline
Do that in the event breakers and just add sourcetype fields at the bottom of the ruleset.
Why do you recommend that?Unfortunately, some of the sourcetype assignments depend on parsing fields from json first
Use Filters at the Route and a pipeline for each type of event?
can do that by checking for lack of sourcetype. i.e., if the event has a sourcetype that processing (in that Function) will be skipped
The events come in with a sourcetype already, but yeah good idea to check existence of more specific sourcetype!
Sign up
Already have an account? Login
Login to the community
No account yet? Create an account
Using your Cribl Curious or University Account
User Login Employee loginEnter your E-mail address. We'll send you an e-mail with instructions to reset your password.