Hello, is there a way to make a variable from one event in a pipeline and add that variable to all events in the pipeline? So if one event has "Name=John Smith" but no other event from that file has the Name field, can I add "Name=John Smith" to all events? I have information at the beginning of a file that has fields I want added to all events, and after that "header" information are the actual events. So if this is something that can be done at the event breaker level then great.
Question
How to make a variable from one event and add that variable to all events in the pipeline?
you'd need to use a redis function for this i think.
which means you'll need a redis server. And cluster, to be honest.
If these events are all coming in together (for example in an JSON that has an array records it contains), you could do the event breaking in a pipeline function (or maybe unroll). But basically you would need the set of events to be together.
I originally read in the whole file as an event, then extracted the beginning variables, then parsed the data from there. This worked, but not for files that get bigger than the event size limit.
Default event size might be kind of low for your needs. Max configurable is ~134 MB/event. Are your events larger than that?
Looking at the files again, I don't think they're bigger than that. I'll test out an event breaker with the max configurable event size
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.