This message originated from Cribl Community Slack.
Click here to view the original link.
Hello, I have configured a rest collector to fetch Fastly WAF requests. the collector is working and getting the correct requests. due to limitations in the API I have to fetch 1000 requests and then utilize a suppression in the pipeline to deduplicate the requests that have already been fetched in prior runs. The problem I am running into is that the collector job is running on both workers in my WG. I tried to disable the source on one of the workers manually but the rest sources are not present in the individual worker UI. does anyone have any recommendations for configuring this so that this collection only runs on one worker? do I need to run this in a worker group with only a single worker?
Solved
Fastly WAF Collector Job Running On Both Workers In Worker Group
Best answer by Jon Rust
Better option would be to limit the API request to only show new events since last retrieval
Better option would be to limit the API request to only show new events since last retrieval
Are you using a discover phase?
No discovery phase. I thought I had ruled out that option but upon further digging into the API there is a querying capability that lets me time filter from last run. Should be able to get this working. thanks!
Collector jobs are designed to run on all nodes in a group much like the other inputs and outputs. The only situations in which this wouldn't occur is if there isn't enough data to be collected and thus tasks for the job to be assigned to all processes or if using the DB collector, which only runs on 1 process each job run.
Sign up
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.