Click here to view the original link.
Hello, Looking for a sanity check on a firewall log aggregation issue I ran into. Setup: Cribl Stream, distributed on-prem, 2 primary FW workers running an Aggregations function to correlate Fortigate FW logs before sending to Splunk. Spec I was aiming for: aggregate events sharing Source IP, Destination IP, Source Port, Destination Port, and Action within a 1-2 second window, with an
Event_Count field showing how many raw events were combined.
While validating against the original raw index, I found a case where my aggregated output showed Event_Count = 2 for a given 5-tuple, but the raw index actually had 11 matching events — with a burst of at least 3 events all within a few seconds of each other (well under any reasonable gap threshold), not 10+ seconds apart as I'd assumed.
Root cause I landed on: two compounding issues —
1. My Aggregations Time window was set to 1s, way too short relative to a multi-second continuous burst, so a single burst was getting chopped into several tumbling windows.
2. The 2 primary workers each run Aggregations independently (shared-nothing per Worker Process), splitting one correlated burst into partial aggregates across both workers, with no merge step downstream.
My fix: consolidate FW log routing so all events for these flows hit a single worker (removing the cross-worker split), and set Time window: 2s, Lag tolerance: 5s, Idle bucket time limit: 10s, Flush on stream close: OFF.
Question for the community: for firewall log correlation like this — same 5-tuple, tight time proximity — is single-worker consolidation + tuned tumbling window the standard approach, or does anyone handle this differently? Want to make sure I'm not building around a limitation there's a cleaner native pattern for. Thanks!