Symptom
- Cribl Stream Worker Processes consume most or all available memory after Suppress functions are enabled or their cache settings increase.
- Worker Processes restart, become unresponsive, or worker nodes become unavailable after memory pressure or an operating-system out-of-memory event.
Environment
- Cribl Stream
- Pipeline Suppress functions
- Distributed deployment with multiple Worker Processes
Resolution
-
Review each Suppress function’s filter, key expression, suppression period, cache size limit, and cleanup settings.
-
Estimate the distinct key count for each key expression during one suppression period.
-
Identify high-cardinality keys, such as source or destination fields with many unique values.
-
Reduce the key cardinality when the suppression requirement allows it.
-
-
Keep the default cache size limit unless required
-
Increase the cache size incrementally in a controlled test when a larger cache is required.
-
-
-
Disable duplicate Suppress functions that run simultaneously in replicated development and production packs.
-
Verify that the Worker Process count and host memory provide sufficient headroom for all stateful functions. Default limit per worker process is 2048 but you need to confirm node has enough total memory to meet that allocation for number of worker processes on the node.
Cause
This can be caused by:
- High-cardinality key expressions that create many in-memory suppression entries.
- Multiple Suppress functions or duplicated packs that multiply in-memory state.
- Suppress state being maintained independently by each Worker Process and not being accounted for.
- Increasing cache limits without measuring cardinality, process count, and host-memory headroom.
Additional Information
- The Suppress Function documentation describes cache size limits, suppression-period timeouts, and cache cleanup settings.
- The Sizing and Scaling documentation explains per-Worker-Process heap memory and how Suppress memory usage increases with key cardinality.
- A larger cache does not provide cross-Worker-Process suppression. A shared-state design like Redis is required when suppression must be coordinated across processes.