Symptom
Events are being dropped at a high rate on a UDP syslog source. The source > Status page shows a large number of events received and dropped, but very few buffered. There are no errors or warnings in the source logs and only the destination logs show errors such as “connection timeouts”, "Sender is at capacity," or “socket hang ups”. Live capture on the source may show zero events during active drops.
Environment
- Cribl Stream
- UDP syslog source
- One or more connected destinations with backpressure behavior set to
block
Resolution
- Check the destination logs for errors such as “connection timeouts”, "Sender is at capacity," or "socket hang up."
- Identify which destinations are unhealthy or unreachable.
- For any destination that is completely unreachable (e.g., connection timeout), disable the associated route to provide immediate relief. Navigate to Routes, find the route, and toggle it off. Commit & Deploy.
- If the deployment includes a large number of worker processes, allow several minutes for the config to fully propagate before verifying the drops have stopped back on the syslog source Status page.
- If drops persist, check remaining destinations for backpressure warnings (e.g., "Sender is at capacity"). Tuning options include:
- Increase the destination's flush period (e.g., from 1 sec to 5 sec) to batch more events per request.
- Increase the destination's request concurrency if the downstream endpoint can handle additional load.
- Have the customer engage their networking team to resolve connectivity issues for any disabled destinations. Re-enable routes once connectivity is confirmed.
- Consider enabling Source Persistent Queue (sPQ) on the UDP syslog source as a long-term safety net. This buffers events to disk instead of dropping them during future backpressure events.
Cause
When a destination's backpressure behavior is set to block and when that destination cannot keep up or is unreachable, a backpressure signal is sent upstream to the syslog source, telling it to stop accepting new data - which is true for TCP-based sources as backpressure propagates upstream and tells the sender to slow down. However, UDP senders are connectionless so they cannot be told to slow down. So when the data path is blocked by an unhealthy destination, the UDP source has nowhere to buffer incoming events and drops them.
This effect is compounded when a single source sends to multiple destinations. If any single destination is unhealthy, it can stall the entire pipeline.
There is no explicit "dropping events" log on the source because the drops are a side effect of the data path being blocked, not a source-level failure.
Last Validated
4.17.0