Symptom
UDP syslog traffic distributed through an F5 BIG-IP appears unevenly balanced across Cribl Stream worker nodes. Some workers receive significantly more traffic than others, and per-node event counts (on Cribl side, not F5 side) may appear lower than expected when monitoring individual workers after a load balancer change.
Environment
- Cribl Stream (any version)
- F5 BIG-IP as a UDP load balancer in front of Cribl Stream worker nodes
- UDP syslog sources
Resolution
Enabling datagram load balancing with an immediate idle timeout causes F5 to distribute each UDP datagram independently, similar to per-packet round-robin. This appears to be the correct mode for stateless protocols like UDP syslog.
Setting Idle Timeout to Immediate is the primary driver of the fix. The immediate timeout removes session affinity entirely, preventing the BIG-IP from pinning traffic from a persistent UDP sender to a single pool member. Enabling Datagram Load Balancing alongside it ensures each datagram is individually load-balanced, equivalent to per-packet round-robin.
Cause
By default, F5 BIG-IP UDP profiles use flow-based affinity. With datagram-load-balancing disabled and idle-timeout 60, the BIG-IP treats all UDP packets from the same source IP and port as a single connection for the duration of the idle timeout (60 seconds). For persistent UDP syslog senders, which reuse the same source IP and port continuously, this pins all traffic from a given sender to one pool member for the entire timeout duration. Round-robin rebalancing cannot occur until the flow expires.
The result is that one Cribl worker node receives the majority of traffic from high-volume senders while other nodes in the same worker group receive little or none. This produces the appearance of uneven load or dropped events when monitoring per-node counts.
Setting datagram-load-balancing enabled with idle-timeout immediate removes the session-affinity constraint. Each datagram is load-balanced independently, producing even distribution across all pool members regardless of source IP and port persistence.
Additional Information
When switching from a session-affinity UDP profile to a datagram load-balancing profile, per-node event counts will decrease if multiple workers were previously sharing traffic. This is expected redistribution, not event loss. Confirm by checking the worker group aggregate in the Cribl Stream monitoring UI, the total group event count should remain stable.
If individual worker nodes are still showing high packet drop rates after enabling datagram load balancing (visible as elevated UdpRcvbufErrors in /proc/net/snmp on Linux hosts), the worker VM may be undersized for the traffic volume it is receiving. Increasing VM resources or adding additional nodes to the worker group will reduce per-node traffic density.