Persistent Queue Does Not Drain When Ingesting UDP Syslog with Cribl Stream

Symptom

Persistent Queue (PQ) on one or more worker nodes is not draining efficiently after upgrading Cribl Stream. One worker process accumulates significantly more data (100s of GBs) in its PQ than others, and events appear in downstream destinations (e.g., Microsoft Sentinel) with a delay.

Environment

• Cribl Stream 4.17.x, 4.18.x (Self-Managed)
• Linux (RHEL 9.x)
• Syslog source receiving both UDP and TCP traffic on port 9514
• Multiple worker nodes behind an F5 load balancer
• PQ mode: Smart or Always On
• High-volume Palo Alto firewall syslog traffic over UDP

Resolution

1. Switch high-volume syslog traffic (e.g., Palo Alto firewalls) from UDP to TCP on the source device side. The existing syslog input already listens on both TCP and UDP, so no new input is required.
2. Set TCP Socket max lifespan to `600` seconds (10 minutes) in the syslog source Advanced Settings to prevent TCP pinning (one worker monopolizing all TCP connections):
   • Sources > in_syslog > Advanced Settings > TCP Socket max lifespan: `600`
3. Disable Proxy Protocol if it is not being used by the load balancer:
   • Sources > in_syslog > Advanced Settings > Enable proxy protocol: Off
4. Enable TCP load balancing on the source:
   • Sources > in_syslog > Advanced Settings > Enable TCP load balancing: On
5. Set PQ mode to Smart or Always (note: Smart mode is being deprecated; Always On will become the only option in future releases).
6. Optionally set file size limit to `10MB` in the source settings.
7. Monitor the following after the change:
   • Worker distribution (traffic should be more evenly spread)
   • PQ drain behavior
   • Dropped events
   • Destination ingestion volume

Cause

The root cause is uneven UDP traffic distribution across worker processes, compounded by erratic and increasing data volume.

How UDP Distribution Works at the Kernel Level

UDP data is received into an OS/kernel-level buffer. Multiple processes may technically be able to read from that buffer, but the kernel does not evenly distribute read notifications across all processes. Instead, the kernel often wakes up the same process repeatedly using soft interrupts. As a result, one or two worker processes may receive most of the UDP data while others receive little or none.

Why this happens:

The kernel/network stack uses soft interrupts to notify processes that UDP data is ready to read. For efficiency, the kernel tends to wake the process that was last active on the CPU associated with that NIC queue. The chain is:

1. UDP data arrives on a network interface.
2. The NIC places data into one of its internal queues.
3. A CPU is associated with that queue.
4. The kernel chooses a process that was recently running on that CPU.
5. That same process gets interrupted repeatedly to read more UDP data.
6. That process accumulates more data/PQ than others.

The desired behavior would be to spread UDP data more evenly across worker processes, but Cribl Stream does not have direct control over this behavior because it is determined by the kernel. This is not a defect in Stream or the OS — it is by design in the kernel. With a single UDP input/port, there is no mechanism to guarantee fair distribution.

What the Diagnostic Data Showed

Based on diagnostic data reviewed, the persistent queue growth was strongly influenced by how UDP traffic was being distributed across worker processes.

Incoming UDP traffic was not being evenly distributed across all worker processes on the affected node. Instead, one worker process on a worker was consistently receiving a much larger portion of the traffic than the others — often processing around 90–120 MB per minute while other workers received very little.

Because that worker process was continuously receiving a high volume of new incoming data, it did not get enough opportunity to fully drain its existing queue. The process was draining data, but not fast enough to overcome the combination of the existing queue backlog and the ongoing incoming traffic volume. The queue would decrease during lower-volume periods, but begin increasing again when traffic levels remained steady or rose.

The queue behavior also correlated with traffic volume patterns:

• When total incoming volume decreased, the queue began to drain.
• When traffic volume remained high or did not drop significantly, the queue continued to grow or failed to recover fully.

This indicated that the issue was not that PQ draining was slow (internal tests showed no difference in PQ draining speed between 4.16, 4.17, and 4.18), but rather that worker 5 simply had 100s of GBs of data in its queue. In combination with how much data it received in real-time, the total volume was too much. Constant PQ usage requires constant CPU involvement, which slows down the process when also needing to push events through routes, pipelines, etc. Other processes performed better simply because they had much less data volume.

Confirming the Shared Socket

All worker processes on a worker shared the same UDP socket inode, confirmed by inspecting `/proc/<PID>/fd/` entries:

ls -l /proc/303649/fd/32 # -> socket:[12674974]

ls -l /proc/303674/fd/32 # -> socket:[12674974]

ls -l /proc/303663/fd/31 # -> socket:[12674974]

ls -l /proc/303662/fd/31 # -> socket:[12674974]

ls -l /proc/303651/fd/31 # -> socket:[12674974]

All five wp PIDs pointed to the same socket inode `socket:[12674974]`. This confirms the kernel is making a single distribution decision for all UDP data arriving on that port — there is only one socket, not one per worker.

Why TCP Resolves the UDP Issue

Unlike UDP (where data goes into a shared buffer and the kernel picks one process to wake), TCP connections create file descriptors that are handed off to individual worker processes. The LB process can service TCP connections, break the stream into events, and round-robin them across workers. This provides a more predictable and even distribution path.

Additional Information

Commands Used During Investigation

Check PQ consumer cursor progress (run 6 times at 10-second intervals):

cat `find . -name consumer.cursor`

Check disk I/O to rule out storage issues:

iostat -x 1 30 > "iostat-$(date +%Y%m%d-%H%M%S).out"

Verify all worker processes share the same UDP socket inode:

ls -l /proc/<CRIBL_PID>/fd/32

Check PQ size per worker process:

du -h --max-depth=2 | grep inputs

Other recommended Mitigation Options

There are three primary mitigation options to improve traffic distribution and reduce the likelihood of one process becoming overloaded:

1. Switch high-volume sources to TCP (Recommended)

This is the approach that resolved this issue. TCP provides a fundamentally different distribution mechanism — the LB process round-robins events across workers rather than relying on kernel soft interrupt affinity. For sources that support TCP (such as Palo Alto firewalls), this is the most effective single change.

If switching to TCP is not possible:

2. Split the UDP input into multiple inputs

If TCP is not feasible, splitting the current UDP input into multiple inputs can help distribute traffic more effectively. The split can be based on device type, vendor, source group, or another logical grouping.

The key benefit is that each additional input uses a separate port. With multiple ports, the kernel will need to choose another process as the main process for receiving interrupts for each port. What multiple ports gains is making the kernel have to choose an additional process to favor for receiving the software interrupts for another UDP socket.

This does not guarantee perfectly even distribution, but it improves the likelihood of better spread across processes compared with a single UDP input.

3. Add additional worker nodes

Adding more worker nodes reduces the total amount of traffic handled by each existing node, assuming the load balancer is distributing traffic appropriately. Each node may still have a process that receives more traffic than others, but the total amount of traffic per node should be lower, giving affected processes more capacity to drain queued data.

Combined approach: The best improvement may come from combining approaches — splitting the input improves process-level distribution within the worker group, while adding nodes improves node-level distribution across the environment. Switching to TCP addresses the root cause entirely.

Key Takeaway

TCP provides a more predictable distribution path across workers compared to UDP. For high-volume syslog sources that support TCP (such as Palo Alto firewalls), switching to TCP is the recommended approach to resolve uneven PQ growth across worker processes.