ECONNRESET on Cribl HTTP Source Behind Layer 4 Load Balancer Due to Asymmetric Routing

Symptom

Senders targeting Cribl Stream workers through a load balancer receive ECONNRESET errors on any TCP-based destination (Cribl HTTP, Syslog TCP, Splunk TCP, Cribl TCP, etc.). The worker's corresponding source shows zero inbound events despite the worker process being healthy. A tcpdump on the worker shows SYN received → SYN-ACK sent → immediate RST received from the load balancer direction.

Environment

• Cribl Stream — worker nodes behind a Layer 4 load balancer
• Layer 4 LB operating without SNAT/Source NAT
• Linux-based worker hosts with a default gateway that does NOT point to the load balancer
• Any TCP-based source/destination pair traversing the LB

Resolution

Note: The steps below are examples for common Linux distributions. Consult server/network administrators as needed when making system-level routing changes, as procedures may vary by environment and organizational change policies.

1. Check the current default gateway on the affected worker server:

   ip route show default

2. If the gateway points to a network router instead of the LB floating/self IP, replace it:

   sudo ip route replace default via <LB_FLOATING_IP> dev <INTERFACE>

3. Confirm traffic is restored by checking the Cribl HTTP source live data view or re-running tcpdump for complete handshakes (SYN → SYN-ACK → ACK).

4. Persist the route to survive reboots:

   • RHEL/CentOS (ifcfg): Add default via <LB_FLOATING_IP> dev <INTERFACE> to /etc/sysconfig/network-scripts/route-<interface>

   • RHEL 8+ (nmcli): nmcli connection modify <conn> ipv4.gateway <LB_FLOATING_IP> && nmcli connection up <conn>

   • Ubuntu/Debian (Netplan): Set routes: [{to: default, via: <LB_FLOATING_IP>}] in /etc/netplan/<config>.yaml, then sudo netplan apply

5. Repeat for all worker servers behind the same LB VIP.

6. (Alternative) If changing the default gateway is not feasible, enable SNAT on the LB (AutoMap on F5, masquerade on HAProxy). Note: the worker will lose visibility of the original client source IP.

Cause

The worker server's default gateway does not route return traffic through the load balancer, creating asymmetric routing. In L4 LB deployments without SNAT, the LB preserves the original client IP when forwarding to backends. The worker must reply back through the LB so it can track both sides of the TCP session. When the worker's gateway points elsewhere, return packets bypass the LB. The LB considers the connection invalid and sends a RST on the next inbound client packet.

Common triggers: Server reboot wiping a non-persistent ip route replace command, OS patching resetting network config, or subnet migration without updating return-path routing.

Additional Information

Diagnostic tcpdump signature (run on the worker: tcpdump -i any port 10200 -nn):

<edge_ip>.<port> > <worker_ip>.10200: Flags [S] # SYN arrives via LB <worker_ip>.10200 > <edge_ip>.<port>: Flags [S.] # SYN-ACK sent by worker <lb_ip>.<port> > <worker_ip>.10200: Flags [R] # RST from LB direction

LB health checks may mask this issue — probes originating from the LB's self-IP naturally route back to the LB, so pool members appear healthy while real client traffic fails.

This is not a Cribl software defect — it is an infrastructure routing configuration issue.