Cribl Stream Syslog Destination TCP Connection Reset Troubleshooting with Packet Capture

Symptom

Destination health shows intermittent issues and it appears that Cribl resets the connection when sending to a Syslog Destination.

Environment

• Cribl Stream
• Linux
• Syslog (receiving server)

Resolution

1. Setup

Set up a destination on Cribl to a port that is open to the outside.

This test scenario will test just the connection without any payload and check over a long period (a few days) whether either Cribl or the syslog server will maintain the connection even if the payload is zero-sized. At the end, we will also discuss other possibilities such as how the packets would behave if there were a reset from the receiving side, or a box in the middle.

The following JSON can be used to create a destination. Modify the name, IP, and PORT.

{ "systemFields": [ "cribl_pipe" ], "streamtags": [], "protocol": "tcp", "facility": 1, "severity": 5, "appName": "Cribl", "messageFormat": "rfc3164", "timestampFormat": "syslog", "throttleRatePerSec": "0", "logFailedRequests": false, "loadBalanced": true, "connectionTimeout": 10000, "writeTimeout": 60000, "tls": { "disabled": true }, "onBackpressure": "block", "excludeSelf": false, "hosts": [ { "tls": "inherit", "weight": 1, "host": "xxx.xxx.xxxx.xxx", "port": 9514 } ], "dnsResolvePeriodSec": 600, "loadBalanceStatsPeriodSec": 300, "maxConcurrentSenders": 0, "id": "syslog_out", "type": "syslog" }

2. Syslog Side

Next, set up an ng-syslog to listen on the selected port, as specified in the above configuration.

Once the syslog server is up and listening, let's verify the connectivity:

Sample response:

tcp 0 0 0.0.0.0:9514 0.0.0.0:* LISTEN 3779/syslog-ng <<<==== listening to Cribl tcp 0 0 0.0.0.0:6514 0.0.0.0:* LISTEN 3779/syslog-ng <<<<=== listening to Logstash tcp 0 0 xxx.xxx.xxx.xxx:9514 yyy.yyy.yyy.yyy:49318 ESTABLISHED 3779/syslog-ng tcp 0 0 xxx.xxx.xxx.xxx:9514 yyy.yyy.yyy.yyy:37248 ESTABLISHED 3779/syslog-ng tcp 0 0 xxx.xxx.xxx.xxx:9514 yyy.yyy.yyy.yyy:51958 ESTABLISHED 3779/syslog-ng udp 0 0 0.0.0.0:514 0.0.0.0:* 3779/syslog-ng Where: xxx.xxx.xxx.xxx:9514 Syslog server:port yyy.yyy.yyy.yyy - Cribl Worker

To measure over time, we can set up iptables to measure the connection:
sudo iptables -I INPUT -p tcp --dport 9514

Then run:
sudo iptables -v -n -L

Result:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 243K 13M tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9514 243K 13M tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9514 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

You can also run a watch and see the packets flowing every few seconds.
watch sudo iptables -v -n -L

The next step is to collect a tcpdump. Here we use a version that is verbose and decoded that payload to a human-readable form:
sudo tcpdump -i any -AeEnnvvXxSs0 port 9514

The payload:

19:17:33.410445 In fe:ff:ff:ff:ff:ff (oui Unknown) ethertype IPv4 (0x0800), length 68: yyy.yyy.yyy.yyy.49318 > xxx.xxx.xxx.xxx.9514: Flags [.], ack 1, win 502, options [nop,nop,TS val 793969057 ecr 885026467], length 0 E@.4=.@.0.O.....-O....%*g..'............... /S..4.n. 19:17:33.410475 Out f2:3c:91:0d:48:21 (oui Unknown) ethertype IPv4 (0x0800), length 68: xxx.xxx.xxx.xxx.9514 > yyy.yyy.yyy.yyy.49318: Flags [.], ack 1, win 510, options [nop,nop,TS val 885036707 ecr 790261420], length 0 E..4..@.@..?-O......%*......g..(.....%..... 4.../.n. ^C 14 packets captured 14 packets received by filter 0 packets dropped by kernel

Here is what the data collected on the Layer 4 tells us:

• Flags [.] → the “.” means ACK only (no SYN, FIN, RST, PSH in this display).
• length 0 → no TCP payload (no application bytes in these segments).
• ack 1 → relative ack; together with length 0, it means these are pure acknowledgments on an already-established connection (handshake finished earlier).
• options [nop, nop, TS val … ecr …] → TCP timestamps are on; both directions echo the peer’s timestamp (ecr).

So all we see are (two) tiny IPv4 TCP segments (68-byte frames), each direction once, ACK-only, no data.
This is often seen as idle-connection chatter (ACKs after data or stack behavior around keepalives). There are a few other options, but in this instance, it means the two servers talk to each other on an open connection (...indefinitely).

3. Pcap Check for Resets

We also captured a pcap and analyzed it using Wireshark, which lets us follow the connection. This pcap was captured for several hours:
sudo tcpdump -i any -AeEnnvvXxSs0 port 9514 -w on_syslog.pcap

When searching for resets using: (tcp.flags.syn == 1) || (tcp.flags.push == 1) || (tcp.flags.reset == 1) ( Wireshark Article ), we don't find any resets at all:

Other Scenarios to Consider

Reset at the destination or on the box in the middle.

• If nothing is listened to at the destination:
  • We would see the client’s SYN to dst_ip:dst_port, then a quick reply from the same destination IP with RST, ACK (exact flag text varies by OS; often R. or RA in tcpdump).
• If the application (service) rejects the connection:
  • Handshake completes: SYN → SYN-ACK → ACK.
  • After that, you see normal PSH/ACK with length > 0 (payload). The “rejection” is inside the payload, not as a TCP RST.
  • Connection may stay open or close cleanly with FIN later—often still no RST.
• What if the box in the middle, like an LB, FW, etc., resets?
  • RST injected to tear the session down (which might be the case for LBs and Firewalls)
  • The TCP segment with RST (sometimes RST, ACK), source IP, and port chosen to look like one end of the flow so the receiver’s stack accepts it.
• How about Drop/blackhole instead of RST?
  • Re-transmissions (SYN retries, or data/ACK retries), no reply, then eventual timeout—no RST from anyone.
• Reject with ICMP (some firewalls)

Real Server vs Box in the Middle

A reset by the receiving side would look something like this:

Note: These were mocked up; the real payload may look different.

From receiving server: 19:32:33.000001 Out f2:3c:91:0d:48:21 ethertype IPv4 (0x0800), length 54:yyy.yyy.yyy.yyy.9514 > xxx.xxx.xxx..49318: Flags [R.], seq 1, ack 1, win 0, length 0 From sender: 19:32:33.000045 In fe:ff:ff:ff:ff:ff ethertype IPv4 (0x0800), length 54:yyy.yyy.yyy.yyy.49318 > xxx.xxx.xxx.xxx.9514:Flags [R], seq 1, win 0, length 0