QRadar Syslog Connections Close When the Per-Host Limit Is Reached with Cribl Stream

Symptom

Cribl Stream reports "Other party ended the socket connection" for TCP Syslog Destinations sending events to QRadar. QRadar closes connections shortly after the TCP handshake, causing repeated destination errors.

Environment

• Cribl Stream with a TCP Syslog Destination
• IBM QRadar Event Collector receiving TCP syslog

Resolution

1. Review QRadar logs for connectionsPerHost[<current-limit>] maximum [<current-limit>] reached.
2. Verify that the logs also report dropping connection.
3. Verify that the logged client IPs belong to the Cribl Stream Worker hosts.
4. Identify the number of concurrent TCP connections from each Worker host.
5. Configure QRadar’s Max TCP Syslog Connections Per Host above that connection count.
6. Save the QRadar configuration.
7. Monitor QRadar logs for new connection-limit messages.
8. Monitor Cribl destination metrics for recurring connection errors.

Note: In this particular instance, increasing the QRadar value from 10 to 20 resolved the issue. Do not treat 20 as a universal maximum.

Cause

This can be caused by:

• QRadar’s Max TCP Syslog Connections Per Host being lower than the number of concurrent connections from a Cribl Worker host.
• QRadar dropping new TCP syslog connections after the per-host limit is reached.

Additional Information

The Cribl Stream Syslog Destination documentation describes the connection limit as applying per Worker Process. Size QRadar’s per-host limit for the aggregate connections from each Worker host.