Symptom
Data sent through a persistent TCP Destination is not distributed evenly across receivers. One receiver becomes overloaded while others stay underused. This can surface as backpressure, persistent queue growth, blocked sends, or intermittent delivery failures. In some environments the disruption repeats on a fixed interval that matches when the Destination re-resolves DNS.
Environment
- Cribl Stream
- Any persistent TCP Destination, including TCP JSON, Cribl TCP, Syslog (TCP), and Splunk Load Balanced
- A downstream receiver pool, such as a Cribl TCP Source, Syslog receiver, or other persistent TCP listeners
- A Destination address that relies on round-robin DNS, a single rotating A record, or another connection-level IP rotation as the balancing method
Resolution
Option 1
Use Cribl's built-in Destination load balancing rather than relying on DNS rotation to spread long-lived TCP connections.
- Open the affected TCP Destination and enable the Load balancing toggle in General Settings.
- Under Destinations, give Cribl the full set of receivers. Either add each receiver explicitly in the Destinations list, or point it at a hostname that returns the complete backend set in each DNS answer.
- Enable load balancing even when you target a single hostname. A single hostname can resolve to multiple receiver IPs, and load balancing lets Cribl use all of them at once.
- Avoid DNS patterns that return only one backend IP per lookup and rotate that IP on each TTL. When only one IP is available per resolution, Cribl has only one receiver to balance across until the next re-resolution, so the symptom persists even with load balancing enabled.
Option 2
If you cannot enable built-in load balancing and must balance outside the product, place a Layer 4 (TCP) load balancer or a stable VIP with health-aware backend management in front of the receivers. Round-robin DNS alone is not a reliable balancing method for long-lived TCP connections, because it influences only where new connections start and does not redistribute connections that are already established.
Cause
A persistent TCP connection stays pinned to the receiver that was selected when the connection was established. Round-robin DNS changes where the next connection may start, but it does not move data that is already flowing over an open TCP session. As long as the connection stays up, all of its traffic continues to the originally selected receiver.
When built-in Destination load balancing is disabled, each Worker Process opens a single connection to the configured address and port. DNS is resolved once, at connect time, and the connection stays pinned to that IP for its entire lifetime. There is no periodic re-resolution in this mode — the connection only re-resolves DNS if it drops and reconnects. If the hostname resolves to one IP at a time, traffic stays pinned to whichever receiver that connection selected, which produces the uneven load.
When built-in Destination load balancing is enabled and Cribl can see multiple receivers, Cribl maintains parallel connections, applies per-receiver weights, tracks recent traffic history, and steers outbound data toward the less-loaded receivers so the distribution converges over time. Cribl also re-resolves the configured hostnames on the DNS resolution period (default 600 seconds) to pick up receivers added behind the name.
When built-in load balancing is enabled but DNS returns only one backend IP per lookup, Cribl still has only one receiver for that resolution window. It cannot spread load across receivers it has not been given, so behavior effectively reverts to a single active backend until the next re-resolution.
Additional Information
How Cribl balances across multiple resolved IPs
When load balancing is enabled, Cribl resolves each configured hostname and opens a separate connection to every IP that the hostname returns (one connection per IP, per Worker Process). If a hostname resolves to five IPs, Cribl maintains five connections to that hostname in parallel rather than picking one. Explicitly listed receivers are added the same way.
For each batch of data, Cribl chooses a receiver based on two factors:
- Load Weight — a per-receiver weight (default
1). Cribl normalizes each receiver's recent traffic by its weight, so a receiver weighted2is steered roughly twice the data of one weighted1. A weight of0excludes a receiver entirely. - Recent traffic history — Cribl tracks how much each receiver has recently received and steers new data toward the receivers that are behind, so the distribution converges toward the configured weights over time. A receiver that is currently blocked is temporarily penalized so data shifts to healthier receivers.
Cribl periodically re-resolves the configured hostnames to pick up receivers added or removed behind the name. The relevant Advanced Settings, all left at their defaults for this fix, are:
- DNS resolution period (seconds) (default
600) — how often Cribl re-resolves hostnames and updates the connection set from the returned A records. - Load balance stats period (seconds) (default
300) — how far back Cribl keeps per-receiver traffic history when balancing. - Max connections (default
0, unlimited) — caps concurrent connections per Worker Process; if a hostname resolves to more IPs than this cap, Cribl picks a random subset on each resolution.
This is also why a single rotating A record defeats the feature: if DNS returns only one IP per lookup, Cribl has only one connection to balance across for that whole resolution window, regardless of the settings above.
In plain terms
- Round-robin DNS rotates the target for new connections. It does not move an already-open TCP connection to a different receiver.
- Built-in Destination load balancing works best when Cribl can see or define multiple receivers at the same time. Cribl then opens a connection to each one and uses weights and recent traffic history to balance data across the set.
- If DNS hands back only one receiver at a time, even built-in load balancing has only one backend to work with during that lookup cycle.
Which Destinations support built-in load balancing
The Load balancing toggle and its load-balancing engine are shared across Cribl's persistent TCP Destinations, including TCP JSON, Cribl TCP, and Syslog (TCP), as well as Splunk Load Balanced. The fix described here applies to any of them, not just one TCP Destination type.
Note: The Round-robin DNS checkbox you may see on Cribl HTTP-based Destinations (such as Cribl HTTP, Splunk HEC, Elasticsearch, Loki, and OpenTelemetry) is a separate, HTTP-only option. The raw TCP Destinations covered here do not expose that checkbox; for them, the correct lever is the Load balancing toggle and the load-balancing Advanced Settings.