Skip to main content

Cribl Stream Worker Stuck on Config Deploy with Lost Leader Connectivity and Destination Backpressure Due to Host-Wide DNS Failure

Jessica Bracken

Symptom

A Worker (or several Workers in the same group) shows what looks like three unrelated problems at the same time:

  1. One or more Destinations report backpressure, for example:
    {"level":"error","message":"dns lookup failed","ioType":"destination", ... "err":{"message":"getaddrinfo ENOTFOUND <destination-hostname>","code":"ENOTFOUND","syscall":"getaddrinfo"}}
    with recurring Sender is at capacity / Sender has too many buffers waiting for retry entries.
  2. A commit/deploy to the Worker Group gets stuck in an "updating" state for an extended period (an hour or more), while other Workers in the same group update normally within minutes.
  3. Restarting the Cribl process on the stuck Worker does not restore its connection to the Leader — in some cases the Worker drops out of the Leader's online Worker list entirely after the restart.
  4. No events are reaching the affected Destination(s) from this Worker, even though the pipeline and Destination configuration have not changed.

Taken individually, each symptom suggests a different investigation path (Destination health, a hung deploy, a dead RPC/Teleport path). They are usually the same root cause.

Environment

  • Cribl Stream (observed on 4.17.0 Worker / 4.20.1 Leader-managed config; applies to any supported version)
  • Distributed deployment — on-premises Workers, with either a Cribl.Cloud-managed Leader (hybrid) or a self-managed on-prem Leader
  • Affected Destination(s) configured with On Backpressure set to block, with no Persistent Queue enabled

Resolution

  1. On the affected Worker, grep cribl.log (and the per-Worker-Process logs under log/worker/<n>/cribl.log) for getaddrinfo ENOTFOUND and "message":"dns lookup failed".
  2. List every distinct hostname that appears in those errors. In a host-wide DNS outage, the same Worker fails to resolve several unrelated external hostnames at once, for example:
    • The Leader hostname (*.cribl.cloud, or the self-managed Leader's hostname)
    • The hostname backing the config bundle download (for Cribl.Cloud, an S3 bucket such as bundle-*.s3.<region>.amazonaws.com)
    • The hostname behind one or more Destinations
    • diag-upload.cribl.io (used when sending a diagnostic bundle to Cribl Support)
  3. If two or more of these otherwise-unrelated hostnames fail identically in the same time window on the same host, treat this as a network/DNS-layer outage, not a Cribl configuration or pipeline problem. Do not spend time tuning individual Destinations or pipelines — fix the resolver path first.
  4. From the affected host itself (not from a jump box or another Worker), confirm independently with nslookup or dig against the same set of hostnames: the Leader domain, the cloud storage domain backing the config bundle, the specific Destination hostname(s), and diag-upload.cribl.io.
  5. Engage the network/security team responsible for the host's DNS resolver, egress path, or proxy, and ask what changed around the time the failures started — resolver configuration changes, firewall/egress rule changes, and proxy changes are the most common causes.
  6. Once DNS resolution is confirmed restored (the nslookup/dig checks above succeed from the affected host), no Cribl-side configuration change is required. The Worker reconnects to the Leader and resumes sending to the affected Destination(s) on its own.
  7. Only restart the Cribl Worker process if it still does not reconnect a few minutes after DNS is confirmed healthy. Restarting while DNS is still broken will not fix anything, because DNS resolution happens at the network layer, underneath the application restart.

Workaround

There is no Cribl-side workaround for the DNS outage itself. To reduce the blast radius of a future resolver outage, consider:

  • Enabling a Persistent Queue, or switching the affected Destination(s) from block to drop for On Backpressure, so a single unreachable Destination does not stall the whole pipeline route it shares.
  • Keeping Worker/Leader versions in sync, so that once connectivity is restored, no separate version-skew issue complicates the next config push.

Cause

A DNS resolution failure on the Worker's host (or on its upstream resolver/egress path) breaks name resolution for every external hostname the Worker process needs, not just one. This single failure surfaces as three apparently separate symptoms:

  • Stuck config update: the Worker cannot download the config bundle because it cannot resolve the Leader's hostname or the hostname of the storage location backing the bundle. It is not hung — it has no network path to the config source. A Worker in the same group that still has a cached, not-yet-expired DNS resolution may continue updating normally while others fail, which can make the problem look host-specific rather than network-wide.
  • Lost Leader connectivity that a restart does not fix: the heartbeat/RPC channel to the Leader cannot be established for the same reason. Restarting the Cribl process re-initializes the application layer only; it does not touch the underlying DNS resolution, so the reconnect attempt fails for the identical reason the config update was stuck.
  • Destination backpressure: when a Destination is configured with On Backpressure set to block and no Persistent Queue, losing the ability to resolve that Destination's hostname causes the Destination to block. With no PQ to absorb the backlog, that blocked condition propagates back through the pipeline, surfacing as backpressure on the route(s) feeding that Destination.

Because all of these trace back to the same unresolved hostnames on the same host at the same time, they are one incident, not three independent bugs.

Additional Information

Terms & ConditionsAccessibility statement