Symptom
After a Cribl Cloud Leader upgrade, Cribl Outpost nodes show as disconnected and all Edge nodes connecting through those Outposts lose connectivity. Downstream log forwarding to destinations may get interrupted for all affected Edge nodes.
Environment
- Cribl Cloud Leader
- Cribl Outpost
- Cribl Edge
Resolution
- Obtain the Leader FQDN for the affected Cribl Cloud organization (format: main-<org-id>.cribl.cloud).
- On each affected Outpost, run the following command to reconfigure the upstream Leader connection to use the FQDN:
su -cribl -c "./cribl mode-outpost -H <leader_fqdn> -p 4200 -u "<authToken>" -S 1" && systemctl restart cribl-outpost - Verify the Outpost reconnects to the Leader in the Cribl Cloud UI under Products > Edge > Outposts.
- Confirm all Edge nodes behind the Outpost have reconnected.
Cause
The Outpost nodes were configured with hardcoded Leader IP addresses in their instance.yml files. During a Cribl Cloud Leader upgrade, the underlying cloud instances are replaced, which rotates the IP addresses. The Outposts could no longer resolve the Leader at the old IPs, breaking the control-plane connection. Since Edge nodes connect to the Leader through the Outpost, all downstream Edge nodes also lost connectivity.
Last Validated
4.17.1
Additional Information
Best Practice: Always configure Outpost nodes using the Leader FQDN rather than IP addresses for Cloud deployments. The FQDN allows DNS to gracefully resolve any rotated IPs during future upgrades.