Cribl Stream Leader Starts in Single Mode After Enabling HA Failover

Symptom

After enabling Resiliency: Failover on a Cribl Stream Leader, the Leader restarts in Single mode and appears to have lost its configuration.

The restart might complete without an obvious error in the Leader logs.

When the configured failover path already contains configuration data, the Leader might instead start with unexpected or fresh configuration, an unexpected license, or without copying the current local configuration to the failover volume.

Environment

• Cribl Stream self-managed distributed Leader
• Leader High Availability with Resiliency: Failover enabled
• Shared NFS failover volume located outside $CRIBL_HOME

Resolution

1. Stop the affected Cribl Stream Leader service before changing configuration files.
2. Verify that the configured failover volume is mounted and accessible to the Cribl service account.
3. Verify that the failover volume is outside the Cribl installation directory.
4. If this is a new HA configuration, verify that the configured failover path is empty before enabling failover.
5. If the failover path is not empty, inspect its contents before reusing it. Existing .git, data, groups, local, or state directories indicate that Cribl may treat the path as an existing configuration.
6. On the affected Leader, inspect $CRIBL_HOME/local/cribl/leader.yml for a stale file from an earlier HA configuration.
7. Back up the stale leader.yml file.
8. Remove or rename the stale leader.yml file according to your change-control procedure.
9. Navigate to Settings > Global > Distributed Settings > General Settings.
10. Select Mode: Leader.
11. Navigate to Leader Settings.
12. Select Resiliency: Failover.
13. Enter the same shared NFS path in Failover volume for each Leader.
14. Click Save to restart the Leader and initialize HA.
15. Verify that the current configuration is present on the failover volume.
16. Verify that one Leader is Primary and the other Leader is Standby.

Cause

This can be caused by:

• A stale $CRIBL_HOME/local/cribl/leader.yml file being read during HA initialization, causing the Leader to return in Single mode.
• An existing or incomplete failover-volume configuration being treated as the source of truth instead of being overwritten with the current local configuration.
• A failover volume that contains .git, data, groups, local, or state directories before HA initialization.

Additional Information

• Leader High Availability/Failover explains the relationship between leader.yml, instance.yml, and the shared failover volume.
• Configure Standby Leader Nodes describes the required Leader settings and states that the failover volume must be outside $CRIBL_HOME.
• Verify the contents of an existing failover volume before enabling or re-enabling HA. Once configuration exists there, treat the failover-volume contents as authoritative.