Cribl Stream Worker and Edge Disconnects After a 4.20 Upgrade

Symptom

When upgrading a self-managed Cribl Stream Leader to 4.20.0 or later, Stream Workers or Cribl Edge Nodes using the legacy criblmaster token disconnect or fail to reconnect.

Environment

• Self-managed Cribl Stream or Cribl Edge distributed deployment.
• A Leader with Stream Workers or Edge Nodes using the legacy criblmaster token.
• A Leader upgrade to Cribl Stream 4.20.0 or later.
• The Leader’s effective legacy authentication token is not explicitly persisted in the configuration used by the installed version and deployment.

Resolution

1. Restore the last known-good pre-4.20 snapshot or release if the deployment is already running 4.20 and the nodes are disconnected.

2. Verify that the Leader and affected nodes are running the pre-4.20 release.

3. Verify that the affected Workers and Edge Nodes reconnect.

4. Stop Cribl Stream on the Leader.

5. Back up the Leader configuration.

6. Identify the authoritative Leader configuration for the installed version and deployment.

   • For a non-HA file-based deployment, this is typically $CRIBL_HOME/local/_system/instance.yml.

   • For an HA deployment, verify whether the failover-volume leader.yml is authoritative before changing instance.yml.

7. Locate the existing Leader or distributed.master configuration section. The authToken property may be absent.

8. Add authToken: criblmaster inside that existing Leader/master section.

   In configurations that use the nested distributed structure, the setting appears under distributed.master:

   distributed: mode: master master: authToken: criblmaster

   Older releases may use a different configuration layout. Do not copy a configuration block from another release into the file or replace the surrounding configuration.

9. Save the configuration without removing or restructuring existing settings.

10. Start Cribl Stream on the Leader.

11. Verify that the affected Workers and Edge Nodes reconnect.

12. Retry the upgrade to 4.20 or later only after connectivity is restored.

Note: Do not perform this configuration change on the disconnected 4.20 deployment. The workaround is applied after restoring the prior pre-4.20 state and before retrying the upgrade.

Cause

This can be caused by:

• Cribl Stream 4.20.0 and later migrating the Leader’s authentication token to the provisioning-token store.
• The effective legacy token being absent from the Leader configuration consumed during migration.
• Workers or Edge Nodes continuing to present criblmaster after the upgrade while the Leader has no migrated token record for it.

Additional Information

• See Cribl Stream 4.15 instance.yml reference for the older configuration layout.
• See the current instance.yml reference for the nested distributed.master.authToken layout.
• See Leader High Availability/Failover for leader.yml precedence in HA deployments.
• See Create and Manage Authentication Tokens for current token-management guidance.
• After the upgrade, plan a controlled migration from the legacy token to a new provisioning token.