Skip to main content

New Workers Can't Join a Cribl Stream Leader Over TLS/mTLS Because cribl.secret Is Missing From the Config Bundle

Jessica Bracken

Symptom

A new worker joins a Leader, exchanges its first config sync, then disconnects after a few seconds and never comes back online. cribl.log on the worker shows:

{"channel":"criblsecret","level":"error","message":"Secret decrypt failed","confType":"certificates","path":"$CRIBL_HOME/local/cribl/auth/certs/worker-cert.key","error":{"message":"error:1C800064:Provider routines::bad decrypt"}}

immediately followed by:

{"channel":"cribl","level":"error","message":"failed to start API server, exiting..."}

The worker connects normally with TLS disabled on the worker group's API Server settings. Enabling TLS (with or without mutual TLS) reproduces the failure on every affected worker, immediately after the first bundle sync from the Leader.

Note: A failed to start API server, exiting... error with ENOENT: no such file or directory, open '.../_system/worker-rootca.pem' can accompany this symptom but is usually a separate, unrelated issue — a stale ssl.caPath entry left in the worker's local cribl.yml from an earlier sync attempt, pointing at a certificate path that was never actually provisioned. Correct or remove that caPath entry first. If the Secret decrypt failed error on worker-cert.key persists afterward with TLS/mTLS enabled, it is the issue this article addresses.

Environment

  • Cribl Stream, distributed deployment (Leader + Workers), version 4.20.1 (same mechanism applies to other versions where bundler.bundleGitIgnoredPatterns exists — see Cause)
  • Self-managed (on-prem or cloud VM)
  • Worker group with TLS or mutual TLS enabled on the API Server settings (Worker Group Settings > System > General Settings > API Server Settings > TLS)
  • New worker group / first-time worker bootstrap against the Leader

Resolution

  1. On the Leader, open $CRIBL_HOME/local/cribl/cribl.yml.

  2. Add a bundler.bundleGitIgnoredPatterns entry for the Leader's cribl.secret path:

    bundler: bundleGitIgnoredPatterns: - 'local/cribl/auth/cribl.secret'

    If the same worker group also delivers custom TLS certificates through the config bundle (rather than placing them on each worker out-of-band), use the broader local/cribl/auth pattern instead to cover both the secret and the certificate files in one entry.

  3. Save the change. No Leader restart is required — the setting is read on the next bundle build.

  4. Trigger Commit & Deploy on the affected worker group.

  5. Have the worker reconnect (or restart the worker process) and confirm it stays connected with TLS/mTLS enabled, with no further Secret decrypt failed entries in cribl.log.

Verification

Compare $CRIBL_HOME/local/cribl/auth/cribl.secret on the Leader and on the worker after the deploy — they must be byte-identical. Before the fix, a worker that bootstrapped without this setting generates and keeps its own cribl.secret, which will differ from the Leader's.

Cause

cribl.secret is the Leader’s master key for encrypting sensitive configuration values, including the TLS ssl.passphrase used to protect worker-cert.key.

Cribl excludes cribl.secret from version control by default through its internal .gitignore rules. The Worker Group bundler normally includes only Git-tracked files, so the secret is omitted from the configuration bundle unless the Leader’s cribl.yml explicitly includes it with bundler.bundleGitIgnoredPatterns.

When a new worker bootstraps without the Leader’s cribl.secret, it generates its own random key. The TLS passphrase in the Worker Group configuration was encrypted with the Leader’s key, so the worker’s different key cannot decrypt it. The worker then logs Secret decrypt failed for worker-cert.key, fails to start its API server, and disconnects from the Leader.

This explains why the worker connects when TLS is disabled: without TLS, it does not need to decrypt the protected certificate passphrase during startup. Enabling TLS or mTLS exercises that decryption path and exposes the key mismatch.

bundler.bundleGitIgnoredPatterns is a Leader-side system setting. Setting it in a worker’s cribl.yml or in Worker Group configuration does not change the bundle. The pattern applies to files within the bundler’s supported subtrees, including local/cribl/auth/cribl.secret; use local/cribl/auth when the bundle must also include custom TLS certificate files.

This article addresses the specific case in which a worker’s first bundle sync omits the Leader’s secret because of the bundler configuration. Other causes of a Leader/Worker cribl.secret mismatch—such as upgrade timing, worker redeployment, or a KMS configuration change—require separate troubleshooting.

Additional Information

Terms & ConditionsAccessibility statement