Certificates Missing on Workers and HEC Tokens Hashed After Teleport Due to .gitignore in Cribl Stream

Symptom

One or more of the following symptoms are observed within Cribl Stream:

• A certificate deployed via Worker Group Settings > Security > Certificates is visible on the Leader but is missing or shows the warning "Certificate <name> has been moved or removed from the filesystem and is unavailable. Please restore or re-create it." when teleporting to a Worker
• A HEC-based destination token appears in plaintext on the Leader but is hashed (displayed as #42::...) when teleporting to a Worker
• A source or destination using TLS surfaces SSL/TLS errors on Workers after commit and deploy
• Running find / -name "*.crt" on the Worker returns no results for certs that are deployed on the Leader

Environment

• Cribl Stream
• Any version
• Leader deployed on-prem or containerized (e.g., Kubernetes/AKS)

Resolution

1. On the affected Worker, run:
   find / -name "*.crt"

find / -name "*.key"
   If the cert files deployed from the Leader are not present, the config bundle is not delivering them.
2. On the Leader, cat the .gitignore file. Note that the path may differ depending on deployment type:
   • Standard install: /opt/cribl/.gitignore
   • Containerized install with mounted config volume: /opt/cribl/config-volume/.gitignore
cat /opt/cribl/.gitignore

# or

cat /opt/cribl/config-volume/.gitignore
3. Look under ### CUSTOM SECTION -- DO NOT REMOVE ### for any entries matching:
   • local/cribl/auth/
   • local/cribl/auth/cribl.secret
   • local/cribl/auth/certs
   • *.crt, *.key, *.pem
4. If matching entries are found, add bundler.bundleGitIgnoredPatterns to the Leader's cribl.yml. On the Leader, edit cribl.yml. Note the path may differ depending on deployment type:
   • Standard install: /opt/cribl/local/cribl/cribl.yml
   • Containerized install with mounted config volume: /opt/cribl/config-volume/local/cribl/cribl.yml
5. Add the following block. If a bundler: block already exists, add bundleGitIgnoredPatterns under it rather than creating a duplicate top-level key:
   bundler:

bundleGitIgnoredPatterns:

- 'local/cribl/auth'
   • Note: No Leader restart is required after editing cribl.yml.
6. From the Leader UI, perform a Commit & Deploy to the affected Worker Group.
7. Verify:
   • Re-run find / -name "*.crt" on the Worker and the cert files should now be present under /opt/cribl/local/cribl/auth/certs/
   • Teleport to the Worker and confirm certificates are visible in Worker Group Settings > Security > Certificates
   • Confirm HEC tokens are no longer hashed when teleporting to the Worker
   • Confirm any TLS-related source/destination errors are resolved

Cause

In Cribl Stream, any file or directory listed in .gitignore is excluded not only from pushes to a remote Git repository, but also from the Worker Group config bundle that is delivered to Workers during a Commit & Deploy.

If the auth directory or cert files are listed under the ### CUSTOM SECTION ### of .gitignore, any files in that directory will never be delivered to Workers via the config bundle.

Without cribl.secret, the Worker cannot decrypt any value encrypted by the Leader, causing tokens and passphrases to appear as raw #42::.... Without the cert files, Cribl cannot find them on the filesystem and surfaces the "moved or removed" warning.

The bundler.bundleGitIgnoredPatterns setting in cribl.yml instructs the bundler to include specified gitignored files in the config bundle without tracking them in Git, giving the security benefit of .gitignore while still delivering the files to Workers. Patterns are evaluated relative to $CRIBL_HOME/groups/<group>, so the path local/cribl/auth covers both cribl.secret and cert files for all Worker Groups.

Additional Information

For more information, see Worker Deployment: How to Handle Customer-Defined .gitignore Exclusions.