Cribl Stream Splunk HEC and Cribl HTTP Sources Fail with TLS Validation Error: Bad Decrypt

Symptom

Cribl Stream Splunk HEC and Cribl HTTP Sources fail to initialize when TLS uses an encrypted private key and a certificate is selected.

The Worker log shows:

TLS validation error: bad decrypt. Is the passphrase correct?

The passphrase has been confirmed correct, the certificate and key match, and an unencrypted PEM copy of the exact same private key works successfully.

Environment

• Cribl Stream 4.20.2
• Splunk HEC Source or Cribl HTTP Source
• TLS-enabled source with a selected certificate
• PEM-encoded PKCS#8 private key, encrypted with AES-256 / PBKDF2 /SHA-256

Resolution

Splunk HEC Source

1. Navigate to Data > Sources.
2. Open the affected Splunk HEC source.
3. Open TLS Settings (Server Side).
4. Leave the existing certificate selected in Certificate name.
5. Clear the Passphrase field.
6. Enter the private-key passphrase in Passphrase.
7. Click Save.
8. Commit the updated configuration.
9. Deploy the updated configuration.

Cribl HTTP Source

1. Open the affected Cribl HTTP source.
2. Open TLS Settings (Server Side).
3. Leave the existing certificate selected in Certificate name.
4. Clear the Passphrase field.
5. Enter the private-key passphrase in Passphrase.
6. Click Save.
7. Commit the updated configuration.
8. Deploy the updated configuration.
9. Do not select the certificate again after entering the passphrase.

Cause

This can be caused by:

• Selecting a certificate for a source without separately entering its private-key passphrase in the source’s TLS settings. The passphrase from the certificate is not carried onto the source.
• Selecting the certificate again after entering the source passphrase, which replaces the passphrase entered on the source.

Additional Information

• Splunk HEC Source
• Cribl HTTP Source