Splunk UF Failing To Connect: Invalid Payload Size Error While Using Cribl.Cloud TCP With TLS

Hi @user, what UF version are you using? Did you check for possible TLS version mismatches? Check the min and max version values in the TLS settings and compare against what the UF provides.

So narrowed it down last night, it was not TLS mismatch, token mismatch, S2S version mismatch. It was sslVerifyServerCert = true . Even with everything configured (tlsHostName, sslCommonNameToCheck, sslAltNameToCheck) setting this to true receives the following errors in Splunk:

ERROR SSLCommon [4500 TcpOutEloop] - Can't read ca certificate file  SSL error code=2147483648 message="error:80000000:system library::No error"
ERROR TcpOutputProc [4500 TcpOutEloop] - Failed to create SSL context for default.main.REDACTEDTENANT.cribl.cloud:9997: tls_error_type="SSL context initialization failed", tls_error_msg="Failed to initialize SSL client context"

Even though the cribl cert is valid, having SAN and CN, even putting those names in to verify didn't work.

You might want to try using:

[sslConfig]
sslRootCAPath = C:\Program Files\SplunkUniversalForwarder\etc\auth\<your-CA>.pem

in UF server.conf

Obviously needs the CA installed in that path.

Yeah I was following the cribl documentation to see if there was anything extra that needed, so using the default CAs configured in Splunk as the example configuration defines.

You can download the root cert and reference it specifically in the configs as state by @user. CA bundles are not added by default in Splunk. Alternative iirc there is a bundle with public CA under: /opt/splunk/etc/apps/splunk_secure_gateway/lib/certifi/cacert.pem This can be added: # server.conf [sslConfig] sslRootCAPath = /opt/splunk/etc/apps/splunk_secure_gateway/lib/certifi/cacert.pem

Unfortunately that app doesn't exist on a clean Windows UF install but yes it could be installed. Looking a default install with the Cribl documentation/instructions provided for reference. So creating your own combined cert, downloading cribl's CA, etc. are all valid methods but if following the instructions on a clean install, with sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.perm if sslVerifyServerCert = true is configured, it will fail.

Bringing this thread back up for others to see: @user @user @user $SPLUNK_HOME/etc/auth/appsCA.pem has the Globalsign Root R1 CA that is used with signing Cribl's TLS certs but even setting sslRootCAPath to that in server.conf or in outputs.conf still threw errors and quarantined the tcpout connection when sslVerifyServerCert = true was set. Thanks @user who found some internal documentation that had the tidbit. Setting clientCert to the forwarders server cert (not client cert, no mTLS authentication is happening here and it is disabled on the Cribl Stream Splunk TCP source) and configuring the default sslPassword did it. So the following configuration is what ultimately worked

[tcpout]
defaultGroup = criblcloud
disabled = false
useACK = true

[tcpout:criblcloud]
server = GROUP.WORKSPACE.TENANT.cribl.cloud:9997
sslCommonNameToCheck = WORKSPACE-TENANT.cribl.cloud
sslVerifyServerCert = true
sslRootCAPath = $SPLUNK_HOME/etc/auth/appsCA.pem
clientCert = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password
useSSL = true
sendCookedData = true
token = 199a1f44-799b-4313-b8c6-7f78739562ac

Now sslRootCAPath could be set to a combined PEM of SplunkCloud CA's, on-prem CA's and located under $SPLUNK_HOME/etc/auth/ or an app directory, but even if those were valid and worked with openssl verify splunk still failed. I'm still a little confused on why clientCert was necessary because the Cribl Stream Source was configured with TLS Authentication disabled and even it if was enabled this configuration would fail because server.pem is a self-signed cert signed by the local cacert created with the Splunk install, but hey it works.

Not to mention those are default values. (clientCert and sslRootCAPath) Meaning you didn't need to explicitly put them there in the first place.

@user not with the latest Splunk 10 UF install those aren't set by default: outputs.conf

splunkfwd@debian:~/etc/apps$ splunk btool outputs list --debug | grep -iE "rootca|clientcert"
splunkfwd@debian:~/etc/apps$

server.conf

splunkfwd@debian:~/etc/apps$ splunk btool server list --debug | grep -iE "rootca|clientcert"
splunkfwd@debian:~/etc/apps$

but you do need to put them in the config, testing without those explicitly defined in outputs.conf tcpout stanza generates the SSLCommon and TcpOutputProc errors I shared previously.

Hi @user, Thanks for the update. Checking the outputs.conf revealed that useSSL

* The special value "legacy" means the forwarder uses the 'clientCert' property to
  determine whether or not to use SSL to connect.

Default: legacy Meaning if you set useSSL = true should avoid setting clientCert = ... as this is only needed for mTLS.

Big shoutout and thanks to Brenden. This saved us after upgrading our UFs to 10.4 :blob-heart:

Glad it was helpful @user I'm going to grab some older versions of Splunk UF and figure out where setting the clientCert became required.