Events Sent to a Splunk LB Destination Do Not Appear in Splunk When _raw Is Present but Blank

Symptom

Events are counted as sent on a Cribl Stream Splunk LB destination, but the events do not appear in Splunk search results. This can occur when the event contains an _raw field that exists but has no value.

This same condition does not have the same impact on a Splunk HEC destination. A blank _raw can still be handled by Splunk HEC and the event can appear in Splunk.

Environment

• Cribl Stream
• Splunk LB destination
• Splunk HEC destination
• Splunk HEC source
• Events that contain an _raw field with an empty value

Resolution

Use one of these two approaches:

1. Prevent the blank _raw from being created in the first place.
   • When sending data to a Splunk HEC source, send the payload using an event key instead of a fields key.
   • Using event allows _raw to be built correctly.
   • Using fields can create top-level fields while also creating a blank _raw.
2. If you cannot change the upstream payload format, remove _raw when it is blank before sending the event to the Splunk LB destination.
   • If _raw is absent, the Splunk LB destination can construct _raw from the top-level fields, excluding Splunk control fields and Cribl internal fields.
   • If _raw is present but blank, the Splunk LB destination treats it as the event body and the event can be silently dropped after it reaches Splunk.

Cause

This issue is caused by a difference between a blank _raw field and an absent _raw field.

A blank _raw means the event contains an _raw key, but the value is empty.

For Splunk LB:

• If _raw is present but blank, the destination still shows the event as sent.
• After the event reaches Splunk, Splunk silently drops it.
• If _raw is removed entirely, the Splunk LB destination can build _raw from the top-level fields.

For Splunk HEC:

• A blank _raw does not appear to have the same impact.
• The event can still be handled and show up in Splunk.

One common way to create this condition is to send data to a Splunk HEC source using a fields key. That can create a blank _raw along with top-level fields. Sending the same payload with an event key instead creates _raw correctly.

Additional Information

• If the Splunk LB destination shows events as sent but the data never appears in Splunk, check whether _raw exists and is blank rather than missing.
• In this scenario, the fix is not to populate _raw manually unless needed. The safer fix is often to remove the blank _raw field so the destination can construct it correctly.
• This behavior can be confusing because Splunk HEC and Splunk LB do not behave the same way when _raw is present but empty.
• The exact implementation path could not be confirmed from indexed code, so this example is based on observed product behavior and the reproduction pattern described here.