Question
Why are events being lost when sending batched HEC payloads to Splunk, even though most events in the batch are valid?
Environment
- Splunk HTTP Event Collector (HEC) destination
- Batched HEC payloads containing multiple events
Answer
This is expected Splunk HEC behavior. When Splunk receives a batched HEC payload containing a single invalid event (for example, an event with an empty event field producing the error Event field cannot be blank), Splunk processes all events up to the invalid event and then discards the entire remainder of the payload. All valid events that appear after the bad event in the batch are lost.
For example, if a batch contains 100 events and event number 50 is invalid:
- Events 1 through 49 are indexed successfully
- Event 50 triggers the error
- Events 51 through 100 are silently dropped, even if they are perfectly valid
The behavior described above applies when HEC indexer acknowledgment is disabled (the default). When HEC acknowledgment is enabled, Splunk rejects the entire batch immediately and no events from the batch are indexed. The Splunk HEC error response includes an invalid-event-number field that identifies the position of the offending event.
To mitigate this issue:
- Use a Pipeline in Cribl Stream to filter or fix invalid events before they reach the Splunk HEC destination.
- Add a validation step that checks for empty event fields or other conditions that Splunk considers invalid.
- Route suspect data through a separate pipeline for inspection.
Note that Cribl does not have this same limitation when acting as a HEC receiver (Splunk HEC Source). The Cribl HEC Source processes each event in a batch independently and returns per-event errors without discarding valid events in the same payload. This independent processing applies to the Cribl HEC Source only; at the Splunk HEC Destination level, Cribl sends events in batches and does not currently perform selective retry for partially failed batches.