Symptom
Zscaler NSS Cloud reports latency delivering events through the Splunk HEC Source (in_splunk_hec) in Cribl Stream. Event batches accumulate before being acknowledged, even though Cribl Stream shows no resource saturation, backpressure, or rejected connections. Increasing worker group size does not resolve the latency.
Environment
- Cribl Stream (validated 4.19.0; underlying mechanism is not version-specific)
- Source: Splunk HEC (in_splunk_hec), receiving from Zscaler NSS Cloud
- Sender: Any HEC client with a fixed concurrent-connection limit and serialized per-connection delivery (Zscaler NSS Cloud calls this setting UploadWorkers)
Resolution
- Confirm the mechanism using the spent value on completed log lines for the affected Source (channel: "input:in_splunk_hec"). A spent value that scales with event volume and correlates with the sender-reported latency confirms pipeline and route processing time is driving the delay.
- Reduce per-batch processing time in the Pipelines and Routes the events traverse. Remove duplicate or overlapping Route branches that reprocess the same events for similar destinations, and simplify expensive Lookup, Parser, or serialization steps.
- For Zscaler NSS Cloud, request Zscaler to raise the UploadWorkers setting to allow more batches in flight at once.
- Do not rely on disabling HEC acknowledgements to reduce this latency.
Cause
Two factors combine to produce this symptom:
- Sender-side connection concurrency ceiling defined by UploadWorkers. In this Cloud NSS→HEC instance, Zscaler reported 24 configured UploadWorkers. Each connection carried one outstanding batch and waited for the receiving server's application response before sending the next batch. When the outgoing rate exceeds the max throughput, a queue can grow on the Zscaler side while Cribl remains below its request, CPU, memory, rejection, Persistent Queue, and destination-backpressure limits. A fixed number of connections sets a hard ceiling on how many batches can be in flight at once, independent of Cribl Stream's available capacity.
- Response timing tied to pipeline completion. The Splunk HEC Source sends its initial application ack for an event batch only after that batch finishes processing through the full configured Pipeline and Route path. This is unconditional Source behavior, it occurs whether or not Enable.
- Splunk HEC acknowledgements is turned on. That setting only controls a separate acknowledgement-request mechanism (see Working with HEC Acks in Cribl Docs) and does not change when the initial response is sent.
The sender ceiling is approximately:
Maximum completed batches per second ≈ UploadWorkers / per-batch response time
Increasing pipeline or route complexity increases per-batch processing time, which accumulates as latency at the sender, without any error, rejection, or resource exhaustion visible on Cribl Stream.
Disabling Enable Splunk HEC acknowledgements does not change this behavior. That setting affects only the separate acknowledgement-request mechanism, not the timing of the initial response.
Additional Info
It appears that the number of connections allowed is based on memory so comes in powers of two (24, 36, 48, …).
Users aren’t charged based on value so Zscaler likely tries to reserve whenever possible.
Zscaler NSS is the log-egress layer between the Zscaler Nanolog and a customer SIEM. Nanolog retains the originals; NSS emits filtered, formatted copies. The two delivery modes behave differently:
- VM-based NSS: A customer-managed virtual appliance retrieves and decodes Nanolog data, filters and formats it, then sends it over raw TCP. It has a bounded VM-memory buffer for SIEM-side interruptions.
- Cloud NSS: Zscaler pushes HTTPS POSTs directly to a customer endpoint such as Cribl's Splunk HEC Source. There is no customer-local NSS VM or VM-memory buffer.
UploadWorkers is a Zscaler Cloud NSS-side sender-concurrency control. It is not a Cribl Worker Process setting and should not be generalized automatically to VM-based NSS raw-TCP feeds.