The TCP output processor has paused the data flow

Question

Afternoon (at least on my side).Need some help on my side. I've got a Splunk HF cluster sending events to my Cribl cluster. The HFs are experiencing this issue every now and then:
08-16-2023 09:29:32.056 +0200 WARN TcpOutputProc [12084 indexerPipe_1] - The TCP output processor has paused the data flow. Forwarding to host_dest=xx.xx.xx.xx inside output group cribl_out_group from host_src=xxxxxx has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.It's not happening too often, so I guess the questions are:

Should I be worried?
If yes, where do I start looking for what might be causing the issue on the Cribl side?

Best answer by Oliver Hoppe

that depends

https://docs.cribl.io/stream/persistent-queues/#source-andor-destination-pq

if you have a single destination I would configure source PQ, if you have multiple use destination PQ as you can ensure that way that even if one destination is blocked your other destination(s) will get the data

Oliver Hoppe · Accepted Answer

that dependshttps://docs.cribl.io/stream/persistent-queues/#source-andor-destination-pqif you have a single destination I would configure source PQ, if you have multiple use destination PQ as you can ensure that way that even if one destination is blocked your other destination(s) will get the data

Oliver Hoppe · Answer

Hi @Thinus Swart

Usually that happens when Cribl Stream is creating backpressure towards your HFs. Root cause in most cases is either that one of your downstream systems is set to block and is either down or cannot handle the load Cribl Stream is sending or your Stream instance have not sufficient compute.
Please check your Stream destinations health first

Sign up

Using your Cribl.Cloud account

Login to the community

Using your Cribl.Cloud account

Scanning file for viruses.

This file cannot be downloaded