What Is The Best Way To Determine If A Slow Pipeline Is Causing Backpressure That Is Impacting Sources

Thanks, that is good information! The reason I asked this is because we recently started testing with Redis to store ComputerName to LocalAddressIP4 mappings and it seemed to cause a bottleneck in our FDR pipeline. The usual ~2-10 messages in the queue grew to a much larger number than expected. We disabled Redis and Stream was able to recover, but we are hoping to make some changes and increase visibility around this as we continue testing again. During the first attempt, we were basically updating/creating a key-value in Redis for every CrowdStrike FDR log. Since the issue, we have tested with aggregations and dynamic sampling to reduce the volume of Redis writes, and we are also thinking about using a dedicated Worker Group and Cribl HTTP destination/source to decouple the Redis writing from our primary CrowdStrike FDR source. To monitor the pipeline performance, we are calculating an internal __duration field and monitoring route throughput and the number of messages in the FDR queue. I was also hoping there might be other metrics or logs in Stream that could help us identify any backpressure from the Redis pipeline or issues that could cause the source to stop downloading FDR messages from the queue.

I recommend a pipeline profile to see what the latency is with that pipeline. If you can't reduce the latency then you may need more processes to spread out the data to lessen the impact of the latency.

Decoupling could possibly work as well.