This message originated from Cribl Community Slack.
Click here to view the original link.
We set up Cribl TCP destination to send logs to a Cribl TCP source in another Worker Group. We are sending a high-volume source (~1TB/day) and using Always On source persistent queuing. The Cribl TCP destination does not have persistent queuing enabled and is set to drop events when backpressure signals are received from the destination (the Cribl TCP source) as we felt that using PQ on both sides would be redundant and we don't want to place any additional load on the sending Worker Group. However, we are seeing events being dropped at the destination with this configuration, and it looks like the Cribl TCP source is constantly exerting backpressure, even in Always On mode with plenty of disk available to queue logs. Any ideas why this would happen? I would expect the queue disk to fill up for a Worker Process before the source would excerpt backpressure. I read this (https://docs.cribl.io/stream/persistent-queues-sources#report-to-sender) and it seems like it could be related to the "Buffer size limit" setting, which is 1000 by default, but I even set this to 1,000,000 and it was still happening. Cribl TCP source PQ settings:
Links for this message:
image.png
Solved
Any Ideas Why This Would Happen
Best answer by matthew.filler985
Yeah - I want to say its an "undocumented" feature that high-volume datasets should be using cribl_http. Furthermore you can tweak/tune those HTTPS settings too for your environment.
i would not have backpressure behavior set to drop, but to block
Thanks, Jon. In this case, we are hoping to not affect the primary FDR source, so that is why we selected drop mode. We are fine dropping events if there is backpressure from the Cribl TCP source, but not quite sure why we are getting backpressure in the first place.
I'm only saying data is written into sPQ after transmission. Using disk is always slower and uses more CPU than not doing so.
Yeah - I want to say its an "undocumented" feature that high-volume datasets should be using cribl_http. Furthermore you can tweak/tune those HTTPS settings too for your environment.
The LB process can handle about 100 MB/s and then fans out data to the worker processes behind it so it has very high throughput.
was compression being used for cribl tcp? Was any internal fields like __raw still being sent via cribl tcp ? That would almost double the data being transmitted.
We were using compression, but we were not dropping any internal fields. I overlooked the fact that Cribl TCP/HTTP destinations send almost all internal fields by default. Good catch.
We saw throughput level off around 20 MBps while using Cribl TCP, but after switching to Cribl HTTP it was closer to 40 MBps. I thought this might be related to general throughput limitations like what was mentioned above. Maybe the issue was with the Cribl TCP destination not being able to keep up, not the Cribl TCP source and Always On sPQ? Anyways, after switching to Cribl HTTP and dropping __* fields, we are in good shape and no events are being dropped at the Cribl HTTP destination even in backpressure "Drop" mode. Thanks everyone.
Sign up
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.