This message originated from Cribl Community Slack.
Click here to view the original link.
I am shipping log data to an OTEL destination and I keep getting a RESOURCE_EXHAUSTED: gprc: received message after decompression larger than max 4194304
I am not really sure what I need to be tuning for this, in my pipeline I have my function set to batch in size of 4096 (see screenshot) and my destination is configured to have a body limit size of 4096 KB (see screenshot) - which should place me under that threshold per batch
Has anyone encountered this or like error with an OTEL destination? any tips on what I can check out?
Links for this message:
image.png
image.png
Solved
I Am Shipping Log Data To An Otel Destination And I Keep Getting A Resource_exhausted: Gprc: Received Message After…
Best answer by Emily Ashley
Let me take a look! I believe in the pipeline that
batch size is an event count and the batch size limit (kb) is where you'd adjust your batch groupings based on bytes.
But also, the logical batches created with batch functions are not guaranteed to map 1:1 to network transmissions. The OTel Destination optimizes for performance so it may send multiple logical batches in a single HTTP or gRPC call.
Let me double check on the OTel Destination settings to get us under this limit from snow HLA
Let me take a look! I believe in the pipeline that
batch size is an event count and the batch size limit (kb) is where you'd adjust your batch groupings based on bytes.
But also, the logical batches created with batch functions are not guaranteed to map 1:1 to network transmissions. The OTel Destination optimizes for performance so it may send multiple logical batches in a single HTTP or gRPC call.
Let me double check on the OTel Destination settings to get us under this limit from snow HLAI see a past case where dropping the OTel Destination Body Size Limit to 3500 KB helped resolve these errors.
I think that should cover the difference of the limit of after decompression (HLA) vs setting the limit while it is still compressed (OTel Destination).
If not, please open a support case! I'm happy to chime in and test with our OTel friends over there.
really appreciate it, I am working with someone on their OTel side as well and have a call with them later today to validate everything
just like that, everything is already looking much healthier - thank you for the direction on this
Links for this message:
image.png
Links for this message:
image.png
old man yells at cloud
Let me know which numbers work best (~3000?) and I can make sure to keep a note of that for future questions :slightly_smiling_face:
I am waiting to confirm with SNOW but right now the magic number has been 3000; the step to 3500 took us to about 21% failure rate and 3000 over the past 2 or 3 hours has been 0%
Sign up
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.