Maximum Buffered Data in Memory per HTTP Destination per Worker Node

Question

What is the maximum buffered data in memory on a HTTP destination per worker node?

Environment

• Cribl Stream
• HTTP destinations

Answer

Maximum buffered data in memory (per HTTP destination per worker node) = RequestConcurrency * 3 * BodySizeLimit * WorkerProcessCount.

Last Validated

Version 4.17.0

Additional Information

First, lets clarify some terms from the Advanced Settings tab on HTTP destinations:

• Request concurrency
  • Maximum number of ongoing (in-flight) requests before blocking. A request is complete when the client receives a response back from the server, reaches a time out, or a network error. For example, a response back from the server could be a 200 (ok) or a 500 (internal server error). If the server sends a response fairly quickly, then request concurrency shouldn't be reached. However, if a server takes 30 seconds to send a response, then request concurrency could be reached pretty quickly. When request concurrency is reached, we will start seeing blocking and backpressure to prevent new data from being ingested.
• Body size limit (KB)
  • The maximum size in KB of the request body before it is sent to the destination.
• Events-per-request limit
  • Maximum number of events to include in the request body before it is sent to the destination. Default is 0 (unlimited).
• Flush period (sec)
  • Maximum time between requests.
• Request timeout
  • If the client doesn't get a response back within this period, then the request will be terminated by the client.

Regarding body size limit, flush period, and events-per-request limit, they are essentially completing with each other for whichever is reached first. For example, if the flush period is 5 second, and body size limit is 1024 KB, the payload will only ever reach 1024 KB if there is enough data transferred within that 5 second period. Otherwise, it will just get sent to the destination after 5 seconds passes, regardless of what size it is.

Next, lets run through of what exactly occurs when data is received on a worker process:

1. Data is received on a source and is handled by a worker process. The same worker process handles this data from the source (e.g. Syslog), through pipelines/processing, and to the destination (e.g. CrowdStrike) within memory.
2. When the data reaches the destination, the payload is compiled in a "build" memory buffer (stored in external memory), and size of the payload is determined by body size limit, flush period, or events-per-request limit, as previously mentioned,
3. The payload is then sent to the destination.
4. If the data is sent successfully i.e. a 200 is received from the server, then we are good!
5. If anything other than a 200 is received, then the payload will either be dropped, or retried if it matches a retry condition.
6. If the retry condition is met, the payload is moved into its own retry buffer, and retried periodically based on your retry configuration. At this point, other events can still be processed and attempted to send to the destination.
7. If we are seeing continuous non-200 outcomes on the destination, then we will continue to create new retry buffers. Also, some information to note is that once we start seeing retry buffers, the flush period no longer comes into play.
8. When the max number of retry buffers is reached, then we will start seeing backpressure and blocking. The max number of retry buffers (per worker node per destination) is determined by the RequestConcurrency × 3 × WorkerProcessCount. The "3" is just a fixed number hardcoded into our platform (protected maxBuffers). For example, if I have a request concurrency of 3, and a single worker process, then my retry buffers will max out at 9 before we start seeing backpressure: