Cribl Stream Kubernetes Workers Show High CPU at Low Ingest and Are OOMKilled

Symptom

Cribl Stream Worker pods deployed in Kubernetes show unexpectedly high CPU and memory usage even though processing and log-ingestion volume is low. One or more pods may later restart with OOMKilled.

The two symptoms can have different causes. CPU usage can come from too many Worker Processes or from work pinned to one Worker. The OOMKill can come from memory used by Cribl, or by an external process that Cribl started and that is not represented in Cribl Worker Process metrics.

kubectl describe pod may report:

• Reason: OOMKilled
• Exit code 137 or an equivalent Kubernetes out-of-memory termination message

Environment

• Cribl Stream Worker Group deployed in Kubernetes.
• Worker pods have Kubernetes CPU and memory requests or limits configured.

Resolution

1. Capture both symptoms from the affected pod.

   Run:

   kubectl describe pod <pod-name> -n <namespace> kubectl top pod <pod-name> -n <namespace> --containers

   Record the pod name, OOMKill timestamp, restart count, container resource requests and limits, per-container CPU and memory usage, and the Events section. If Metrics Server is unavailable, use the customer’s container monitoring data instead.

2. Check Worker Process sizing before investigating application load.

   Ask the customer to provide their current Helm values or Kubernetes Deployment manifest, along with the Worker Process settings in Stream and the Worker diagnostic bundle. Compare:

   • resources.requests.cpu

   • resources.limits.cpu

   • CRIBL_K8S_CPU_LIMIT

   • CRIBL_MAX_WORKERS

   • The configured and actual Worker Process count

   Ask the customer to include system.json or the relevant diagnostic output so you can compare the CPU count visible to Cribl with the Kubernetes CPU request and limit. If the default relative Process count is being calculated from more CPUs than the pod should use, recommend setting a positive Process count or configuring CRIBL_MAX_WORKERS to an appropriate value. Keep CRIBL_K8S_CPU_LIMIT aligned with the pod’s CPU limit, then have the customer roll out the change.

3. Determine whether CPU is spread across Workers or pinned to one Worker.

   Use the Worker Process metrics or metrics.log to compare CPU usage across Workers:

   • If all Workers are busy while ingest is low, check for over-provisioning or an expensive pipeline/function before increasing pod resources.

   • If one Worker is much busier than the others, check for a single high-volume connection or source pinning. For Syslog TCP Sources, verify that CPU load balancing is enabled when the deployment requires it.

   • Review Exec Sources and pipelines that run external commands. These can contribute to CPU usage even when the normal ingest rate is low.

4. Confirm whether Kubernetes, rather than Cribl, performed the restart.

   Review the container’s Last State, Reason, exit code, and Events in kubectl describe. Retrieve the previous logs:

   kubectl logs <pod-name> -n <namespace> --previous

   If the termination reason is not OOMKilled, follow the matching Kubernetes or Cribl troubleshooting path instead.

5. Separate Cribl memory from memory used by external processes.

   Compare the memory reported by Cribl Worker Process diagnostics with the container or pod memory trend. Cribl’s Memory (MB) setting is a per-process heap ceiling; it does not represent all memory charged to the container.

   Review the pod specification for sidecars, agents, and log shippers. Also review configured Exec Sources and pipelines that call exec, execFile, or another external script. A child process can be short-lived, absent from a diagnostic bundle, and still count against the Kubernetes memory limit while it is running.

   Capture kubectl top pod --containers and, if available, cgroup or process-level memory data for the 15–30 minutes before the OOMKill. If an external child process is responsible, disable or correct the script/source, or isolate it in a separately resourced container. In the reported case, disabling the external child process stopped the OOMKills.

6. If Cribl itself is using the memory, investigate the workload.

   Check Worker Process heap and RSS, destination buffers, backpressure, persistent queues, large lookups, stateful or high-cardinality functions, and the number of Worker Processes. Scale the workload or pod resources only after identifying the applicable consumer.

7. Handle JavaScript heap errors as a separate path.

   If the previous logs contain FATAL ERROR: ... JavaScript heap out of memory, follow the heap and backpressure troubleshooting procedure instead of treating the event as an unexplained Kubernetes OOMKill.

8. Confirm both fixes during a representative peak period.

   After changing Worker Process sizing or addressing the external process, monitor CPU distribution, pod memory, Cribl memory, pod restarts, throughput, and destination health. Confirm that CPU is no longer saturated at low ingest and that the pods remain stable during the conditions that previously caused the OOMKill.

Cause

The two reported symptoms can be related, but they do not necessarily share a root cause:

• High CPU at low ingest: The pod may start more Worker Processes than its Kubernetes CPU request or limit can support. A relative Process count can also be calculated from CPUs visible to the container rather than the intended allocation. Alternatively, a single connection or Syslog TCP Source can pin work to one Worker, or an Exec Source/pipeline can run an expensive external command.

• Kubernetes OOMKilled: The memory consumer may be Cribl heap or external memory, but it may also be a sidecar, agent, log shipper, Exec Source child process, or another process sharing the container or pod memory boundary.

The Cribl Memory (MB) Worker Process setting controls the process heap. It does not represent the entire container memory budget and does not govern external memory. A Cribl process can therefore remain below its configured heap or observed RSS while an external child process causes Kubernetes to terminate the container.

In this example issue, the Worker Process sizing was corrected to match the pod’s CPU allocation, resolving the CPU-sizing concern. The OOMKills continued until an external child process was disabled, confirming that the CPU and memory symptoms required separate fixes.

Additional Information

• Kubernetes Worker Deployment — Configure CRIBL_K8S_CPU_LIMIT, CRIBL_MAX_WORKERS, and Kubernetes Worker Process settings.
• Sizing and Scaling — Review Worker Process memory, external memory, process counts, and workload sizing.
• Diagnose Issues — Create diagnostic bundles and collect CPU or memory evidence for Support.
• Considerations for Cribl Stream on Kubernetes — Review Kubernetes resource allocation, Worker Process limits, persistent queues, and Worker lifecycle considerations.
• If the evidence still does not identify the memory consumer, include the kubectl describe output, prior container logs, per-container memory data, pod resource specification, OOM timestamp, and Cribl diagnostic bundle in a Support case.