Cribl Stream OTel Source High CPU and Data Loss When maxActiveCxn Is Unlimited

Symptom

After upgrading Cribl Stream or redeploying OpenTelemetry (OTel) Source configuration, you observe one or more of the following:

• The master/API process CPU stays near 100% while Worker Process CPU remains relatively low.
• OTel data stops flowing or is severely degraded.
• Logs show ERR_IPC_CHANNEL_CLOSED errors and repeated restarting worker process messages.
• Upstream OTel agents report TLS or gRPC handshake failures (for example, authentication handshake failed: context deadline exceeded).
• Reducing Worker Process count lowers CPU somewhat but does not restore data flow.

Environment

• Cribl Stream 4.15.1
• Self-managed deployment
• One or more OpenTelemetry Sources using gRPC protocol
• Many persistent upstream OTel collector agents (for example, 100+ hosts)
• OpenTelemetry gRPC sources configured with an active connection limit (maxActiveCxn) of 0 (unlimited) before remediation

Resolution

1. Navigate to Manage > Worker Groups > <your worker group> > Sources.
2. Open each affected OpenTelemetry Source.
3. Expand Advanced Settings.
4. Set Active connection limit (maxActiveCxn) to 1000 (the default). Do not use 0 (unlimited) in production.
5. Repeat for every OTel Source in the worker group that receives agent traffic. A single source left at unlimited can keep the master process saturated.
6. Click Save, then Commit & Deploy.
7. Monitor the master/API process CPU and source metrics. CPU should drop and data flow should resume within minutes.
8. Verify upstream agents can connect and that ERR_IPC_CHANNEL_CLOSED errors stop recurring.

Note: If you have multiple OTel Sources (for example, one for metrics and one for service monitoring), apply the same limit to all of them.

Cause

This issue is caused by configuring OpenTelemetry gRPC sources with maxActiveCxn set to 0 (unlimited), which removes connection backpressure during periods of high connection churn.

The master/API process handles connection acceptance, TLS negotiation, gRPC setup, Worker Process orchestration, and inter-process communication. During post-upgrade instability or repeated Worker Process restarts, upstream OTel agents reconnect. That creates connection churn — many new connection opens per minute, often far above steady-state levels.

With maxActiveCxn: 0:

• The source never rejects excess connection attempts.
• The master/API process continues accepting TLS handshakes and gRPC setup work.
• The event loop saturates before Worker Process CPU is stressed.
• ERR_IPC_CHANNEL_CLOSED appears because the overloaded master cannot respond to Worker Process RPCs during restart storms.
• OTel gRPC listeners may close and reopen during Worker Process restarts, triggering additional agent reconnect waves.

Restoring maxActiveCxn: 1000 re-enables rejection and backpressure. The master stops accepting an unbounded connection workload and can recover.

Note: This is typically not a Worker Process compute problem. Diagnostic evidence from affected deployments shows the master/API process at ~100% CPU while Worker Processes remain at low CPU. Reducing Worker Process count may lower CPU somewhat but does not restore ingestion if unlimited connection acceptance remains. In documented issues, setting one OTel Source to 1000 while leaving a second source at 0 did not restore stability — all affected OTel Sources must use the default limit.

Additional Information

For more information, see OpenTelemetry (OTel) Source.