Cribl Stream Sentinel and Azure Data Explorer Destinations Backpressure When Azure Client Secret Expires

Symptom

When using Cribl Stream to send data to Microsoft Sentinel and Azure Data Explorer (ADX), all destinations targeting these services begin to report backpressure and stop delivering events.

Typical observed behavior includes:

• All Microsoft Sentinel and Azure Data Explorer destinations in a worker group show backpressure and/or “sender at capacity” status.
• Worker logs show repeated OAuth authentication failures when requesting tokens from login.microsoftonline.com for the Sentinel and ADX destinations.
• Persistent Queue (PQ) begins to grow for the affected destinations, and upstream push-based sources may report throttling or blocked connections.
• No new events appear in the Sentinel or ADX tables even though upstream sources are actively sending data.

This pattern often appears suddenly at a specific time (for example, shortly after an Azure client secret reaches its expiration date), with all affected destinations failing in the same window.

Environment

• Product: Cribl Stream
• Version: 4.17.1 (and later)
• Deployment Type: Cribl.Cloud / Managed AWS or self-managed leader + worker group
  • Destinations:
  • Microsoft Sentinel destination(s) using OAuth client secret authentication
• Azure Data Explorer (ADX) destination(s) using OAuth client secret authentication
• Azure configuration:
  • One or more Azure App Registrations used as OAuth client credentials for the Sentinel and ADX destinations
  • Authentication method: Client secret (not certificate)

Resolution

1. Identify the Azure app registration used by the affected destinations.
   • In Cribl Stream, open a failing Sentinel or ADX destination.
   • On the Authentication tab, note the Client ID and Tenant ID.
   • These values correspond to an app registration in Azure.
2. Check the client secret status in Azure.
   1. Sign in to the Azure portal.
   2. Navigate to Azure Active Directory → App registrations.
   3. Select the app registration used by your Cribl destinations (matching the Client ID noted above).
   4. Go to Certificates & secrets → Client secrets.
   5. Review the secret(s) used by Cribl:
      • If the secret shows as Expired, or
      • The Expires date is in the past or very near term,

      then the destinations will not be able to obtain OAuth tokens using that secret.

3. Create a new client secret in Azure
   1. Immediately copy the Value of the new secret and store it securely; you will not be able to see it again after leaving the page.
4. Update the secret in all affected Cribl Stream destinations. For each Microsoft Sentinel and Azure Data Explorer destination that uses this app registration:
   1. In Cribl Stream, navigate to Manage → Data → Destinations and open the destination.
   2. Go to the Authentication tab.
   3. In the OAuth secret (or Client secret) field, paste the new client secret value.
   4. Save the destination configuration.
5. Commit and deploy the updated configuration.
6. Monitor destination health and PQ drain.
   1. After deployment, confirm that the Sentinel and ADX destinations transition out of backpressure and are able to send data successfully.
   2. Verify that PQ size for the impacted destinations begins to decrease as events are drained and delivered.
   3. During recovery, you may temporarily observe:
      • 429 (rate limiting) responses from Azure as queued events are retried.
      • Transient backpressure warnings as PQ drains.
      These are expected while the backlog clears and do not indicate a new issue.
7. Validate data arrival in Sentinel and ADX.
   • In Microsoft Sentinel / Log Analytics, run a query on the relevant tables (for example, Syslog, SecurityEvent, CommonSecurityLog, or your configured streams) and confirm that new events are arriving with current timestamps.
   • In Azure Data Explorer, query the target database/table to confirm that new records are being ingested.

Cause

The Azure AD client secret used by the Cribl Stream Microsoft Sentinel and Azure Data Explorer destinations expired.

When the secret expires:

• Cribl Stream continues using the old secret value to request OAuth tokens from login.microsoftonline.com.
• Azure rejects the token requests with authentication failures.
• The destinations cannot send data and enter backpressure, which causes:
  • Persistent Queue (PQ) growth on the affected destinations.
  • Upstream push-based sources to slow or stop sending data.
  • No new events arriving in Sentinel or ADX until the secret is rotated and destinations are updated.

Last Validated

• Cribl Stream 4.17.1