How to Investigate Delayed Events in Cribl Stream

Objective

Identify whether delayed events originate in the downstream system, Cribl Stream Destination, Cribl Stream Source, or sending application.

Environment

• Cribl Stream with an enabled streaming Source and Destination.
• Access to the downstream system and its ingestion or arrival metadata.
• Access to the sending application or forwarder host.

Procedure

1. Define the investigation window and identify a distinctive event.
   • Record the event’s source, host, index or dataset, unique message content, event timestamp, and observed arrival time.
   • Use the same identifying values at each investigation point.
2. Check the downstream system.
   • Search a broad time range around the expected arrival time.
   • Verify whether the event exists with a delayed ingestion or arrival time.
   • Review receiver health, ingestion rate, throttling, indexing, consumer lag, or equivalent downstream metrics.
   • If the event exists downstream with a later ingestion time, investigate downstream processing or queueing.
   • If the event is absent downstream, continue with the Destination Live Data capture.
3. Check the Destination Live Data capture.
   • Navigate to Data > Destinations.
   • Click Live beside the Destination that should receive the event.
   • Filter the capture using the event’s distinctive source, host, index, or message content.
   • Set a capture duration long enough to collect representative events.
   • Record the event payload, _time, and relevant metadata.
   • Review the Destination Status, Charts, and Logs tabs for blocked status, errors, backpressure, or persistent-queue activity.
   • If the Destination capture shows current events but the downstream system does not, investigate the Destination connection or receiver.
   • If the Destination capture shows delayed events, continue with the Source Live Data capture.
4. Check the Source Live Data capture.
   • Navigate to Data > Sources.
   • Click Live beside the Source receiving the events.
   • Apply the same identifying filter used for the Destination capture.
   • Compare the event timestamp and payload with the Destination capture.
   • If the Source capture shows current events but the Destination capture is delayed, investigate routing, pipelines, worker health, or Destination backpressure.
   • If the Source capture shows delayed events, continue with the sending application or forwarder.
5. Check the actual source on the sending application.
   • Inspect the application output, log file, queue, or broker for current events.
   • For file-based collection, observe the file in real time with an appropriate operating-system command such as tail -f.
   • Compare the newest event written by the application with the newest event shown in the Source capture.
   • If the application output is delayed, investigate the application, its logging process, or its local queue.
   • If the application output is current but the Source capture is delayed, investigate the forwarder’s monitored-file state, read position, output queue, connection health, and forwarder logs.
   • For a Splunk Universal Forwarder, review splunkd.log for blocked output, queue, connection, or retry messages covering the delay window.
6. Identify the delay boundary.
   • Application current and Source delayed indicates a sender, forwarder, or transport issue.
   • Source current and Destination delayed indicates a Cribl routing, pipeline, worker, or Destination issue.
   • Destination current and downstream delayed indicates a receiver, network, indexing, or downstream processing issue.
   • Current payloads with unexpectedly old or future _time values indicate timestamp extraction or timezone handling rather than transport delay.
   • Document the timestamps, captures, status observations, and logs supporting the identified boundary.

Additional Information

For more information, see the following resources.

• Sources Overview explains how to start a Live Data capture from a Source and review Source status.
• Managing Destinations explains how to capture outgoing Destination data and review Destination status.
• Splunk TCP Source documents receiving data from Splunk Universal or Heavy Forwarders.
• Planning for Persistent Queues describes how Source- and Destination-side queueing affects metrics and captures.
• Tuning Splunk Universal Forwarders to Send to Stream provides additional Universal Forwarder performance guidance.