Symptom
Events are visible entering a pipeline in Live Capture or Preview, but rarely or never reach the configured Destination(s). Destination output metrics show near-zero throughput for
extended periods (sometimes hours), punctuated by occasional large bursts of delayed events.
Environment
- Cribl Stream (Worker Node or Hybrid Worker Node)
- Syslog Source using RFC3164-style timestamps (no timezone/offset in the raw timestamp)
- A pipeline containing an Aggregations function downstream of the Syslog Source
Resolution
- Open Live Capture on the affected Syslog Source, before the pipeline stage.
- Compare a captured event's _time value to the actual current time.
- Note the offset if _time is ahead of or behind actual time by a consistent number of hours.
- Navigate to the Syslog Source's configuration.
- Open Advanced Settings.
- Set the Default Timezone field to the sending device's actual timezone, instead of leaving it at the default local-time assumption.
- Adjust the selected timezone by the observed offset if the device's nominal timezone alone does not fully correct the drift.
- Save the Source configuration and apply changes to affected Worker Groups.
- Recheck Live Capture to confirm _time now matches actual time.
- Confirm events resume flowing to the configured Destination(s).
Cause
RFC3164 syslog timestamps carry no timezone or UTC offset information. When a Syslog Source's Default Timezone is left at its default (the Worker's local OS timezone) or set incorrectly, and the sending device's actual timezone differs, Cribl calculates the wrong epoch value for _time. In this case, the mismatch caused _time to be translated to a time in the future.
Downstream, if the pipeline contains an Aggregations function, that function buckets events by their own _time, snapped to a fixed time window. A bucket only becomes eligible to flush once wall-clock time has passed the bucket's window boundary. Because the affected events carried a future-dated _time, their bucket's window boundary stayed ahead of real time indefinitely, so the time-based flush condition never triggered. With no explicit event-count or memory limit configured on the Aggregations function, the buffered events were never released under normal operation. They were only ever flushed when the pipeline process restarted, producing the intermittent large bursts observed.
Additional Information
- A related but distinct issue is documented in Events Not Appearing in Splunk Due to Autotimestamp Timezone Mismatch in Stream, which covers the same timezone-assumption mechanism via the pipeline's Autotimestamp function rather than the Syslog input's own Default Timezone setting, and results in delayed search visibility rather than indefinite buffering.
- See How does a worker’s syslog input convert the timestamp in the _raw field to epoch time in _time field? for general background on how the Syslog input resolves _time.
- As a defense-in-depth measure independent of this fix, consider configuring an explicit Flush Event Limit or Flush Memory Limit on any Aggregations function in a pipeline. This bounds how long events can be buffered even if a similar timestamp mismatch recurs from a different source.