Symptom
Events that arrive through a Splunk TCP Source from some Splunk Universal Forwarders (UFs) have a _time that is exactly one hour off (earlier than the real time) after the daylight saving time (DST) change. Other UFs sending to the same Source, with the same Pipeline and Event Breaker rules, show the correct _time.
A live capture of an affected event shows a __TZ field with a single offset and no end date, for example:
__TZ: name: s2s-forwarder-tz offsets: [-120] untils: [null]
An unaffected event from another UF shows a long list of entries (for example 174 offsets and 174 untils). The _time is the same before and after the Pipeline, so the Pipeline does not cause the shift.
Environment
- Cribl Stream 4.17.1 (self-managed). Other versions with S2S v4 support behave the same way in the code reviewed.
- Splunk TCP Source
- Splunk Universal Forwarders 9.3.x and 9.4.x, on Windows and Linux
- Events whose raw text has a timestamp without a time zone offset
Resolution
- Capture live data on the Splunk TCP Source with internal fields shown (Live Data). Expand
__TZand compare an affected event with an unaffected one. - If the affected UFs send a single entry (
offsetswith one value,untils: [null]) and the unaffected UFs send a long list, check the time zone configured on the affected UF hosts:- Windows: run
tzutil /gandw32tm /tz. A zone without DST rules showsDate:(unspecified)for Standard and Daylight. - Linux: run
timedatectlanddate +"%Z %z".
- Windows: run
- Set the host time zone to a zone with DST rules that matches the time zone in which the applications write their logs (for example
W. Europe Standard Timeon Windows,Europe/Berlinon Linux). - Restart the UF so that it reads the new time zone. The UF sends its time zone once per connection, at the start of the connection.
- Capture live data again and confirm that
__TZnow shows a long list of entries and that_timeis correct.
Workaround
If the host time zone cannot be changed, let the Event Breaker rule decide the time zone:
- Open the Splunk TCP Source and go to Advanced Settings.
- Set Use Universal Forwarder time zone to Off.
- In the Event Breaker rules for the affected sourcetypes, set Timezone to the time zone in which the application writes its timestamps.
This setting applies to all UFs that send to the Source. Check that the Event Breaker rules of all sourcetypes on that Source have a correct time zone before turning it off.
Cause
A Splunk Universal Forwarder sends the time zone of its host to Cribl Stream at the start of each S2S v4 connection. The data consists of one or more local time entries and a list of transitions (the dates on which the offset changes).
- A Windows host in a zone with DST rules sends both entries (DST and Standard) and the full list of transitions. Turning off Adjust for daylight saving time automatically on such a host does not change this.
- A Windows host in a zone without DST rules (for example a fixed UTC+02:00 zone) sends one entry and no transitions.
When Use Universal Forwarder time zone is on, Cribl Stream stores this information as __TZ on every event the UF has not already broken. If the data has no transitions, Cribl Stream uses the single entry for all time (offsets with one value, untils: [null]). The Event Breaker uses __TZ for timestamps without a time zone, and it takes precedence over the time zone configured in the Event Breaker rule.
If the application writes local time that observes DST (for example Berlin time) but the UF host advertises a fixed offset (for example +02:00), the offset is correct only while that local time is on summer time. After the change back to standard time, the events are one hour early.
Events that have an explicit offset in the raw text (for example +0200) are not affected.
Additional Information
The meaning of __isBroken = true on these events: the Event Breaker has produced complete events. It does not mean that events are truncated and is not related to this issue.
