cribl HEC to only log data from last 24hrs

Pre-processing pipeline with a drop function and the following filter: _time * 1000 + 86400000 < Date.now()
 

when is use drop filter to drop events in pre-processing pipeline , would it go till route where i use the filter or i use the filter in pipeline itself.

The pre-processing pipeline is not part of the Route (so all the “intermediate” pipeline).  The logs are dropped as soon as they arrive in Cribl. If you need them, then don’t use this method, but put it on the Routes part.

I just want cribl to not log data older than 1d

for example in splunk we setup ignoreOlderThan = 1d

, how can that be achieved in cribl?

can i just use a time filter in route so it will automatically drop data older than 1d or i need to setup any pipeline 

In the way I told you. If you are certain that you do not need these logs, just filter them out as soon as you can through a pre-pipeline with the Drop function I posted above. 

Currently, Cribl have no time filter as I think you can imagine it; you need to create a specific pipeline with some logic to be able to drop logs that you don’t need.

Of course, one thing you need to check is that the Timestamp from events is the SAME in Unix time epoch of the field _time; otherwise unintended behaviour could arise and messing with the ingestion. This is usually true if the Event Breaker is correctly set up.