This message originated from Cribl Community Slack.
Click here to view the original link.
I am trying to collect CATO logs using the REST Collector.
The current issue is that logs retrieved via the REST API are being duplicated. When retrieving CATO logs through the REST API, I would like to avoid collecting data that was already fetched previously.
Is it possible to prevent previously collected data from being ingested, for example by using timestamp information? If this is not supported as a built-in Cribl feature, could you please share any best practices to handle this?
Solved
CATO Logs Duplicated When Collected Via REST API
Best answer by Stefan Laschitzki
You can either use the earliest and latest vales from the collection jobs and configure a "lookback" interval or use state tracking. The latter is more reliable, but a bit challenging to set up
https://docs.cribl.io/stream/usecase-rest#state-tracking-time
https://docs.cribl.io/stream/collectors-rest#time-range
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.