Office 365 Activity Source Ingestion Lag Setting Confusion With Event Timestamp Filtering

Thanks, so it sounds like this skew is applied to the content blob discovery performed on the /subscriptions/content?contentType={ContentType}&startTime={0}&endTime={1} path. So this will ensure if a content blob for a subscription becomes available with a contentCreated timestamp in the past, it would still be collected, but the individual events within each content blob, which have expected delays, are not affected.

If this is the case, why is Ingestion lag (minutes) setting even required if contentCreated is the time the content blob became available for retrieval? The documentation defines the value as "The datetime when the content was made available", so it doesn't sound like there would be delays that need to be handled during the discovery phase, especially not delays up to 24-72 hours, but maybe Cribl has observed Microsoft publishing content blobs in the past.

Cribl only asks for that time frame once. If the ingestion delay on MS side is larger than your time skew (ingest lag configured in Cribl), you will just miss the event.

I think my main point of confusion is whether the skew is applied during the discovery phase when available blobs are discovered or during the collection phase when events are extracted from within each discovered blob. Based on what I have read from Microsoft's documentation, the only delay that should be expected is the event delivery delay within each blob. For example, a blob can contain events from 5 minutes ago and 5 days ago. But the actual contentCreated value that is used to mark when each blob is made available to retrievers and is used by collectors (I am assuming this is what Cribl uses) to find the blobs within the defined interval for each subscription should not be delayed. If the the ingestion lag being set to 0 causes you to miss events that are delayed on the Microsoft side, then it sounds like the Ingestion lag targets the actual event timestamps within each content blob and behaves similar to the standard collector event time filtering.

Okay, I found Cribl's implementation of this REST collector and see the ingestion lag is applied during the discovery phase. https://github.com/criblio/collector-templates/blob/main/collectors/rest/o365_activity/O365_Activity-SharePoint.json `` "discovery": { "discoverType": "http", "discoverMethod": "get", "pagination": { "type": "response_header", "maxPages": 0, "attribute": [ "nextpageuri" ] }, "enableStrictDiscoverParsing": false, "enableDiscoverCode": false, "discoverUrl": "https://manage.office.com/api/v1.0/${C.vars.o365_tenant_id}/activity/feed/subscriptions/content", "discoverRequestParams": [ { "name": "PublisherIdentifier", "value": "${C.vars.o365_publisher_id}" }, { "name": "contentType", "value": "\"Audit.SharePoint\"" }, { "name": "startTime", "value": "${state.latestTime != null ? new Date(state.latestTime 1000).toISOString() : earliest != null ? new Date((earliest - (C.vars['o365_ingestion_lag_min'] || 0) 60) 1000).toISOString() : new Date(Date.now() - (C.vars['o365_ingestion_lag_min'] || 0) 60 1000 - (C.vars['o365_polling_interval_min'] || 5) 60 * 1000).toISOString()}" }, { "name": "endTime", "value": "${latest != null ? new Date((latest - (C.vars['o365_ingestion_lag_min'] || 0) 60) 1000).toISOString() : new Date(Date.now() - (C.vars['o365_ingestion_lag_min'] || 0) 60 1000).toISOString()}" } ] }``

Knowing that the ingestion lag is used during the discovery phase, I still am not sure why this is even required. It seems as if this implementation is designed to address delays in individual event delivery, but discovery filters based on the time the actual blob became available - independent of the timestamps of the events within the blob. From the Microsoft docs:

• The contentCreated property is not the date that the event being notified was created. This is the date the notification was created. The events detailed in that blob may have been created well before the content blob was created. Therefore, you can never query the API directly for events that occurred within any given period.

The only way I could miss events with an ingestion delay set to 0 is if Microsoft published a blob with a contentCreated time in the past, but if this value is the exact time it becomes available for retrieval and thus would be discoverable by the Cribl collector job, that does not seem plausible.

If the blob is really available at "contentCreated" time, you are right. However, I assume that the ingestion lag has been added for a reason.

Yeah, I agree. Thanks for looking into this with me!