What Report URL are you using to collect Office 365 Message Trace collector logs?

Not sure if anyone is successfully getting message trace logs. Something has changed on the Microsoft side with the API causing issues. I know all the add on for Splunk are having the same issue and they are working with Microsoft to figure out the issue.

Thanks Shawn. We originally thought our outage was because Microsoft recently disabled basic authentication to their API (we were using basic authentication) and we weren't using OAuth2. But it sounds like, even if we get OAuth2 working, we may still not be able to acquire O365 Message Trace logs.

Yeah the first big change was the basic auth being disabled. But I think they inadvertently changed something else with the final swap to OAUTH. We were using OAUTH for at least 2 months or so for message trace and it was working fine then stopped working (only pulls in a few events and then a 401 error pops).

Thank you for sharing that information Shawn. I will post here if I have any further significant updates.

This seems to be significant. From Microsoft:Under the Creating Service Account section:"Before accessing Office 365 Message Trace service we need to create Office 365 service account. This account needs to have very strong password (as there's *no* OAuth 2.0)."https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/office-365-email-activity-and-data-exfiltration-detection/ba-p/1169652

That link is from 2020, they did not have OAUTH then

k

<@UJYBKAR2Q&gt;This may be old information to you, but someone from Cribl says they (Cribl) can reproduce the error and have a case open with Microsoft on this. Cribl says that sometimes it will work (as you've have described) while other times the behavior varies with the API providing a redirect page (which Cribl isn't expecting in their O365 MT collector).It may not help solve our issue, atm, but at least I have a better understanding of what's happening.

There's supposed to be a "Known Issue" posted about this.

Whoops, I was just told they've already posted this in Known Issues:https://docs.cribl.io/stream/known-issues/#2023-04-14--v2241x--requests-from-office-365-message-trace-source-fail-intermittently-cribl-16929

Quick update: Microsoft has acknowledged the issue on their side and their support engineers are able to repro the same thing we're seeing. No ETA yet, but they've told us they're working on a fix

<@UJYBKAR2Q&gt;Hi Shawn. Just an update. We've changed our authentication from Basic to OAuth and now we're obtaining O365 Message Trace logs again. We're also getting the occasional "Non-whitespace before {[" error, but is occurring about 1/14 times.

Our Splunk support case said Microsoft is releasing a patch on their side to correct this so we are waiting for confirmation that has been done

Good news. Thanks Shawn!

Ugh. Scratch my last statement about the error rate (1 error out of 14 tries). Cribl isn't "happy" about that connector right now: 9 errors out of 13 tries. Looking forward to the MS patch!:

Looks like it's back up:

from MSFT:> We got an update from Engineering Team that the issue should be resolved. The patch has been deployed and is at ~99% saturation. Please check and let me know if you are still seeing the Reporting Web Services randomly redirecting to the Office Login Page or the 403.

We've had 5 clean runs in a row since you posted. The previous errors below were 500 codes.