Provide support for MailboxNotEnabledForRESTAPI

[jimdigriz]

I have a mailbox that up until recently worked fine with EWS, but it has now stopped working. This was the configuration I used:

davmail.mode=O365Manual
davmail.url=https://outlook.office365.com/EWS/Exchange.asmx
davmail.oauth.clientId=d3590ed6-52b3-4102-aeff-aad2292ab01c
davmail.oauth.redirectUri=urn:ietf:wg:oauth:2.0:oob

As an error I now see:

DEBUG [ImapConnection-55458] org.apache.http.wire  - http-outgoing-1 << "{"error":"invalid_grant","error_description":"AADSTS9002313: Invalid request. Request is malformed or invalid. Trace ID: ... Correlation ID: ... Timestamp: ...","error_codes":[9002313],"timestamp":"...","trace_id":"...","correlation_id":"...","error_uri":"https://login.microsoftonline.com/error?code=9002313"}"

[snipped]

WARN  [ImapConnection-55458] davmail.exchange.ExchangeSession  - All network interfaces down or host unreachable !
DEBUG [ImapConnection-55458] davmail.exchange.ExchangeSession  - java.io.IOException: invalid_grant AADSTS9002313: Invalid request. Request is malformed or invalid. Trace ID: ... Correlation ID: ... Timestamp: ...
java.io.IOException: invalid_grant AADSTS9002313: Invalid request. Request is malformed or invalid. Trace ID: ... Correlation ID: ... Timestamp: ...	at davmail.exchange.auth.O365Token.setJsonToken(Unknown Source)
	at davmail.exchange.auth.O365Token.executeRequest(Unknown Source)
	at davmail.exchange.auth.O365Token.<init>(Unknown Source)
	at davmail.exchange.auth.O365Token.build(Unknown Source)
	at davmail.exchange.auth.O365ManualAuthenticator.authenticate(Unknown Source)
	at davmail.exchange.ExchangeSessionFactory.getInstance(Unknown Source)
	at davmail.exchange.ExchangeSessionFactory.getInstance(Unknown Source)
	at davmail.imap.ImapConnection.run(Unknown Source)

I knew about your work in #404 so I went and tried it and though I can get the access token, when trying to access the mailbox the error MailboxNotEnabledForRESTAPI is returned.

Unfortunately for me it looks like both EWS and Graph is a non-option; I am not an administrator of O365 and the IT team refuse to allow anything other than Outlook.

So I dug around a bit, and I get closer if I extract the refresh token from the O365 web based Outlook, apply my commits in #439 and #441 and use the following values:

davmail.oauth.clientId=9199bf20-a13f-4107-85dc-02114787ef48
davmail.oauth.redirectUri=https://outlook.office.com/mail/oauthRedirect.html
davmail.oauth.scope=openid profile offline_access https://outlook.office.com/Mail.ReadWrite https://outlook.office.com/Calendars.ReadWrite https://outlook.office.com/MailboxSettings.Read https://outlook.office.com/Mail.ReadWrite.Shared https://outlook.office.com/Contacts.ReadWrite https://outlook.office.com/Tasks.ReadWrite https://outlook.office.com/Mail.Send
davmail.oauth.refreshTokenOrigin=https://outlook.office.com
davmail.enableGraph=true
davmail.graphUrl=https://outlook.office.com
davmail.graphPrefix=api/v2.0
davmail.url=https://outlook.office365.com
davmail.oauth.bob@example.com.refreshToken={AES}...

We are though leaning into the Outlook REST API which is similar but though we can do the refresh token dance now we get:

DEBUG [ImapConnection-34534] org.apache.http.headers  - http-outgoing-9 >> GET /api/v2.0/me/mailFolders/msgfolderroot?%24expand=singleValueExtendedProperties%28%24filter%3Did+eq+%27SystemTime+0x3008%27+or+id+eq+%27String+0x3613%27+or+id+eq+%27SystemTime+0x670a%27+or+id+eq+%27Integer+0x6751%27%29 HTTP/1.1
DEBUG [ImapConnection-34534] org.apache.http.headers  - http-outgoing-9 >> Host: outlook.office.com

[snipped[

DEBUG [ImapConnection-34534] org.apache.http.impl.conn.PoolingHttpClientConnectionManager  - Connection released: [id: 3][route: {s}->https://outlook.office.com:443][total available: 1; route allocated: 1 of 5; total allocated: 1 of 20]
ERROR [ImapConnection-34534] davmail  - RequestBroker--ParseUri Could not find a property named 'id' on type 'Microsoft.OutlookServices.SingleValueLegacyExtendedProperty'.
java.io.IOException: RequestBroker--ParseUri Could not find a property named 'id' on type 'Microsoft.OutlookServices.SingleValueLegacyExtendedProperty'.
        at davmail.exchange.graph.JsonResponseHandler.handleResponse(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.executeJsonRequest(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.getWellKnownFolderId(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.getSubFolderIdIfExists(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.getFolderIdIfExists(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.getFolderId(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.internalGetFolder(Unknown Source)
        at davmail.exchange.graph.GraphExchangeSession.internalGetFolder(Unknown Source)
        at davmail.exchange.ExchangeSession.getFolder(Unknown Source)
        at davmail.imap.ImapConnection.run(Unknown Source)
DEBUG [ImapConnection-34534] davmail  - > a0003 BAD unable to handle request: RequestBroker--ParseUri Could not find a property named 'id' on type 'Microsoft.OutlookServices.SingleValueLegacyExtendedProperty'.

The Compare Microsoft Graph and Outlook REST API endpoints shows similarity, but there is obviously a bit more to it than just a straight up swap of the endpoint URL :)

Is it feasible to also support the Outlook REST API? It still seems to be used by the O365 Outlook web client.

I am happy to help; the prospect of having to use Outlook turns out to be a great motivator. :)

[mguessan]

So it seems your admins blocked Outlook desktop and you only have access to Outlook web ?

I played with your patches a bit and couldn't get past the "AADSTS9002325: Proof Key for Code Exchange is required for cross-origin authorization code redemption." error.

Issue is not on token endpoint, I get it directly on the authorize request, can you please double check the clientId/redirect_uri you used to obtain a refresh token ?

[jimdigriz]

So it seems your admins blocked Outlook desktop and you only have access to Outlook web ?

I do not use have access to a copy of Outlook Desktop, if it helps do you want me to spin up a Windows VM and install Outlook?

My guess, like Outlook Web, it would just use the same scopes (https://outlook.office.com/Mail.ReadWrite, etc)?

The 'invalid grant' I thought was more "client ID is not authorized for these scopes", where the end user is unable to provide consent for the scopes and it requires an administrator? I assume this might mean Outlook Desktop used to use these scopes, they no longer do.

Issue is not on token endpoint, I get it directly on the authorize request, can you please double check the clientId/redirect_uri you used to obtain a refresh token?

Here is a shell implementation of the juicey part of what my browser does (I have the HAR archived), note this must use PKCE but response_mode=fragment not necessary (only used here to guard against browser caching):

CHALLENGE=$(dd if=/dev/urandom bs=32 count=1 status=none | base64 -w0)
CHALLENGE_S256=$(printf '%s' "$CHALLENGE" | openssl sha256 -binary | base64 -w0 | tr '+/' '-_' | tr -d '=')
printf 'Open in your browser: %s\n' "https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize?client_id=9199bf20-a13f-4107-85dc-02114787ef48&scope=https%3A%2F%2Foutlook.office.com%2F.default%20openid%20profile%20offline_access&redirect_uri=https%3A%2F%2Foutlook.office.com%2Fmail%2FoauthRedirect.html&response_type=code&response_mode=fragment&code_challenge_method=S256&code_challenge=${CHALLENGE_S256}"

Now extract the value of code from the fragment response and run this to get your tokens (including the refresh_token):

curl -s --fail-with-body -D /dev/stderr \
        -H 'accept: application/json' \
        -H 'content-type: application/x-www-form-urlencoded' \
        -H 'origin: https://outlook.office.com' \
        --data-urlencode client_id=9199bf20-a13f-4107-85dc-02114787ef48 \
        --data-urlencode redirect_uri=https://outlook.office.com/mail/oauthRedirect.html \
        --data-urlencode scope=https://outlook.office.com/.default openid profile offline_access \
        --data-urlencode code=... \
        --data-urlencode code_verifier=$CHALLENGE \
        --data-urlencode grant_type=authorization_code \
        'https://login.microsoftonline.com/organizations/oauth2/v2.0/token'

N.B. HTTP origin header needs to be set as this is for a SPA