Digest auth: Retry PROPFIND after 403?



  • Im am using DAVdroid 2.0.1-ose and tried to add an account using a URL with user and password.
    Using Thunderbird, the CALDAV account can be added and synchronization works. On android i tried e.g. CalDAV-Sync, which also works. In DAVdroid i get a DEBUG log (i replaced the actual URL with XXX.de because the server is located in the company i am working for, and i do not want to expose the information such that it can be found in the world wide web), where it say that access of the URL is forbidden (403). The debug log can be found here:
    https://paste.fedoraproject.org/paste/-WqQXPG36p1J2HYhWiI04Q
    This puzzles me, because if using e.g. curl to send a GET or PROPFIND without providing the credentials returns 401, as i would expect:

    curl -X PROPFIND -k -i https://XXX.de/calendars/users/philippr/calendar/ gives
    HTTP/1.1 401 Unauthorized

    However executing curl -X PROPFIND -k -i -u philippr:MY_PASS https://XXX.de/calendars/users/philippr/calendar/ gives
    HTTP/1.1 403 Forbidden

    and executing curl -X PROPFIND -k -i -u philippr:MY_PASS --digest https://XXX.de/calendars/users/philippr/calendar/ gives
    HTTP/1.1 401 Unauthorized
    followed by
    HTTP/1.1 207 Multi-Status with the actual content.

    My guess is that this is related to DAVdroid immediately trying to use basic auth when a HTTPS URL is used. In this case, it gets 403 by the server, because it only supports digest auth. Could DAVdroid be improved by trying both, basic auth and digest auth if it gets 403 before giving up?


  • developer

    @preinkemeier said in DAVdroid does not recognize CALDAV calendars (HTTP 403):

    However executing curl -X PROPFIND -k -i -u philippr:MY_PASS https://XXX.de/calendars/users/philippr/calendar/ gives
    HTTP/1.1 403 Forbidden

    So this is exactly what DAVdroid receives.

    My guess is that this is related to DAVdroid immediately trying to use basic auth when a HTTPS URL is used. In this case, it gets 403 by the server, because it only supports digest auth.

    If the server is configured properly, it will return 401 in this case and tell DAVdroid that only Digest is supported. DAVdroid will then retry the request with Digest.

    Could DAVdroid be improved by trying both, basic auth and digest auth if it gets 403 before giving up?

    As said above, this seems to be a server problem. 403 means Forbidden, and a client should not try again that request. If you want to tell a client that it shall use Digest, you'd have to use 401 for that.

    Does that help?

    Which server software are you using?

    And just for curiosity: Is there any reason to use Digest over HTTPS?



  • @rfc2822 said in Digest auth: Retry PROPFIND after 403?:

    @preinkemeier said in DAVdroid does not recognize CALDAV calendars (HTTP 403):

    However executing curl -X PROPFIND -k -i -u philippr:MY_PASS https://XXX.de/calendars/users/philippr/calendar/ gives
    HTTP/1.1 403 Forbidden

    So this is exactly what DAVdroid receives.

    My guess is that this is related to DAVdroid immediately trying to use basic auth when a HTTPS URL is used. In this case, it gets 403 by the server, because it only supports digest auth.

    If the server is configured properly, it will return 401 in this case and tell DAVdroid that only Digest is supported. DAVdroid will then retry the request with Digest.

    Could DAVdroid be improved by trying both, basic auth and digest auth if it gets 403 before giving up?

    As said above, this seems to be a server problem. 403 means Forbidden, and a client should not try again that request. If you want to tell a client that it shall use Digest, you'd have to use 401 for that.

    Does that help?

    Well... No. I cannot use DAVdroid with the current server configuration. But i understand your argument and agree that the response by the server is not really what one would expect.
    Given that other clients Like Thunderbird, CalDAV-Sync and CalDAV Sync Free Beta just work, the situation is still disappointing though.
    This makes me think of another approach: What If DAVdroid would, only during account setup, first try to PROPFIND the specified URL without sending any authorization header. The server then has to respond with the requested kind of authorization and DAVdroid could then choose the one requested by the server. To me this sounds like a more robust strategy, which is also more in line with how the http authorization framework is expected to work. Of course this comes with the cost of additional HTTP requests, but this could be limited to the account setup. Later synchronizations done by DAVdroid could simply use the authorization scheme detected during account setup.
    What do you say?

    Which server software are you using?

    The DAVdroid log says "Server: Twisted/8.1.0 TwistedWeb/[twisted.web2, version 0.2.0] TwistedCalDAV/1.2 (r)>"
    However i die not verify this.

    And just for curiosity: Is there any reason to use Digest over HTTPS?

    Personal preference by the admin i think... Don't know.


  • developer

    @preinkemeier This seems to be a server problem, as you have already shown by curl. Why not fix it on the server side? Did you already contact your admin/server vendor?



  • @rfc2822 said in Digest auth: Retry PROPFIND after 403?:

    @preinkemeier This seems to be a server problem, as you have already shown by curl. Why not fix it on the server side? Did you already contact your admin/server vendor?

    I already agreed with you that this is a server issue and yes, i contacted the admin. Maybe he is willing to either fix the issue or to enable Basic Auth over HTTPS (no response yet).
    I still think my suggestion made above is worth considering, independently of whether the server i used gets fixed, because it should increase robustness of DAVdroid regarding CalDAV and CardDAV service detection during account setup. DAVdroid could then reliably detect these services even for servers that are not properly configured like the one i use. If i would have made a suggestion to introduce a dirty workaround for a particular server software, then certainly i would understand that you are not willing to add such dirty workarounds. But i think the suggestion i made would just increase the number of server software and their configuration DAVdroid works with, without introducing nasty server software specific code paths.
    In case you or another developer with commit access does not think it is worth the effort, would you at least take a patch, if i find some spare time in implementing my suggestion? And if so, could you give me some pointers regarding the source code, where i have to look into?


  • developer

    @preinkemeier Thanks for your suggestions. As I understand it, retrying a request with the same credentials after receiving 403 is against RFC 7231 6.5.3:

    If authentication credentials were provided in the request, the server considers them insufficient to grant access. The client SHOULD NOT automatically repeat the request with the same credentials. The client MAY repeat the request with new or different credentials. However, a request might be forbidden for reasons unrelated to the credentials.

    and RFC 7235 3.1, which should be used instead:

    If the request included authentication credentials, then the 401 response indicates that authorization has been refused for those credentials. The user agent MAY repeat the request with a new or replaced Authorization header field (Section 4.2).

    This would have severe implications. For instance, if DAVdroid suddently receives 403 while syncing using Basic auth, does that mean that it should try Digest at will for specific resources, too? Or maybe even other authentication methods?

    My interpretation is that 403 means: server has understood and accepted user name/password, but the client is not allowed to access the resource with these credentials, while 401 means: unauthorized = credentials not accepted.

    Returning 403 instead of 401 is broken behavior and I think it can't be fixed by client-side workarounds. They will only make it worse. Implementation would also be quite cumbersome and dirty, because the whole authentication concept (of okhttp) relies on correct HTTP status code usage.

    DAVdroid tries Basic automatically (without waiting for 401) on HTTPS connections, which shouldn't cause problems according to RFC 7235 2.1:

    A user agent that wishes to authenticate itself with an origin server -- usually, but not necessarily, after receiving a 401 (Unauthorized) -- can do so by including an Authorization header field with the request.



  • @rfc2822 said in Digest auth: Retry PROPFIND after 403?:

    @preinkemeier Thanks for your suggestions. As I understand it, retrying a request with the same credentials after receiving 403 is against RFC 7231 6.5.3:

    I agree.

    If authentication credentials were provided in the request, the server considers them insufficient to grant access. The client SHOULD NOT automatically repeat the request with the same credentials. The client MAY repeat the request with new or different credentials. However, a request might be forbidden for reasons unrelated to the credentials.

    and RFC 7235 3.1, which should be used instead:

    If the request included authentication credentials, then the 401 response indicates that authorization has been refused for those credentials. The user agent MAY repeat the request with a new or replaced Authorization header field (Section 4.2).

    Again, i agree.

    This would have severe implications. For instance, if DAVdroid suddently receives 403 while syncing using Basic auth, does that mean that it should try Digest at will for specific resources, too? Or maybe even other authentication methods?

    No, or should not. But contrary to the title of this thread, i do NOT propose anymore to retry on receiving 403.

    My interpretation is that 403 means: server has understood and accepted user name/password, but the client is not allowed to access the resource with these credentials, while 401 means: unauthorized = credentials not accepted.

    Returning 403 instead of 401 is broken behavior and I think it can't be fixed by client-side workarounds. They will only make it worse. Implementation would also be quite cumbersome and dirty, because the whole authentication concept (of okhttp) relies on correct HTTP status code usage.

    I agree.

    DAVdroid tries Basic automatically (without waiting for 401) on HTTPS connections, which shouldn't cause problems according to RFC 7235 2.1:

    A user agent that wishes to authenticate itself with an origin server -- usually, but not necessarily, after receiving a 401 (Unauthorized) -- can do so by including an Authorization header field with the request.

    And this ist exactly what i propose to change. DAVdroid could just be more friendly during service detection and first issue a request, parsing the server response for the expected authorization scheme, instead of trying basic auth right away. Afterwards, when synchronizing, it could use the detected auth scheme, directly sending the auth Header with the credentials.
    I repeat, i am NOT proposing to retry authentication upon receiving 403.


  • developer

    @preinkemeier I see. I think it would be possible, but add much complexity and confusion just to work around a server bug, so I don't think it should be implemented in DAVdroid.

    Please keep us up to date about whether this can be solved on server side. If we could know which server software it is, we could even test it ourselves and have a look.



  • @rfc2822 said in Digest auth: Retry PROPFIND after 403?:

    @preinkemeier I see. I think it would be possible, but add much complexity and confusion just to work around a server bug, so I don't think it should be implemented in DAVdroid.

    OK, then i know at least that it would be a waste of time to provide a patch.


  • developer

    @preinkemeier You could use the patched version for yourself and maybe others in your company 🙂



  • @rfc2822 said in Digest auth: Retry PROPFIND after 403?:

    @preinkemeier You could use the patched version for yourself and maybe others in your company 🙂

    Yes, good idea. I just have to figure out how to compile DAVdroid. Could you give me a pointer?


  • developer

    @preinkemeier This should work: https://gitlab.com/bitfireAT/davdroid/wikis/how-to-compile (linked from the "Open Source" section on davdroid.com)