0.9.0.4 regression with OwnDrive

[kousu]

Since I installed 0.9.0.4 (via F-Droid) my Contact downloading is broken. Uploading still seems to work. Calendar and Task sync work. My server is my.owndrive.com which currently claims to be "OwnDrive 8.1.1 (stable)" (I assume that means ownCloud + custom edits?). Downgrading to 0.8.4.1 makes everything work again, when it eventually gets through negotiating (which takes forrrever on first sync).
Here's the log: https://gitlab.com/snippets/10055.

Isn't it suspicious that 0.9 has "DAVdroid now uses dav4android for WebDAV/CalDAV/CardDAV – completely rewritten library" while downgrading to 0.8 fixes it?

I've seen https://forums.bitfire.at/topic/805/sogo-contacts-gone-with-0-9-0-3 and https://forums.bitfire.at/topic/815/regression-davdroid-0-9-0-4-and-owncloud-8-0-4/3 but the error message is a blatant 404 so those issues don't seem like the same thing.

[rfc2822]

Hello,

I can't see anything suspicious or unclear. DAVdroid sends a REPORT addressbook-query, the server responds with 404, meaning the address book isn't there. Assuming that it is there, this is a server error.

... while downgrading to 0.8 fixes it?

It doesn't "fix" it. DAVdroid 0.8 uses PROPFIND instead of REPORT addressbook-query, so the (server) error doesn't occur. However, REPORT addressbook-query should be used for various reasons, so it has been introduced with DAVdroid 0.9.

Please contact your server manufacturer.

[kousu]

This is the CardDAV link from the ownCloud UI: "https://my.owndrive.com/remote.php/carddav/addressbooks/kousu/contacts" which is identical to the one in the backtrace, so the content is definitely there. Looking at that part of the trace closer now, though I don't know the DAV protocol, I do know HTTP, and I see that in HTTP you would never say

VERB https://my.owndrive.com/remote.php/carddav/addressbooks/kousu/contacts/

but only

VERB /remote.php/carddav/addressbooks/kousu/contacts/

Could that have something to do with it?

[rfc2822]

@kousu said:

Could that have something to do with it?

No. The full URL is given in the logs for debugging purposes, but not sent to the server. By the way, the request-uri of an HTTP request may be an absolute URI, too (which is for instance used for proxied connections). Also,

To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies.

But, as said before, DAVdroid doesn't even send the absolute uri, but an absolute path, as you would expect it.

[kousu]

I never knew HTTP was supposed to accept full URLs. Thank you for the cite.

I've tried poking my server manually.

With PROPFIND:

$ openssl s_client -quiet -connect my.owndrive.com:443
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL Wildcard, CN = *.owndrive.com
verify return:1
PROPFIND https://my.owndrive.com/remote.php/carddav/addressbooks/kousu/contacts/ HTTP/1.1
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxx=
Host: my.owndrive.com
Connection: close
HTTP/1.1 207 Multi-Status
Date: Sat, 31 Oct 2015 00:49:31 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-src *; img-src *; font-src 'self' data:; media-src *; connect-src *
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: Sameorigin
X-Robots-Tag: none
Set-Cookie: PHPSESSIDI=xxxxxxxxxxxxxxxxxxxxxxx; path=/; secure; HttpOnly
X-Sabre-Version: 2.1.6
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol, addressbook, access-control, calendarserver-principal-property-search
Server: Apache

3544
<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/remote.php/carddav/addressbooks/kousu/contacts/</d:href><d:propstat><d:prop><d:getlastmodified>Fri, 30 Oct 2015 04:50:02 GMT</d:getlastmodified><d:resourcetype><d:collection/><card:addressbook/></d:resourcetype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/carddav/addressbooks/kousu/contacts/a4b48ac12e.vcf</d:href><d:propstat><d:prop><d:getlastmodified>Mon, 19 Jan 2015 03:59:09 GMT</d:getlastmodified><d:getcontentlength>371</d:getcontentlength><d:resourcetype/><d:getetag>"43a9f3f8c2e23a1f1e19e3be6bc0436f"</d:getetag><d:getcontenttype>text/vcard; charset=utf-8</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/carddav/addressbooks/kousu/contacts/1079352a-7a63-4622-86ac-7e5409ff18c9.vcf</d:href><d:propstat><d:prop><d:getlastmodified>Wed, 01 Jul 2015 21:46:21 GMT</d:getlastmodified><d:getcontentlength>4403</d:getcontentlength><d:resourcetype/><d:getetag>"5b31341c4e897667efb5d11709de7e1d"</d:getetag><d:getcontenttype>text/vcard; charset=utf-8</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/carddav/addressbooks/kousu/contacts/https%253A%252F%252Fkousu%2540my.owndrive.com%252Fremote.php%252Fcarddav%252Faddressbooks%252Fkousu%252Fcontacts%252F3e977fee-f0fa-4a95-8779-14249c3b2862.vcf.vcf</d:href><d:propstat><d:prop><d:getlastmodified>Sat, 11 Jan 2014 00:06:15 GMT</d:getlastmodified><d:getcontentlength>733</d:getcontentlength><d:resourcetype/><d:getetag>"6b8c57e884192c360441c6c64e622b0c"</d:getetag><d:getcontenttype>text/vcard; charset=utf-8</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/carddav/addressbooks/kousu
[.....]

The REPORT addressbook-query spec confuses me: I think it's saying that not specifying a filter nor a set of properties should be the same as not filtering (i.e. do the same as unparameterized PROPFIND) and that the return format is the same as PROPFIND. Either way, since ownCloud is on SabreDAV, I should be able to trust its own docs, you'd think, and they agree with my interpretation. But if I use the exact example XML query they show I get the same "Card not found" error as DAVDroid causes.

$ openssl s_client -quiet -connect my.owndrive.com:443
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL Wildcard, CN = *.owndrive.com
verify return:1
REPORT https://my.owndrive.com/remote.php/carddav/addressbooks/kousu/contacts/ HTTP/1.1
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxx=
Host: my.owndrive.com
Connection: close
Depth: 1
Content-Type: application/xml; charset=utf-8
Content-Length: 190

<card:addressbook-query xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
    <d:prop>
        <d:getetag />
        <card:address-data />
    </d:prop>
</card:addressbook-query>

HTTP/1.1 404 Not Found
Date: Sat, 31 Oct 2015 00:47:27 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-src *; img-src *; font-src 'self' data:; media-src *; connect-src *
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: Sameorigin
X-Robots-Tag: none
Set-Cookie: PHPSESSIDI=xxxxxxxxxxxxxxxxxxxxxxx; path=/; secure; HttpOnly
X-Sabre-Version: 2.1.6
Server: Apache

ff
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:sabredav-version>2.1.6</s:sabredav-version>
  <s:exception>Sabre\DAV\Exception\NotFound</s:exception>
  <s:message>Card not found</s:message>
</d:error>

0

That error is somehow being triggered via at least 4 layers of indirection that I can't follow here. I wish I could get the server to just tell me that $name it's being called on. I don't care about which $name it chooses, I care about all the $names.

So, I agree that this is on ownCloud's end. You are following the spec and they, somehow, probably unintentionally, are not. However, you've already been making exceptions to deal with the transition:

• "Contacts sync: if REPORT addressbook-query doesn't work, don't ignore other exceptions than HTTP 40x errors"
• "Address book sync.: fall back to PROPFIND when REPORT addressbook-query returns 400, 403, 500 or 501 (for instance, for old OwnCloud versions)"

So, as a compatibility measure, would you add 404 to the list of fallback cases? If PROPFIND also 404s then you know the thing is really gone. I will bother the ownCloud people to see if they have a clue in the meantime.

What a painful protocol; there shouldn't be two ways to get the exact same bytes.

I really really appreciate DAVDroid; I was using proprietary implementations before, and before that exporting to my SDcard whenever I remembered (which ended badly multiple times), and before that Google. DAVDroid is the thing that lets me rely on my phone to behave sensibly in the cloud era.

[rfc2822]

Hello,

The REPORT addressbook-query spec confuses me: I think it's saying that not specifying a filter nor a set of properties should be the same as not filtering (i.e. do the same as unparameterized PROPFIND) and that the return format is the same as PROPFIND.

The only difference is that REPORT addressbook-query only returns address resources (VCards), while PROPFIND returns anything in that collection. As you can read in RFC 6352 5.2:

Address book collections MUST only contain address object resources and collections that are not address book collections.

So, there MUST NOT be other resources like .ics files or photo assets in the same collection, so this would not be a potential problem. However, there may be non-address book-sub-collections, i.e. a "photos/" sub-collection. This would be returned by PROPFIND, but not by REPORT addressbook-query.

Also, I have seen servers that use address book sub-collections (which is not allowed). In this case, they're also returned by PROPFIND, but not by REPORT addressbook-query.

For synchronization, we're not interested in any sub-collections or other data than the addresses themselves, so in my opinion, REPORT addressbook-query is the right way to do it.

That error is somehow being triggered via at least 4 layers of indirection that I can't follow here. I wish I could get the server to just tell me that $name it's being called on. I don't care about which $name it chooses, I care about all the $names.

As far as I know, there are several owncloud/contacts issues on GitHub that deal with this problem: a member resource can't be found for some reason (404), so the whole collection REPORT returns 404.

So, as a compatibility measure, would you add 404 to the list of fallback cases? If PROPFIND also 404s then you know the thing is really gone. I will bother the ownCloud people to see if they have a clue in the meantime.

Ok, thanks. When thinking about which error codes should be used for the workaround, I didn't choose 404 because it indicates a hard error and there's nothing much to interpret (while all other error codes may at least have an understandable reason to be returned). I will consider adding 404 to an upcoming version version of DAVdroid, but please report this to/fix this in OwnCloud

What a painful protocol; there shouldn't be two ways to get the exact same bytes.

They're not exactly the same (see above). Note that PROPFIND is plain WebDAV (you don't need a CardDAV server to synchronize using PROPFIND/GET/PUT/DELETE, a WebDAV server would be enough), while REPORT addressbook-query is CardDAV, so it makes sense (to me) that a CardDAV server is able to process such requests correctly – it's more or less the essence of the CardDAV extension to WebDAV.

I really really appreciate DAVDroid; I was using proprietary implementations before, and before that exporting to my SDcard whenever I remembered (which ended badly multiple times), and before that Google. DAVDroid is the thing that lets me rely on my phone to behave sensibly in the cloud era.

Thanks That's the goal of DAVdroid, and I'm happy when its useful for you.