[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+

rfc2822

Can someone provide a test account with contacts that show the behaviour with DAVdroid 0.8.1, so that I can reproduce the issue?

I guess the file name is stored URL-encoded, unrelated to SQL escaping, but I can’t test as it works here.

AutoImport-julienval

Done, I have just sent you the details through your blog contact page.

rfc2822

Didn’t get anything, maybe hit by the spam filter or so. Can you please send it to play@bitfire.at again

AutoImport-mherzberg

Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.

You’re right, I was too quick to assume that. The encoding issue seems to come from the different http request types (propfind vs report). Anyway, please try if you can reproduce the issue as follows:

Create a new address book in the owncloud web interface and import the following as an contacts.vcf file (the issue is only present with two or more contacts):

BEGIN:VCARD
VERSION:3.0
PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
FN:FOO
N:;FOO;;;
UID:20140623T46782.acdf@foo.bar.vcf
REV:2015-07-15T01:18:20+00:00
END:VCARD

BEGIN:VCARD
VERSION:3.0
PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
FN:BAR
N:;BAR;;;
UID:20140623T00123.086fed@foo.bar.vcf
REV:2015-07-15T01:18:21+00:00
END:VCARD

Import this address book into DavDroid. This creates the following log lines for me. Please note the different encodings of ‘@’ in the Processing multi-status element lines.

I/davdroid.DavSyncAdapter﹕ Performing sync for authority com.android.contacts
D/davdroid.DavSyncAdapter﹕ Creating new DavHttpClient
V/davdroid.URIUtils﹕ Normalized URI https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ -> https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Using preemptive authentication (not compatible with Digest auth)
I/davdroid.SyncManager﹕ Remotely removing 0 deleted resource(s) (if not changed)
I/davdroid.SyncManager﹕ Uploading 0 new resource(s) (if not existing)
I/davdroid.SyncManager﹕ Uploading 0 modified resource(s) (if not changed)
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
D/davdroid.SyncManager﹕ Last local CTag = null; current remote CTag = 1437215802
I/davdroid.SyncManager﹕ Fetching remote resource list
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf
I/davdroid.SyncManager﹕ Fetching 2 new remote resource(s)
I/davdroid.resource﹕ Multi-getting 2 remote resource(s)
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf
D/davdroid.SyncManager﹕ Adding 20140623T00123.086fed@foo.bar.vcf.vcf
D/davdroid.Collection﹕ Committing 2 operations ...
D/davdroid.Collection﹕ ... 0 row(s) affected
D/davdroid.SyncManager﹕ Adding 20140623T46782.acdf@foo.bar.vcf.vcf
D/davdroid.Collection﹕ Committing 2 operations ...
D/davdroid.Collection﹕ ... 0 row(s) affected
I/davdroid.SyncManager﹕ Fetching 0 updated remote resource(s)
I/davdroid.SyncManager﹕ Removing non-dirty resources that are not present remotely anymore

Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.

rfc2822

Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.

Yes, this seems to reproducable here. I’ll have a look.

rfc2822

Let’s have a look at what is happening:

First, DAVdroid 0.8.1 requests a list of all members from the server (OwnCloud 8.1.0):

http-outgoing-0 >> "PROPFIND /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]"
http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/</d:href><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/">1437219316</x3:getctag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response></d:multistatus>[\n]"

The pretty-printed response is:

<d:response>
    <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href>
    <d:propstat>
        <d:prop>
            <d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
<d:response>
    <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href>
    <d:propstat>
        <d:prop>
            <d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
</d:multistatus>

So, there are two (new) members:

/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf (URL-encoded), i.e. member name 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf (URL-encoded), i.e. member name 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded)

DAVdroid requests those two resources using REPORT addressbook-multiget:

http-outgoing-0 >> "REPORT /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]"
…
http-outgoing-0 >> "<CD:addressbook-multiget xmlns:CD="urn:ietf:params:xml:ns:carddav" xmlns="DAV:">[\n]"
http-outgoing-0 >> "   <prop>[\n]"
http-outgoing-0 >> "      <CD:address-data content-type="text/vcard" version="4.0"/>[\n]"
http-outgoing-0 >> "      <getetag/>[\n]"
http-outgoing-0 >> "   </prop>[\n]"
http-outgoing-0 >> "   <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</href>[\n]"
http-outgoing-0 >> "   <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</href>[\n]"
http-outgoing-0 >> "</CD:addressbook-multiget>"

OwnCloud returns:

http-outgoing-0 << "HTTP/1.1 207 Multi-Status[\r][\n]"
…
http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD&#13;[\n]"
http-outgoing-0 << "VERSION:4.0&#13;[\n]"
http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN&#13;[\n]"
http-outgoing-0 << "FN:BAR&#13;[\n]"
http-outgoing-0 << "N:;BAR;;;&#13;[\n]"
http-outgoing-0 << "UID:20140623T00123.086fed@foo.bar.vcf&#13;[\n]"
http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00&#13;[\n]"
http-outgoing-0 << "END:VCARD&#13;[\n]"
http-outgoing-0 << "</card:address-data><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T4678"
http-outgoing-0 << "2.acdf@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD&#13;[\n]"
http-outgoing-0 << "VERSION:4.0&#13;[\n]"
http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN&#13;[\n]"
http-outgoing-0 << "FN:FOO&#13;[\n]"
http-outgoing-0 << "N:;FOO;;;&#13;[\n]"
http-outgoing-0 << "UID:20140623T46782.acdf@foo.bar.vcf&#13;[\n]"
http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00&#13;[\n]"
http-outgoing-0 << "END:VCARD&#13;[\n]"
http-outgoing-0 << "</card:address-data><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>[\n]"

So, if you have a look, you’ll see that DAVdroid requested the resources

/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf (URL-encoded), i.e. the member named 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded [%25 decodes to “%”]) and
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf (URL-encoded) i.e. the member named 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)

However, OwnCloud returned

/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf and
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf

This is clearly not what was requested and OwnCloud does one additional (and erroneous) level of encoding (as DAVdroid did earlier, by the way).

So, the sync process happens like this:

In response to PROPFIND, OwnCloud tells DAVdroid that there are two members:
- 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded) and
- 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)
DAVdroid requests these two (new) members from OwnCloud and adds all resources from the response. The response however
does not contain the requested members, but 20140623T00123.086fed@foo.bar.vcf.vcf and /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf.
These resources are added nevertheless (because DAVdroid expects them to be the requested ones, and it has to trust the server-given resource names to distinguish between the resources).
In the last step (“delete every local non-dirty contacts that are not on the server anymore”), DAVdroid deletes everything except 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded) and 20140623T46782.acdf%40foo.bar.vcf.vcf, so the recently fetched resources are deleted immediately (because their names are 20140623T00123.086fed@foo.bar.vcf.vcf and /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf).

Why does it work when there’s only one new record? In this case, DAVdroid uses a GET request instead of REPORT addressbook-multiget. The response is directly the VCard and the member name is used from the requested URI, so there’s no possibility for OwnCloud to return an erroneous file name.

Why did it work with previous DAVdroid versions? Well, DAVdroid did the same thing (double-decode URLs) earlier, too. This has been fixed to close #482. I guess it was a coincidence that this DAVdroid bug was just the the opposite of the corresponding OwnCloud bug so that it seemed to work (while it was bugged on both sides).

Do you agree?

AutoImport-mherzberg

Yes, I agree with you that the root cause seems to be inconsistent escaping of URIs by owncloud (URI escaped twice (%2540) in href-tags in PROPFIND responses vs no escaping in href-tags in REPORT responses). I don’t know how it’s handled elsewhere in owncloud, but I would expect no escaping at all in those href-tags. So this seems to be a bug in owncloud/the contact app after all. A workaround for this would be to unescape the href-tag twice in received PROPFIND responses, I guess? What do you think?

AutoImport-untitaker

@rfc2822 I think you’re hitting https://github.com/owncloud/contacts/issues/581. It’s going to be tricky/impossible to fix both #482 and simultaneously work around this. Even if ownCloud fixes this soon (it won’t), there will be other servers that do this.

What I’m currently doing is to first decode the filename and then encode it again except for the characters /@%:. This fixes only the most common instances of this problem where the encoded @ is causing trouble.

For the other rare cases I provide a repair command that fixes filenames to only use alphanumeric characters. I don’t think that’s within the scope of a sync app, but I’ve found it necessary as people came to me with the most perverse encoding problems. Now I can tell them to run the repair command and everything just works out.

rfc2822

@untitaker This could be the cause. The SabreDAV issue is closed, so couldn’t this be applied to OwnCloud too?

AutoImport-untitaker

The sabredav issue is closed because it’s a bug in ownCloud (somewhere in that conversation is a link to the exact line). I don’t know if e.g. Baikal has this issue.

rfc2822

Ok, so it seems the problem is caused by OwnCloud. Funny that it worked with older DAVdroid versions just because DAVdroid hat the opposite bug (it was quite a privacy killer to be honest, calling remote URLs just because of a file name).

AutoImport-untitaker

BTW what’s the behavior if a PROPFIND request returns a 404 for some hrefs? Are these hrefs skipped? You might be able to turn such silent failures into hard crashes by raising an error for those (an improvement IMO, but I understand if that’s not the UI you want)

rfc2822

BTW what’s the behavior if a PROPFIND request returns a 404 for some hrefs? Are these hrefs skipped?

For every response in a 207-multi-response that has <status>, the same checkResponse is called as it would be called for a single GET. For those which have <propstat> (i.e. the resource exists), only 10x and 20x are evaluated.

checkResponse will accept 10x and 20x silently and throw an exception for other status codes.

But it’s off-topic

AutoImport-untitaker

It’s somewhat related because I wonder how DavDroid would delete contacts if a PROPFIND request doesn’t return the requested hrefs.

On 18 July 2015 16:54:13 CEST, rfc2822 notifications@github.com wrote:

BTW what’s the behavior if a PROPFIND request returns a 404 for some
hrefs? Are these hrefs skipped?

For every response in a 207-multi-response, the same checkResponse
is
called
as it would be called for a single GET.

checkResponse
will accept 10x and 20x silently and throw an exception for other
status codes.

But it’s off-topic

Reply to this email directly or view it on GitHub:
https://github.com/bitfireAT/davdroid/issues/578#issuecomment-122550656

–
Sent from my phone. Please excuse my brevity.

rfc2822

It’s somewhat related because I wonder how DavDroid would delete contacts if a PROPFIND request doesn’t return the requested hrefs.

I don’t understand the question. DAVdroid deletes all local contacts which are not returned by PROPFIND, based on the file name, at the end of the sync process.

AutoImport-untitaker

Sorry, I think I confused PROPFIND with an addressbook-multiget REPORT.

On 18 July 2015 17:38:27 CEST, rfc2822 notifications@github.com wrote:

It’s somewhat related because I wonder how DavDroid would delete
contacts if a PROPFIND request doesn’t return the requested hrefs.

I don’t understand the question. DAVdroid deletes all local contacts
which are not returned by PROPFIND, based on the file name, at the end
of the sync process.

Reply to this email directly or view it on GitHub:
https://github.com/bitfireAT/davdroid/issues/578#issuecomment-122556983

–
Sent from my phone. Please excuse my brevity.

AutoImport-Eddy1979

Same problem when I restore my second smartphone.
Both phones have the same android (4.4.2) and davdroid (8.1) versions.
On my first smartphone the sync is working perfectly. On my second phone all contacts are deleted after a while.
On the 2nd phone, when I go in the settings: contacts are sent in vcard 4.0 format (always tick on)
On the first phone, when I go in the settings: contacts are sent in vcard 3.0 format (always tick off)
I don’t understand why???

rfc2822

@Eddy1979 Without detailed logs, there’s no way to analyze your problem. It’s very likely that you’re hit by the OwnCloud bug discussed here. The two phones probably behave differently because the accounts on the phones were created with different DAVdroid and/or OwnCloud versions. Uninstall and then install DAVdroid again on both phones and they will most likely behave identically. The VCard version of the server is detected when the account is created and shown in the Settings UI (it can’t be changed).

AutoImport-Eddy1979

@rfc2822 thank you for your fast reply.
If I understand the only one solution is to wait for an OC update.

AutoImport-mherzberg

Well, a possible workaround for the bug discussed here is, as mentioned somewhere above, to export all your contacts, make sure the ids do not contain special characters such as ‘@’, and then re-import them into your owncloud.

On July 24, 2015 1:17:01 PM GMT+02:00, Eddy1979 notifications@github.com wrote:

@rfc2822 thank you for your fast reply.
If I understand the only one solution is to wait for an OC update.

Reply to this email directly or view it on GitHub:
https://github.com/bitfireAT/davdroid/issues/578#issuecomment-124480698

[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+

Similar topics

switch for synchronisation of contacts or calendar only
Ideas, future talk, development • solved • • Merat

Error when syncing contacts (with Log)
DAVx⁵ • • AutoImport-nath5394

Sync tasks
DAVx⁵ • enhancement • • AutoImport-lemmy

[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+

Similar topics

switch for synchronisation of contacts or calendar only Ideas, future talk, development • solved • • Merat

Error when syncing contacts (with Log) DAVx⁵ • • AutoImport-nath5394

Sync tasks DAVx⁵ • enhancement • • AutoImport-lemmy

switch for synchronisation of contacts or calendar only
Ideas, future talk, development • solved • • Merat

Error when syncing contacts (with Log)
DAVx⁵ • • AutoImport-nath5394

Sync tasks
DAVx⁵ • enhancement • • AutoImport-lemmy