[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+

AutoImport-mokkin

In my case the the sync of several addressbooks from Owncloud 8.1 with DavDroid 0.8.1 works fine, but the calendar sync doesn’t work. The account setup works and it says “synced” but nothing is displayed.

AutoImport-julienval

However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.

I would be happy to test your change if you can supply a precompiled apk
I can reproduce the issue on both of my phones

AutoImport-flocke

I would be available for testing as well. Just upload a apk somewhere and give me the link.

AutoImport-mherzberg

Here is the link to my test build: davdroid-debug.apk
Please note that this is just a test build from me related to this issue and should not be used for daily usage. I do not take any responsibility for any harm caused by using this apk.

This is the git diff from the code I used:

diff --git a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
index 65e2891..3479ac9 100644
--- a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
+++ b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
@@ -139,7 +139,7 @@ public class LocalAddressBook extends LocalCollection<Contact> {
                if (remoteResources.length != 0) {
                        List<String> sqlFileNames = new LinkedList<>();
                        for (Resource res : remoteResources)
-                               sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()));
+                               sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", "@"));
                        where = entryColumnRemoteName() + " NOT IN (" + StringUtils.join(sqlFileNames, ",") + ")";
                } else
                        where = entryColumnRemoteName() + " IS NOT NULL";

rfc2822

However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.

I can’t reproduce that.

Log.i(TAG, "SQLite escaped: " + DatabaseUtils.sqlEscapeString("a@b.com"));

gives a@b.com here.

However the sqlEscapeString method escapes the @ to %40

Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.

rfc2822

When trying to reproduce this bug, I ran into https://github.com/owncloud/contacts/issues/808. Is sharing enabled in your installation? If not, can you try to enable it?

Can you please

delete or rename your OwnCloud log (data/owncloud.log)
do a DAVdroid sync
access Contacts module in the Web Interface
post the contents of your owncloud.log

AutoImport-protist

I can confirm that mherzberg’s apk fixes this for me. Thank you!

Previously, I was getting error messages in the notification menu every hour or so. Unfortunately, I forget what they said precisely.

I attempted uninstalling DAVdroid, then installing version 0.7.7. My phone would only import three contacts (out of 150). Oddly enough, I think these were the only contacts that I had added via the phone/DAVdroid. The others had been added/imported via OwnCloud’s web interface a while ago.

I also attempted installing DAVdroid 0.8.0 and 0.8.1. Neither fixed this issue. Only when installing mherzberg fix did it work. Thanks again!

AutoImport-protist

Ahhh… after a while I’ve noticed mherzberg’s apk is buggy too. Every time DAVdroid syncs, it creates an additional copy of each contact, except the three contacts that were the only ones synced in the official versions of DAVdroid (as per my previous post). Hence, after three syncs, I have one copy of these original three in my Android address book as expected, but three copies of every other contact!

AutoImport-julienval

@rfc822 : sharing is enabled on my OC setup. I have followed your steps but my log file is empty after calling the contact page.

I have installed mherzberg’s apk on both of my phones and it seems to fix the issue. I have tried several syncs and I haven’t encountered the issue reported by protist.

rfc2822

Can someone provide a test account with contacts that show the behaviour with DAVdroid 0.8.1, so that I can reproduce the issue?

I guess the file name is stored URL-encoded, unrelated to SQL escaping, but I can’t test as it works here.

AutoImport-julienval

Done, I have just sent you the details through your blog contact page.

rfc2822

Didn’t get anything, maybe hit by the spam filter or so. Can you please send it to play@bitfire.at again

AutoImport-mherzberg

Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.

You’re right, I was too quick to assume that. The encoding issue seems to come from the different http request types (propfind vs report). Anyway, please try if you can reproduce the issue as follows:

Create a new address book in the owncloud web interface and import the following as an contacts.vcf file (the issue is only present with two or more contacts):

BEGIN:VCARD
VERSION:3.0
PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
FN:FOO
N:;FOO;;;
UID:20140623T46782.acdf@foo.bar.vcf
REV:2015-07-15T01:18:20+00:00
END:VCARD

BEGIN:VCARD
VERSION:3.0
PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
FN:BAR
N:;BAR;;;
UID:20140623T00123.086fed@foo.bar.vcf
REV:2015-07-15T01:18:21+00:00
END:VCARD

Import this address book into DavDroid. This creates the following log lines for me. Please note the different encodings of ‘@’ in the Processing multi-status element lines.

I/davdroid.DavSyncAdapter﹕ Performing sync for authority com.android.contacts
D/davdroid.DavSyncAdapter﹕ Creating new DavHttpClient
V/davdroid.URIUtils﹕ Normalized URI https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ -> https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Using preemptive authentication (not compatible with Digest auth)
I/davdroid.SyncManager﹕ Remotely removing 0 deleted resource(s) (if not changed)
I/davdroid.SyncManager﹕ Uploading 0 new resource(s) (if not existing)
I/davdroid.SyncManager﹕ Uploading 0 modified resource(s) (if not changed)
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
D/davdroid.SyncManager﹕ Last local CTag = null; current remote CTag = 1437215802
I/davdroid.SyncManager﹕ Fetching remote resource list
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf
I/davdroid.SyncManager﹕ Fetching 2 new remote resource(s)
I/davdroid.resource﹕ Multi-getting 2 remote resource(s)
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf
V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf assuming that it was an URI or path name
D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf
D/davdroid.SyncManager﹕ Adding 20140623T00123.086fed@foo.bar.vcf.vcf
D/davdroid.Collection﹕ Committing 2 operations ...
D/davdroid.Collection﹕ ... 0 row(s) affected
D/davdroid.SyncManager﹕ Adding 20140623T46782.acdf@foo.bar.vcf.vcf
D/davdroid.Collection﹕ Committing 2 operations ...
D/davdroid.Collection﹕ ... 0 row(s) affected
I/davdroid.SyncManager﹕ Fetching 0 updated remote resource(s)
I/davdroid.SyncManager﹕ Removing non-dirty resources that are not present remotely anymore

Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.

rfc2822

Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.

Yes, this seems to reproducable here. I’ll have a look.

rfc2822

Let’s have a look at what is happening:

First, DAVdroid 0.8.1 requests a list of all members from the server (OwnCloud 8.1.0):

http-outgoing-0 >> "PROPFIND /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]"
http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/</d:href><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/">1437219316</x3:getctag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response></d:multistatus>[\n]"

The pretty-printed response is:

<d:response>
    <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href>
    <d:propstat>
        <d:prop>
            <d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
<d:response>
    <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href>
    <d:propstat>
        <d:prop>
            <d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag>
        </d:prop>
        <d:status>HTTP/1.1 200 OK</d:status>
    </d:propstat>
    <d:propstat>
        <d:prop>
            <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/>
        </d:prop>
        <d:status>HTTP/1.1 404 Not Found</d:status>
    </d:propstat>
</d:response>
</d:multistatus>

So, there are two (new) members:

/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf (URL-encoded), i.e. member name 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf (URL-encoded), i.e. member name 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded)

DAVdroid requests those two resources using REPORT addressbook-multiget:

http-outgoing-0 >> "REPORT /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]"
…
http-outgoing-0 >> "<CD:addressbook-multiget xmlns:CD="urn:ietf:params:xml:ns:carddav" xmlns="DAV:">[\n]"
http-outgoing-0 >> "   <prop>[\n]"
http-outgoing-0 >> "      <CD:address-data content-type="text/vcard" version="4.0"/>[\n]"
http-outgoing-0 >> "      <getetag/>[\n]"
http-outgoing-0 >> "   </prop>[\n]"
http-outgoing-0 >> "   <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</href>[\n]"
http-outgoing-0 >> "   <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</href>[\n]"
http-outgoing-0 >> "</CD:addressbook-multiget>"

OwnCloud returns:

http-outgoing-0 << "HTTP/1.1 207 Multi-Status[\r][\n]"
…
http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD&#13;[\n]"
http-outgoing-0 << "VERSION:4.0&#13;[\n]"
http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN&#13;[\n]"
http-outgoing-0 << "FN:BAR&#13;[\n]"
http-outgoing-0 << "N:;BAR;;;&#13;[\n]"
http-outgoing-0 << "UID:20140623T00123.086fed@foo.bar.vcf&#13;[\n]"
http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00&#13;[\n]"
http-outgoing-0 << "END:VCARD&#13;[\n]"
http-outgoing-0 << "</card:address-data><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T4678"
http-outgoing-0 << "2.acdf@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD&#13;[\n]"
http-outgoing-0 << "VERSION:4.0&#13;[\n]"
http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN&#13;[\n]"
http-outgoing-0 << "FN:FOO&#13;[\n]"
http-outgoing-0 << "N:;FOO;;;&#13;[\n]"
http-outgoing-0 << "UID:20140623T46782.acdf@foo.bar.vcf&#13;[\n]"
http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00&#13;[\n]"
http-outgoing-0 << "END:VCARD&#13;[\n]"
http-outgoing-0 << "</card:address-data><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>[\n]"

So, if you have a look, you’ll see that DAVdroid requested the resources

/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf (URL-encoded), i.e. the member named 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded [%25 decodes to “%”]) and
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf (URL-encoded) i.e. the member named 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)

However, OwnCloud returned

/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf and
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf

This is clearly not what was requested and OwnCloud does one additional (and erroneous) level of encoding (as DAVdroid did earlier, by the way).

So, the sync process happens like this:

In response to PROPFIND, OwnCloud tells DAVdroid that there are two members:
- 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded) and
- 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)
DAVdroid requests these two (new) members from OwnCloud and adds all resources from the response. The response however
does not contain the requested members, but 20140623T00123.086fed@foo.bar.vcf.vcf and /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf.
These resources are added nevertheless (because DAVdroid expects them to be the requested ones, and it has to trust the server-given resource names to distinguish between the resources).
In the last step (“delete every local non-dirty contacts that are not on the server anymore”), DAVdroid deletes everything except 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded) and 20140623T46782.acdf%40foo.bar.vcf.vcf, so the recently fetched resources are deleted immediately (because their names are 20140623T00123.086fed@foo.bar.vcf.vcf and /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf).

Why does it work when there’s only one new record? In this case, DAVdroid uses a GET request instead of REPORT addressbook-multiget. The response is directly the VCard and the member name is used from the requested URI, so there’s no possibility for OwnCloud to return an erroneous file name.

Why did it work with previous DAVdroid versions? Well, DAVdroid did the same thing (double-decode URLs) earlier, too. This has been fixed to close #482. I guess it was a coincidence that this DAVdroid bug was just the the opposite of the corresponding OwnCloud bug so that it seemed to work (while it was bugged on both sides).

Do you agree?

AutoImport-mherzberg

Yes, I agree with you that the root cause seems to be inconsistent escaping of URIs by owncloud (URI escaped twice (%2540) in href-tags in PROPFIND responses vs no escaping in href-tags in REPORT responses). I don’t know how it’s handled elsewhere in owncloud, but I would expect no escaping at all in those href-tags. So this seems to be a bug in owncloud/the contact app after all. A workaround for this would be to unescape the href-tag twice in received PROPFIND responses, I guess? What do you think?

AutoImport-untitaker

@rfc2822 I think you’re hitting https://github.com/owncloud/contacts/issues/581. It’s going to be tricky/impossible to fix both #482 and simultaneously work around this. Even if ownCloud fixes this soon (it won’t), there will be other servers that do this.

What I’m currently doing is to first decode the filename and then encode it again except for the characters /@%:. This fixes only the most common instances of this problem where the encoded @ is causing trouble.

For the other rare cases I provide a repair command that fixes filenames to only use alphanumeric characters. I don’t think that’s within the scope of a sync app, but I’ve found it necessary as people came to me with the most perverse encoding problems. Now I can tell them to run the repair command and everything just works out.

rfc2822

@untitaker This could be the cause. The SabreDAV issue is closed, so couldn’t this be applied to OwnCloud too?

AutoImport-untitaker

The sabredav issue is closed because it’s a bug in ownCloud (somewhere in that conversation is a link to the exact line). I don’t know if e.g. Baikal has this issue.

rfc2822

Ok, so it seems the problem is caused by OwnCloud. Funny that it worked with older DAVdroid versions just because DAVdroid hat the opposite bug (it was quite a privacy killer to be honest, calling remote URLs just because of a file name).

[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+

Similar topics

Timeout problems - syncing contacts
DAVx⁵ • connectivity • • userx404

How to change account name?
Ideas, future talk, development • solved enhancement • • Erwin

Wrong time in calendar
ICSx⁵ • solved 3rd-party issue • • Rik

[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+

Similar topics

Timeout problems - syncing contacts DAVx⁵ • connectivity • • userx404

How to change account name? Ideas, future talk, development • solved enhancement • • Erwin

Wrong time in calendar ICSx⁵ • solved 3rd-party issue • • Rik

Timeout problems - syncing contacts
DAVx⁵ • connectivity • • userx404

How to change account name?
Ideas, future talk, development • solved enhancement • • Erwin

Wrong time in calendar
ICSx⁵ • solved 3rd-party issue • • Rik