@rfc2822 Oh, I see. Thanks for the explanation!
[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+
-
In my case the the sync of several addressbooks from Owncloud 8.1 with DavDroid 0.8.1 works fine, but the calendar sync doesn’t work. The account setup works and it says “synced” but nothing is displayed.
-
However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.
I would be happy to test your change if you can supply a precompiled apk
I can reproduce the issue on both of my phones -
I would be available for testing as well. Just upload a apk somewhere and give me the link.
-
Here is the link to my test build: davdroid-debug.apk
Please note that this is just a test build from me related to this issue and should not be used for daily usage. I do not take any responsibility for any harm caused by using this apk.This is the git diff from the code I used:
diff --git a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java index 65e2891..3479ac9 100644 --- a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java +++ b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java @@ -139,7 +139,7 @@ public class LocalAddressBook extends LocalCollection<Contact> { if (remoteResources.length != 0) { List<String> sqlFileNames = new LinkedList<>(); for (Resource res : remoteResources) - sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName())); + sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", "@")); where = entryColumnRemoteName() + " NOT IN (" + StringUtils.join(sqlFileNames, ",") + ")"; } else where = entryColumnRemoteName() + " IS NOT NULL";
-
However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.
I can’t reproduce that.
Log.i(TAG, "SQLite escaped: " + DatabaseUtils.sqlEscapeString("a@b.com"));
gives
a@b.com
here.However the sqlEscapeString method escapes the @ to %40
Can you back this up? I don’t think
@
should be escaped in SQL strings, andsqlEscapeString
doesn’t behave like that here. -
When trying to reproduce this bug, I ran into https://github.com/owncloud/contacts/issues/808. Is sharing enabled in your installation? If not, can you try to enable it?
Can you please
- delete or rename your OwnCloud log (
data/owncloud.log
) - do a DAVdroid sync
- access Contacts module in the Web Interface
- post the contents of your
owncloud.log
- delete or rename your OwnCloud log (
-
I can confirm that mherzberg’s apk fixes this for me. Thank you!
Previously, I was getting error messages in the notification menu every hour or so. Unfortunately, I forget what they said precisely.
I attempted uninstalling DAVdroid, then installing version 0.7.7. My phone would only import three contacts (out of 150). Oddly enough, I think these were the only contacts that I had added via the phone/DAVdroid. The others had been added/imported via OwnCloud’s web interface a while ago.
I also attempted installing DAVdroid 0.8.0 and 0.8.1. Neither fixed this issue. Only when installing mherzberg fix did it work. Thanks again!
-
Ahhh… after a while I’ve noticed mherzberg’s apk is buggy too. Every time DAVdroid syncs, it creates an additional copy of each contact, except the three contacts that were the only ones synced in the official versions of DAVdroid (as per my previous post). Hence, after three syncs, I have one copy of these original three in my Android address book as expected, but three copies of every other contact!
-
@rfc822 : sharing is enabled on my OC setup. I have followed your steps but my log file is empty after calling the contact page.
I have installed mherzberg’s apk on both of my phones and it seems to fix the issue. I have tried several syncs and I haven’t encountered the issue reported by protist.
-
Can someone provide a test account with contacts that show the behaviour with DAVdroid 0.8.1, so that I can reproduce the issue?
I guess the file name is stored URL-encoded, unrelated to SQL escaping, but I can’t test as it works here.
-
Done, I have just sent you the details through your blog contact page.
-
Didn’t get anything, maybe hit by the spam filter or so. Can you please send it to play@bitfire.at again
-
Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.
You’re right, I was too quick to assume that. The encoding issue seems to come from the different http request types (propfind vs report). Anyway, please try if you can reproduce the issue as follows:
- Create a new address book in the owncloud web interface and import the following as an contacts.vcf file (the issue is only present with two or more contacts):
BEGIN:VCARD VERSION:3.0 PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN FN:FOO N:;FOO;;; UID:20140623T46782.acdf@foo.bar.vcf REV:2015-07-15T01:18:20+00:00 END:VCARD BEGIN:VCARD VERSION:3.0 PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN FN:BAR N:;BAR;;; UID:20140623T00123.086fed@foo.bar.vcf REV:2015-07-15T01:18:21+00:00 END:VCARD
- Import this address book into DavDroid. This creates the following log lines for me. Please note the different encodings of ‘@’ in the
Processing multi-status element
lines.
I/davdroid.DavSyncAdapter﹕ Performing sync for authority com.android.contacts D/davdroid.DavSyncAdapter﹕ Creating new DavHttpClient V/davdroid.URIUtils﹕ Normalized URI https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ -> https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Using preemptive authentication (not compatible with Digest auth) I/davdroid.SyncManager﹕ Remotely removing 0 deleted resource(s) (if not changed) I/davdroid.SyncManager﹕ Uploading 0 new resource(s) (if not existing) I/davdroid.SyncManager﹕ Uploading 0 modified resource(s) (if not changed) V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ D/davdroid.SyncManager﹕ Last local CTag = null; current remote CTag = 1437215802 I/davdroid.SyncManager﹕ Fetching remote resource list V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf I/davdroid.SyncManager﹕ Fetching 2 new remote resource(s) I/davdroid.resource﹕ Multi-getting 2 remote resource(s) V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf assuming that it was an URI or path name D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf D/davdroid.SyncManager﹕ Adding 20140623T00123.086fed@foo.bar.vcf.vcf D/davdroid.Collection﹕ Committing 2 operations ... D/davdroid.Collection﹕ ... 0 row(s) affected D/davdroid.SyncManager﹕ Adding 20140623T46782.acdf@foo.bar.vcf.vcf D/davdroid.Collection﹕ Committing 2 operations ... D/davdroid.Collection﹕ ... 0 row(s) affected I/davdroid.SyncManager﹕ Fetching 0 updated remote resource(s) I/davdroid.SyncManager﹕ Removing non-dirty resources that are not present remotely anymore
Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.
-
Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.
Yes, this seems to reproducable here. I’ll have a look.
-
Let’s have a look at what is happening:
First, DAVdroid 0.8.1 requests a list of all members from the server (OwnCloud 8.1.0):
http-outgoing-0 >> "PROPFIND /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]" http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/</d:href><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/">1437219316</x3:getctag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response></d:multistatus>[\n]"
The pretty-printed response is:
<d:response> <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href> <d:propstat> <d:prop> <d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag> </d:prop> <d:status>HTTP/1.1 200 OK</d:status> </d:propstat> <d:propstat> <d:prop> <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/> </d:prop> <d:status>HTTP/1.1 404 Not Found</d:status> </d:propstat> </d:response> <d:response> <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href> <d:propstat> <d:prop> <d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag> </d:prop> <d:status>HTTP/1.1 200 OK</d:status> </d:propstat> <d:propstat> <d:prop> <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/> </d:prop> <d:status>HTTP/1.1 404 Not Found</d:status> </d:propstat> </d:response> </d:multistatus>
So, there are two (new) members:
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf
(URL-encoded), i.e. member name20140623T46782.acdf%40foo.bar.vcf.vcf
(unencoded)/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf
(URL-encoded), i.e. member name20140623T00123.086fed%40foo.bar.vcf.vcf
(unencoded)
DAVdroid requests those two resources using
REPORT addressbook-multiget
:http-outgoing-0 >> "REPORT /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]" … http-outgoing-0 >> "<CD:addressbook-multiget xmlns:CD="urn:ietf:params:xml:ns:carddav" xmlns="DAV:">[\n]" http-outgoing-0 >> " <prop>[\n]" http-outgoing-0 >> " <CD:address-data content-type="text/vcard" version="4.0"/>[\n]" http-outgoing-0 >> " <getetag/>[\n]" http-outgoing-0 >> " </prop>[\n]" http-outgoing-0 >> " <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</href>[\n]" http-outgoing-0 >> " <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</href>[\n]" http-outgoing-0 >> "</CD:addressbook-multiget>"
OwnCloud returns:
http-outgoing-0 << "HTTP/1.1 207 Multi-Status[\r][\n]" … http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD [\n]" http-outgoing-0 << "VERSION:4.0 [\n]" http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN [\n]" http-outgoing-0 << "FN:BAR [\n]" http-outgoing-0 << "N:;BAR;;; [\n]" http-outgoing-0 << "UID:20140623T00123.086fed@foo.bar.vcf [\n]" http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00 [\n]" http-outgoing-0 << "END:VCARD [\n]" http-outgoing-0 << "</card:address-data><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T4678" http-outgoing-0 << "2.acdf@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD [\n]" http-outgoing-0 << "VERSION:4.0 [\n]" http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN [\n]" http-outgoing-0 << "FN:FOO [\n]" http-outgoing-0 << "N:;FOO;;; [\n]" http-outgoing-0 << "UID:20140623T46782.acdf@foo.bar.vcf [\n]" http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00 [\n]" http-outgoing-0 << "END:VCARD [\n]" http-outgoing-0 << "</card:address-data><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>[\n]"
So, if you have a look, you’ll see that DAVdroid requested the resources
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf
(URL-encoded), i.e. the member named20140623T00123.086fed%40foo.bar.vcf.vcf
(unencoded [%25 decodes to “%”]) and/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf
(URL-encoded) i.e. the member named20140623T46782.acdf%40foo.bar.vcf.vcf
(unencoded)
However, OwnCloud returned
/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf
and/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf
This is clearly not what was requested and OwnCloud does one additional (and erroneous) level of encoding (as DAVdroid did earlier, by the way).
So, the sync process happens like this:
- In response to PROPFIND, OwnCloud tells DAVdroid that there are two members:
20140623T00123.086fed%40foo.bar.vcf.vcf
(unencoded) and20140623T46782.acdf%40foo.bar.vcf.vcf
(unencoded)
- DAVdroid requests these two (new) members from OwnCloud and adds all resources from the response. The response however
does not contain the requested members, but20140623T00123.086fed@foo.bar.vcf.vcf
and/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf
.
These resources are added nevertheless (because DAVdroid expects them to be the requested ones, and it has to trust the server-given resource names to distinguish between the resources). - In the last step (“delete every local non-dirty contacts that are not on the server anymore”), DAVdroid deletes everything except
20140623T00123.086fed%40foo.bar.vcf.vcf
(unencoded) and20140623T46782.acdf%40foo.bar.vcf.vcf
, so the recently fetched resources are deleted immediately (because their names are20140623T00123.086fed@foo.bar.vcf.vcf
and/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf
).
Why does it work when there’s only one new record? In this case, DAVdroid uses a
GET
request instead ofREPORT addressbook-multiget
. The response is directly the VCard and the member name is used from the requested URI, so there’s no possibility for OwnCloud to return an erroneous file name.Why did it work with previous DAVdroid versions? Well, DAVdroid did the same thing (double-decode URLs) earlier, too. This has been fixed to close #482. I guess it was a coincidence that this DAVdroid bug was just the the opposite of the corresponding OwnCloud bug so that it seemed to work (while it was bugged on both sides).
Do you agree?
-
Yes, I agree with you that the root cause seems to be inconsistent escaping of URIs by owncloud (URI escaped twice (
%2540
) inhref
-tags in PROPFIND responses vs no escaping inhref
-tags in REPORT responses). I don’t know how it’s handled elsewhere in owncloud, but I would expect no escaping at all in thosehref
-tags. So this seems to be a bug in owncloud/the contact app after all. A workaround for this would be to unescape thehref
-tag twice in received PROPFIND responses, I guess? What do you think? -
@rfc2822 I think you’re hitting https://github.com/owncloud/contacts/issues/581. It’s going to be tricky/impossible to fix both #482 and simultaneously work around this. Even if ownCloud fixes this soon (it won’t), there will be other servers that do this.
What I’m currently doing is to first decode the filename and then encode it again except for the characters
/@%:
. This fixes only the most common instances of this problem where the encoded@
is causing trouble.For the other rare cases I provide a
repair
command that fixes filenames to only use alphanumeric characters. I don’t think that’s within the scope of a sync app, but I’ve found it necessary as people came to me with the most perverse encoding problems. Now I can tell them to run therepair
command and everything just works out. -
@untitaker This could be the cause. The SabreDAV issue is closed, so couldn’t this be applied to OwnCloud too?
-
The sabredav issue is closed because it’s a bug in ownCloud (somewhere in that conversation is a link to the exact line). I don’t know if e.g. Baikal has this issue.
-
Ok, so it seems the problem is caused by OwnCloud. Funny that it worked with older DAVdroid versions just because DAVdroid hat the opposite bug (it was quite a privacy killer to be honest, calling remote URLs just because of a file name).