[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+



  • I have the same problem with OC 8.1 and DAVdroid 0.8.1 since upgrading owncloud. I have tried various times with various options (and also with different devices).
    If I sync my contacts with “CardDAV-Sync free” from the play store every contact gets transferred (just without the groups). So maybe DAVdroid needs some changes too?


  • developer

    Maybe this is related to the cookie support introduced with DAVdroid/0.8.1?

    Unfortunately, I couldn’t test with OwnCloud/8.1 because after the upgrade from 8.0, the OwnCloud Contact/Calendar apps were removed and I can’t install them anymore.



  • I have also seen a lot of Sync Issues in the notification drawer. I had not yet connected this to the OC 8.1 update, but now it looks like this is the reason. I’m to lazy to downgrade and upgrade again.

    I achieved the cleanest sync by deleting both databases (phone + oc) and then imported the multi-vcf exported from OC on the phone into the newly created empty Davdroid account.

    Took some time until the contacts were uploaded, but then it was at least almost okay. (some addresses seem to have been messed…).

    Will try to reduce my contact editing until either side fixes this…



  • After some debugging I suspect the bug lies within this line in method deleteAllExceptRemoteNames in class LocalCollection:319:
    sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()));
    When I checked my contact database on my device before deleteAllExceptRemoteNames gets executed, all my owncloud contacts were present and their source ids contained ‘@foo.bar.de.vcf’. However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", "@"));, and my contacts seemed to stay on my device.



  • Hi, I get the same issue with both 0.8.0 and 0.8.1 to ownCloud 8.1.

    I did notice that the old ownCloud contacts didnt have the vcard 4.0 option ticked.

    nginx/1.8.0
    PHP 5.4.41-0+deb7u1 (fpm-fcgi) with XCache v2.0.0
    ownCloud 8.1.0 with ‘memcache.local’ => ‘\OC\Memcache\XCache’
    DavDroid/0.8.1 (0.8.0 also) on HTC M8 android 5.0.1



  • However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.

    Then this is DAVdroid bug, after all.



  • In my case the the sync of several addressbooks from Owncloud 8.1 with DavDroid 0.8.1 works fine, but the calendar sync doesn’t work. The account setup works and it says “synced” but nothing is displayed.



  • However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.

    I would be happy to test your change if you can supply a precompiled apk
    I can reproduce the issue on both of my phones



  • I would be available for testing as well. Just upload a apk somewhere and give me the link.



  • Here is the link to my test build: davdroid-debug.apk
    Please note that this is just a test build from me related to this issue and should not be used for daily usage. I do not take any responsibility for any harm caused by using this apk.

    This is the git diff from the code I used:

    diff --git a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
    index 65e2891..3479ac9 100644
    --- a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
    +++ b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
    @@ -139,7 +139,7 @@ public class LocalAddressBook extends LocalCollection<Contact> {
                    if (remoteResources.length != 0) {
                            List<String> sqlFileNames = new LinkedList<>();
                            for (Resource res : remoteResources)
    -                               sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()));
    +                               sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", "@"));
                            where = entryColumnRemoteName() + " NOT IN (" + StringUtils.join(sqlFileNames, ",") + ")";
                    } else
                            where = entryColumnRemoteName() + " IS NOT NULL";
    

  • developer

    However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.

    I can’t reproduce that.

    Log.i(TAG, "SQLite escaped: " + DatabaseUtils.sqlEscapeString("a@b.com"));
    

    gives a@b.com here.

    However the sqlEscapeString method escapes the @ to %40

    Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.


  • developer

    When trying to reproduce this bug, I ran into https://github.com/owncloud/contacts/issues/808. Is sharing enabled in your installation? If not, can you try to enable it?

    Can you please

    1. delete or rename your OwnCloud log (data/owncloud.log)
    2. do a DAVdroid sync
    3. access Contacts module in the Web Interface
    4. post the contents of your owncloud.log


  • I can confirm that mherzberg’s apk fixes this for me. Thank you!

    Previously, I was getting error messages in the notification menu every hour or so. Unfortunately, I forget what they said precisely.

    I attempted uninstalling DAVdroid, then installing version 0.7.7. My phone would only import three contacts (out of 150). Oddly enough, I think these were the only contacts that I had added via the phone/DAVdroid. The others had been added/imported via OwnCloud’s web interface a while ago.

    I also attempted installing DAVdroid 0.8.0 and 0.8.1. Neither fixed this issue. Only when installing mherzberg fix did it work. Thanks again!



  • Ahhh… after a while I’ve noticed mherzberg’s apk is buggy too. Every time DAVdroid syncs, it creates an additional copy of each contact, except the three contacts that were the only ones synced in the official versions of DAVdroid (as per my previous post). Hence, after three syncs, I have one copy of these original three in my Android address book as expected, but three copies of every other contact!



  • @rfc822 : sharing is enabled on my OC setup. I have followed your steps but my log file is empty after calling the contact page.

    I have installed mherzberg’s apk on both of my phones and it seems to fix the issue. I have tried several syncs and I haven’t encountered the issue reported by protist.


  • developer

    Can someone provide a test account with contacts that show the behaviour with DAVdroid 0.8.1, so that I can reproduce the issue?

    I guess the file name is stored URL-encoded, unrelated to SQL escaping, but I can’t test as it works here.



  • Done, I have just sent you the details through your blog contact page.


  • developer

    Didn’t get anything, maybe hit by the spam filter or so. Can you please send it to play@bitfire.at again



  • Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.

    You’re right, I was too quick to assume that. The encoding issue seems to come from the different http request types (propfind vs report). Anyway, please try if you can reproduce the issue as follows:

    • Create a new address book in the owncloud web interface and import the following as an contacts.vcf file (the issue is only present with two or more contacts):
    BEGIN:VCARD
    VERSION:3.0
    PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
    FN:FOO
    N:;FOO;;;
    UID:20140623T46782.acdf@foo.bar.vcf
    REV:2015-07-15T01:18:20+00:00
    END:VCARD
    
    BEGIN:VCARD
    VERSION:3.0
    PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
    FN:BAR
    N:;BAR;;;
    UID:20140623T00123.086fed@foo.bar.vcf
    REV:2015-07-15T01:18:21+00:00
    END:VCARD
    
    • Import this address book into DavDroid. This creates the following log lines for me. Please note the different encodings of ‘@’ in the Processing multi-status element lines.
    I/davdroid.DavSyncAdapter﹕ Performing sync for authority com.android.contacts
    D/davdroid.DavSyncAdapter﹕ Creating new DavHttpClient
    V/davdroid.URIUtils﹕ Normalized URI https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ -> https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Using preemptive authentication (not compatible with Digest auth)
    I/davdroid.SyncManager﹕ Remotely removing 0 deleted resource(s) (if not changed)
    I/davdroid.SyncManager﹕ Uploading 0 new resource(s) (if not existing)
    I/davdroid.SyncManager﹕ Uploading 0 modified resource(s) (if not changed)
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
    D/davdroid.SyncManager﹕ Last local CTag = null; current remote CTag = 1437215802
    I/davdroid.SyncManager﹕ Fetching remote resource list
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf
    I/davdroid.SyncManager﹕ Fetching 2 new remote resource(s)
    I/davdroid.resource﹕ Multi-getting 2 remote resource(s)
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf
    D/davdroid.SyncManager﹕ Adding 20140623T00123.086fed@foo.bar.vcf.vcf
    D/davdroid.Collection﹕ Committing 2 operations ...
    D/davdroid.Collection﹕ ... 0 row(s) affected
    D/davdroid.SyncManager﹕ Adding 20140623T46782.acdf@foo.bar.vcf.vcf
    D/davdroid.Collection﹕ Committing 2 operations ...
    D/davdroid.Collection﹕ ... 0 row(s) affected
    I/davdroid.SyncManager﹕ Fetching 0 updated remote resource(s)
    I/davdroid.SyncManager﹕ Removing non-dirty resources that are not present remotely anymore
    

    Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.


  • developer

    Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.

    Yes, this seems to reproducable here. I’ll have a look.