[OwnCloud] Losing contacts after sync with DAVdroid 0.8.1+



  • Here is the link to my test build: davdroid-debug.apk
    Please note that this is just a test build from me related to this issue and should not be used for daily usage. I do not take any responsibility for any harm caused by using this apk.

    This is the git diff from the code I used:

    diff --git a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
    index 65e2891..3479ac9 100644
    --- a/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
    +++ b/app/src/main/java/at/bitfire/davdroid/resource/LocalAddressBook.java
    @@ -139,7 +139,7 @@ public class LocalAddressBook extends LocalCollection<Contact> {
                    if (remoteResources.length != 0) {
                            List<String> sqlFileNames = new LinkedList<>();
                            for (Resource res : remoteResources)
    -                               sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()));
    +                               sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", "@"));
                            where = entryColumnRemoteName() + " NOT IN (" + StringUtils.join(sqlFileNames, ",") + ")";
                    } else
                            where = entryColumnRemoteName() + " IS NOT NULL";
    

  • developer

    However the sqlEscapeString method escapes the @ to %40, so (almost) no contacts matches the sql query anymore and get deleted. I replaced the line with sqlFileNames.add(DatabaseUtils.sqlEscapeString(res.getName()).replace("%40", “@”));, and my contacts seemed to stay on my device.

    I can’t reproduce that.

    Log.i(TAG, "SQLite escaped: " + DatabaseUtils.sqlEscapeString("a@b.com"));
    

    gives a@b.com here.

    However the sqlEscapeString method escapes the @ to %40

    Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.


  • developer

    When trying to reproduce this bug, I ran into https://github.com/owncloud/contacts/issues/808. Is sharing enabled in your installation? If not, can you try to enable it?

    Can you please

    1. delete or rename your OwnCloud log (data/owncloud.log)
    2. do a DAVdroid sync
    3. access Contacts module in the Web Interface
    4. post the contents of your owncloud.log


  • I can confirm that mherzberg’s apk fixes this for me. Thank you!

    Previously, I was getting error messages in the notification menu every hour or so. Unfortunately, I forget what they said precisely.

    I attempted uninstalling DAVdroid, then installing version 0.7.7. My phone would only import three contacts (out of 150). Oddly enough, I think these were the only contacts that I had added via the phone/DAVdroid. The others had been added/imported via OwnCloud’s web interface a while ago.

    I also attempted installing DAVdroid 0.8.0 and 0.8.1. Neither fixed this issue. Only when installing mherzberg fix did it work. Thanks again!



  • Ahhh… after a while I’ve noticed mherzberg’s apk is buggy too. Every time DAVdroid syncs, it creates an additional copy of each contact, except the three contacts that were the only ones synced in the official versions of DAVdroid (as per my previous post). Hence, after three syncs, I have one copy of these original three in my Android address book as expected, but three copies of every other contact!



  • @rfc822 : sharing is enabled on my OC setup. I have followed your steps but my log file is empty after calling the contact page.

    I have installed mherzberg’s apk on both of my phones and it seems to fix the issue. I have tried several syncs and I haven’t encountered the issue reported by protist.


  • developer

    Can someone provide a test account with contacts that show the behaviour with DAVdroid 0.8.1, so that I can reproduce the issue?

    I guess the file name is stored URL-encoded, unrelated to SQL escaping, but I can’t test as it works here.



  • Done, I have just sent you the details through your blog contact page.


  • developer

    Didn’t get anything, maybe hit by the spam filter or so. Can you please send it to play@bitfire.at again



  • Can you back this up? I don’t think @ should be escaped in SQL strings, and sqlEscapeString doesn’t behave like that here.

    You’re right, I was too quick to assume that. The encoding issue seems to come from the different http request types (propfind vs report). Anyway, please try if you can reproduce the issue as follows:

    • Create a new address book in the owncloud web interface and import the following as an contacts.vcf file (the issue is only present with two or more contacts):
    BEGIN:VCARD
    VERSION:3.0
    PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
    FN:FOO
    N:;FOO;;;
    UID:20140623T46782.acdf@foo.bar.vcf
    REV:2015-07-15T01:18:20+00:00
    END:VCARD
    
    BEGIN:VCARD
    VERSION:3.0
    PRODID:-//ownCloud//NONSGML Contacts 0.3.0.18//EN
    FN:BAR
    N:;BAR;;;
    UID:20140623T00123.086fed@foo.bar.vcf
    REV:2015-07-15T01:18:21+00:00
    END:VCARD
    
    • Import this address book into DavDroid. This creates the following log lines for me. Please note the different encodings of ‘@’ in the Processing multi-status element lines.
    I/davdroid.DavSyncAdapter﹕ Performing sync for authority com.android.contacts
    D/davdroid.DavSyncAdapter﹕ Creating new DavHttpClient
    V/davdroid.URIUtils﹕ Normalized URI https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ -> https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Using preemptive authentication (not compatible with Digest auth)
    I/davdroid.SyncManager﹕ Remotely removing 0 deleted resource(s) (if not changed)
    I/davdroid.SyncManager﹕ Uploading 0 new resource(s) (if not existing)
    I/davdroid.SyncManager﹕ Uploading 0 modified resource(s) (if not changed)
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
    D/davdroid.SyncManager﹕ Last local CTag = null; current remote CTag = 1437215802
    I/davdroid.SyncManager﹕ Fetching remote resource list
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/ -> /remote.php/carddav/addressbooks/test/contacts/ assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf%2540foo.bar.vcf.vcf
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed%2540foo.bar.vcf.vcf
    I/davdroid.SyncManager﹕ Fetching 2 new remote resource(s)
    I/davdroid.resource﹕ Multi-getting 2 remote resource(s)
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T00123.086fed@foo.bar.vcf.vcf
    V/davdroid.URIUtils﹕ Normalized URI /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf -> /remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf assuming that it was an URI or path name
    D/davdroid.WebDavResource﹕ Processing multi-status element: https://foo.bar.de/remote.php/carddav/addressbooks/test/contacts/20140623T46782.acdf@foo.bar.vcf.vcf
    D/davdroid.SyncManager﹕ Adding 20140623T00123.086fed@foo.bar.vcf.vcf
    D/davdroid.Collection﹕ Committing 2 operations ...
    D/davdroid.Collection﹕ ... 0 row(s) affected
    D/davdroid.SyncManager﹕ Adding 20140623T46782.acdf@foo.bar.vcf.vcf
    D/davdroid.Collection﹕ Committing 2 operations ...
    D/davdroid.Collection﹕ ... 0 row(s) affected
    I/davdroid.SyncManager﹕ Fetching 0 updated remote resource(s)
    I/davdroid.SyncManager﹕ Removing non-dirty resources that are not present remotely anymore
    

    Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.


  • developer

    Both contacts get created and immediately deleted afterwards on my phone. I hope this helps in nailing down the issue.

    Yes, this seems to reproducable here. I’ll have a look.


  • developer

    Let’s have a look at what is happening:

    First, DAVdroid 0.8.1 requests a list of all members from the server (OwnCloud 8.1.0):

    http-outgoing-0 >> "PROPFIND /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]"
    http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/</d:href><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/">1437219316</x3:getctag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href><d:propstat><d:prop><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><x3:getctag xmlns:x3="http://calendarserver.org/ns/"/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response></d:multistatus>[\n]"
    

    The pretty-printed response is:

    <d:response>
        <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</d:href>
        <d:propstat>
            <d:prop>
                <d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
        <d:propstat>
            <d:prop>
                <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/>
            </d:prop>
            <d:status>HTTP/1.1 404 Not Found</d:status>
        </d:propstat>
    </d:response>
    <d:response>
        <d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</d:href>
        <d:propstat>
            <d:prop>
                <d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
        <d:propstat>
            <d:prop>
                <x3:getctag xmlns:x3="http://calendarserver.org/ns/"/>
            </d:prop>
            <d:status>HTTP/1.1 404 Not Found</d:status>
        </d:propstat>
    </d:response>
    </d:multistatus>
    

    So, there are two (new) members:

    • /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf (URL-encoded), i.e. member name 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)
    • /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf (URL-encoded), i.e. member name 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded)

    DAVdroid requests those two resources using REPORT addressbook-multiget:

    http-outgoing-0 >> "REPORT /owncloud/remote.php/carddav/addressbooks/test/kontakte/ HTTP/1.1[\r][\n]"
    …
    http-outgoing-0 >> "<CD:addressbook-multiget xmlns:CD="urn:ietf:params:xml:ns:carddav" xmlns="DAV:">[\n]"
    http-outgoing-0 >> "   <prop>[\n]"
    http-outgoing-0 >> "      <CD:address-data content-type="text/vcard" version="4.0"/>[\n]"
    http-outgoing-0 >> "      <getetag/>[\n]"
    http-outgoing-0 >> "   </prop>[\n]"
    http-outgoing-0 >> "   <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf</href>[\n]"
    http-outgoing-0 >> "   <href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf</href>[\n]"
    http-outgoing-0 >> "</CD:addressbook-multiget>"
    

    OwnCloud returns:

    http-outgoing-0 << "HTTP/1.1 207 Multi-Status[\r][\n]"
    …
    http-outgoing-0 << "<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:card="urn:ietf:params:xml:ns:carddav"><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD&#13;[\n]"
    http-outgoing-0 << "VERSION:4.0&#13;[\n]"
    http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN&#13;[\n]"
    http-outgoing-0 << "FN:BAR&#13;[\n]"
    http-outgoing-0 << "N:;BAR;;;&#13;[\n]"
    http-outgoing-0 << "UID:20140623T00123.086fed@foo.bar.vcf&#13;[\n]"
    http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00&#13;[\n]"
    http-outgoing-0 << "END:VCARD&#13;[\n]"
    http-outgoing-0 << "</card:address-data><d:getetag>"f144eaeea375650281aeeff2929788bd"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T4678"
    http-outgoing-0 << "2.acdf@foo.bar.vcf.vcf</d:href><d:propstat><d:prop><card:address-data>BEGIN:VCARD&#13;[\n]"
    http-outgoing-0 << "VERSION:4.0&#13;[\n]"
    http-outgoing-0 << "PRODID:-//Sabre//Sabre VObject 3.4.2//EN&#13;[\n]"
    http-outgoing-0 << "FN:FOO&#13;[\n]"
    http-outgoing-0 << "N:;FOO;;;&#13;[\n]"
    http-outgoing-0 << "UID:20140623T46782.acdf@foo.bar.vcf&#13;[\n]"
    http-outgoing-0 << "REV:2015-07-18T11:35:15+00:00&#13;[\n]"
    http-outgoing-0 << "END:VCARD&#13;[\n]"
    http-outgoing-0 << "</card:address-data><d:getetag>"28c05d52626ce5084a1f991b438ac41d"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>[\n]"
    

    So, if you have a look, you’ll see that DAVdroid requested the resources

    • /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed%2540foo.bar.vcf.vcf (URL-encoded), i.e. the member named 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded [%25 decodes to “%”]) and
    • /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf%2540foo.bar.vcf.vcf (URL-encoded) i.e. the member named 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)

    However, OwnCloud returned

    • /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T00123.086fed@foo.bar.vcf.vcf and
    • /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf

    This is clearly not what was requested and OwnCloud does one additional (and erroneous) level of encoding (as DAVdroid did earlier, by the way).

    So, the sync process happens like this:

    1. In response to PROPFIND, OwnCloud tells DAVdroid that there are two members:
      • 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded) and
      • 20140623T46782.acdf%40foo.bar.vcf.vcf (unencoded)
    2. DAVdroid requests these two (new) members from OwnCloud and adds all resources from the response. The response however
      does not contain the requested members, but 20140623T00123.086fed@foo.bar.vcf.vcf and /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf.
      These resources are added nevertheless (because DAVdroid expects them to be the requested ones, and it has to trust the server-given resource names to distinguish between the resources).
    3. In the last step (“delete every local non-dirty contacts that are not on the server anymore”), DAVdroid deletes everything except 20140623T00123.086fed%40foo.bar.vcf.vcf (unencoded) and 20140623T46782.acdf%40foo.bar.vcf.vcf, so the recently fetched resources are deleted immediately (because their names are 20140623T00123.086fed@foo.bar.vcf.vcf and /owncloud/remote.php/carddav/addressbooks/test/kontakte/20140623T46782.acdf@foo.bar.vcf.vcf).

    Why does it work when there’s only one new record? In this case, DAVdroid uses a GET request instead of REPORT addressbook-multiget. The response is directly the VCard and the member name is used from the requested URI, so there’s no possibility for OwnCloud to return an erroneous file name.

    Why did it work with previous DAVdroid versions? Well, DAVdroid did the same thing (double-decode URLs) earlier, too. This has been fixed to close #482. I guess it was a coincidence that this DAVdroid bug was just the the opposite of the corresponding OwnCloud bug so that it seemed to work (while it was bugged on both sides).

    Do you agree?



  • Yes, I agree with you that the root cause seems to be inconsistent escaping of URIs by owncloud (URI escaped twice (%2540) in href-tags in PROPFIND responses vs no escaping in href-tags in REPORT responses). I don’t know how it’s handled elsewhere in owncloud, but I would expect no escaping at all in those href-tags. So this seems to be a bug in owncloud/the contact app after all. A workaround for this would be to unescape the href-tag twice in received PROPFIND responses, I guess? What do you think?



  • @rfc2822 I think you’re hitting https://github.com/owncloud/contacts/issues/581. It’s going to be tricky/impossible to fix both #482 and simultaneously work around this. Even if ownCloud fixes this soon (it won’t), there will be other servers that do this.

    What I’m currently doing is to first decode the filename and then encode it again except for the characters /@%:. This fixes only the most common instances of this problem where the encoded @ is causing trouble.

    For the other rare cases I provide a repair command that fixes filenames to only use alphanumeric characters. I don’t think that’s within the scope of a sync app, but I’ve found it necessary as people came to me with the most perverse encoding problems. Now I can tell them to run the repair command and everything just works out.


  • developer

    @untitaker This could be the cause. The SabreDAV issue is closed, so couldn’t this be applied to OwnCloud too?



  • The sabredav issue is closed because it’s a bug in ownCloud (somewhere in that conversation is a link to the exact line). I don’t know if e.g. Baikal has this issue.


  • developer

    Ok, so it seems the problem is caused by OwnCloud. Funny that it worked with older DAVdroid versions just because DAVdroid hat the opposite bug (it was quite a privacy killer to be honest, calling remote URLs just because of a file name).



  • BTW what’s the behavior if a PROPFIND request returns a 404 for some hrefs? Are these hrefs skipped? You might be able to turn such silent failures into hard crashes by raising an error for those (an improvement IMO, but I understand if that’s not the UI you want)


  • developer

    BTW what’s the behavior if a PROPFIND request returns a 404 for some hrefs? Are these hrefs skipped?

    For every response in a 207-multi-response that has <status>, the same checkResponse is called as it would be called for a single GET. For those which have <propstat> (i.e. the resource exists), only 10x and 20x are evaluated.

    checkResponse will accept 10x and 20x silently and throw an exception for other status codes.

    But it’s off-topic 😉



  • It’s somewhat related because I wonder how DavDroid would delete contacts if a PROPFIND request doesn’t return the requested hrefs.

    On 18 July 2015 16:54:13 CEST, rfc2822 notifications@github.com wrote:

    BTW what’s the behavior if a PROPFIND request returns a 404 for some
    hrefs? Are these hrefs skipped?

    For every response in a 207-multi-response, the same checkResponse
    is
    called

    as it would be called for a single GET.

    checkResponse
    will accept 10x and 20x silently and throw an exception for other
    status codes.

    But it’s off-topic 😉


    Reply to this email directly or view it on GitHub:
    https://github.com/bitfireAT/davdroid/issues/578#issuecomment-122550656


    Sent from my phone. Please excuse my brevity.



Similar topics