Hello and thank you for this useful software. I have been able to trigger the following syncing problem. While some of the issues encountered are not DAVdroid’s some are and I hope this report helps tackle the problems.

Setup:

• Server: Baïkal 0.2.7
• Client 1: DAVdroid 0.6.11 from F-Droid on Android 4.0.4
• Client 2: Thunderbird 31.4.0 with Lightning 3.3.3 and Inverse SOGo Connecotr 31.0.1 on Xubuntu 14.04

To replicate the issue:
(1) Set up a fresh address book in Baïkal
(2) Connect to the address book from both Thunderbird and DAVdroid
(3) Create the first entry in the addressbook with Thunderbird. An SQL query on the server shows the following entry:

BEGIN:VCARD
VERSION:3.0
PRODID:-//Inverse inc.//SOGo Connector 1.0//EN
UID:C67A8200-8510-0001-4563-ED221C5DD950
N:a1;a1
FN:a1 a1
X-MOZILLA-HTML:FALSE
EMAIL;TYPE=work:a1@a1.a1
EMAIL;TYPE=home:a1@a1.a2
END:VCARD

(4) Edit the entry in the Android phone book. Add a third email address with custom label “customlabel”
(5) Trigger synchronisation in DAVdroid. Feature_Request: It would be nice to have an option to trigger sync on completion of an edit, like SOGo Connector does. Resulting server entry:

BEGIN:VCARD
VERSION:3.0
X-MOZILLA-HTML:FALSE
UID:C67A8200-8510-0001-4563-ED221C5DD950
FN:a1 a1
N:a1;a1;;;
EMAIL;TYPE=x-customlabel:aa@aq.qa
EMAIL;TYPE=home:a1@a1.a2
EMAIL;TYPE=work:a1@a1.a1
PRODID:DAVdroid/0.6.11 (ez-vcard/0.9.6)
REV:20150131T230658Z
END:VCARD

(6) Synced and edited in Thunderbird, adding field “Notes”. SOGo does not update the entry on the server, not even when manually syncing the address book. Bug will be filed against SOGo Connector.
(7) Added “Work Phone” in Thunderbird. This time SOGo updates properly, but it deletes / drops the third email that was added on Adroid. Bug will be filed against SOGo Connector. Resulting server entry:

BEGIN:VCARD
VERSION:3.0
PRODID:-//Inverse inc.//SOGo Connector 1.0//EN
UID:C67A8200-8320-0001-DE97-D15812104910.vcf
N:a1;a1
FN:a1 a1
TEL;TYPE=work:999-999-9999
X-MOZILLA-HTML:FALSE
EMAIL;TYPE=work:a1@a1.a1
EMAIL;TYPE=home:a1@a1.a2
NOTE:tb notes
END:VCARD

(8) Sync with DAVdroid, then edit on Android, add phonetic name, with content “blah”. Resulting server entry:

BEGIN:VCARD
VERSION:3.0
X-MOZILLA-HTML:FALSE
X-PHONETIC-LAST-NAME:blah
UID:C67A8200-8320-0001-DE97-D15812104910.vcf
FN:a1 a1
N:a1;a1;;;
TEL;TYPE=work:999-999-9999
EMAIL;TYPE=home:a1@a1.a2
EMAIL;TYPE=work:a1@a1.a1
NOTE:tb notes
PRODID:DAVdroid/0.6.11 (ez-vcard/0.9.6)
REV:20150131T231427Z
END:VCARD

(9) After syncing, added in Thunderbird a private website “www.example.com”. SOGo Connector syncs immediately after the edit to the server and the resulting server entry is now:

BEGIN:VCARD
VERSION:3.0
PRODID:-//Inverse inc.//SOGo Connector 1.0//EN
UID:C67A8200-8320-0001-DE97-D15812104910.vcf
N:a1;a1
FN:a1 a1
TEL;TYPE=work:999-999-9999
X-MOZILLA-HTML:FALSE
EMAIL;TYPE=work:a1@a1.a1
EMAIL;TYPE=home:a1@a1.a2
URL;TYPE=home:www.example.com
NOTE:tb notes
X-PHONETIC-LAST-NAME:blah
REV:20150131T231427Z
END:VCARD

(10) Sync DAVdroid. bug: the entry disappears from the Android phone book.
(11) Remove the private website entry in Thunderbird. SOGo syncs. The resulting server entry is now:

BEGIN:VCARD
VERSION:3.0
PRODID:-//Inverse inc.//SOGo Connector 1.0//EN
UID:C67A8200-8320-0001-DE97-D15812104910.vcf
N:a1;a1
FN:a1 a1
TEL;TYPE=work:999-999-9999
X-MOZILLA-HTML:FALSE
EMAIL;TYPE=work:a1@a1.a1
EMAIL;TYPE=home:a1@a1.a2
NOTE:tb notes
X-PHONETIC-LAST-NAME:blah
REV:20150131T231427Z
END:VCARD

Other than the fields order, and the PRODID, this is the same as what DAVdroid did sync previously, but now the contact no longer appears on Android, no matter what change I do to it in Thunderbird.
(12) Delete the contact in Thunderbird. Verify on the server that the SQL entry is gone.
(13) Enter again the contact in Thunderbird, including the note and the private website. Resulting entry on server:

BEGIN:VCARD
VERSION:3.0
PRODID:-//Inverse inc.//SOGo Connector 1.0//EN
UID:C67A844A-3890-0001-2D2E-10A613B04CA0
N:a1;a1
FN:a1 a1
TEL;TYPE=work:99-999-9999
X-MOZILLA-HTML:FALSE
EMAIL;TYPE=work:a1@a1.a1
EMAIL;TYPE=home::a1@a1.a2
URL;TYPE=home:www.example.com
NOTE:tb notes
END:VCARD

(14) Synchronize on DAVdroid. Now the entry is there in the Android phone book, with the website’s URL.

I hope this report helps. I will now file a report on Inverse’s bug tracker for SOGo Connector. To summarize in relation to DAVdroid:

• bug: disappearing addressbook entry, even if it is present in Baïkal’s database and does not seem to have anything wrong.
• feature request: automatic sync-after-edit option

Let me know if I can assist with debugging.

Submitted report to SOGo: http://www.sogo.nu/bugs/view.php?id=3086

Thanks for the extensive report.

bug: disappearing addressbook entry, even if it is present in Baïkal’s database and does not seem to have anything wrong.

Would it be possible to get device logs? That would help to see why the contact disappears in DAVdroid (I guess because it’s not parseable).

Is this issue still present with the latest DAVdroid version?

TL;DR

I suggest a two-pronged solution:

1. Use custom lines like X-EVOLUTION-ABCD instead of TEL;TYPE=ABCD. Store all data redundantly in the same VCARD: once to custom X-DAVDROID-… lines and once to “standard” lines.

2. Map custom X-DAVDROID-… lines to the protocol’s “standard” lines and try to keep them in sync when the standard lines are changed by a third-party client (the “3P client”).

What Works for Me Now

Sorry I dropped the ball on this bug report and I will not have time for a meaningful contribution until at least June 16 (bar admission exam). Life goes on and I got things working “good enough for me” (i.e. DAVdroid is a great tool to backup my phone to a server and it is improving every month, but for reasons that are not DAVdroid’s fault my goal of automatic, seamless synchronization across clients and devices cannot be achieved because of what I believe is a fundamental flaw in the CardDAV/CalDAV (the “protocols”) specifications. The underlying assumption of the protocols is that all clients behave the same. Such uniformity is only possible in a controlled environment (walled garden). The protocols (or at least their implementations) fail to account for the heterogeneity of the real world. Naturally occurring variations between clients cause unresolved conflicts that result in a race condition of clients mangling each other’s entries.

The Case for Custom Lines:

On Android I can add a custom phone field that takes any user entry as field’s name. So if I add a custom phone number field “aunt” to my contact Joe on Android and then sync it, DAVdroid will do its job well and on the server I will see

TEL;TYPE=x-aunt:(333) 333-3333

In an ideal world the expectation is that 3P clients would respect the x- prefix. In the real world, after syncing with a 3P client, the entry was corrupted / useless:

TEL;TYPE=work:(333) 333-3333

While it is impossible to predict exactly what every 3P client will do or will not do, it is more likely that a 3P client respects the X- prefix at the beginning of a line than an x- prefix within a sub-field.

Use custom lines profusely to be reasonably sure that your data will not be mangled by 3P clients. The cost of this redundancy (double bandwidth and double storage usage) are more than justified by the many benefits. The biggest benefit in the current context is the prevention of data loss/mangling due to unpredictable 3P clients.

Dealing With the Redundancy

Redundant storage in X-DAVDROID- custom lines is easy. The challenge is to keep those lines synchronized with the “standard” lines. The solution requires some data, some sync logic, some heuristic, and some display logic.

Data

The custom line will have a field, mapping it to a standard line. E.g.

X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
TEL;TYPE=work:(333) 333-3333

Logic

When syncing, DAVdroid will first identify standard lines that are (1) new or (2) changed. The corresponding custom line will be either (1) created or (2) identified. If a corresponding custom line is identified, DAVdroid will either (2a) prompt the user to solve the conflict or (2b) automatically update the custom line. The choice between 2a and 2b is a user-set preference.

Heuristic

There can be one to many correspondences. e.g.

X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
TEL;TYPE=work:(333) 333-3333
TEL;TYPE=work:(444) 444-4444

In such a case, the correspondence could be established by looking at the value in the field. If the situation stays ambiguous, the paramount consideration should be to avoid data loss, i.e. present the user with a dialog and ask the user to resolve the conflict and decide which field should be updated and how.

The challenge is, again, what 3P clients do to the data and is beyond DAVdroid’s control. Consider the following situation:

(a) user enter record on the Android phone, for two phone numbers:

X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=x-aunt:(333) 333-3333
TEL;TYPE=work:(444) 444-4444

(b) the user sync with a 3P client that mangles the server data:

X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=work:(333) 333-3333
TEL;TYPE=work:(444) 444-4444

now DAVdroid is syncing. If no changes was done to the numbers that are used as reference, this is not a problem, but what if the 3P client has updated the actual number?

X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=work:(555) 555-5555
TEL;TYPE=work:(444) 444-4444

This case looks easy, because apparently the two lines to be synced can be unambiguously determined. Wrong. Even such easy situation can be ambiguous. Has the user updated the 333 numbers to 555, or has he added the 555 number and the misbehaving client has deleted the x- prefixed entry without warning? In such ambiguous situation, on sync the user should be presented with the choice of

(1) replace aunt number from (333)333-3333 to (555)555-5555

X-DAVDROID-TEL;TYPE=x-aunt:(555) 555-5555;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=x-aunt:(555) 555-5555
TEL;TYPE=work:(444) 444-4444

or (2) restore aunt number and add the new work number

X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(555) 555-5555;MAP=TEL_TYPE_work
TEL;TYPE=x-aunt:(333) 333-3333
TEL;TYPE=work:(444) 444-4444
TEL;TYPE=work:(555) 555-5555

The worse outcome for the user is if information is silently discarded, hence it is important to protect the X- prefixed data and to make only changes that are explicitly approved by the user, even if that means bugging the user with questions on sync.

Display Logic

Once data integrity is dealt with, the last issue is what is being displayed /edited in the contacts application. I know too little about Android’s contacts app. Logic will have to be coded to make sure that the redundant entries are not displayed twice and that edits to standard fields by the contacts app are properly reflected in the custom field.

(Temporarily) Concluding Thoughts

CardDAV works well in a homogeneous single-client scenario but fails real world heterogeneous multiple-clients scenarios. The problem of data loss/corruption/mangling is a show stopper. I thought of solving it at the server, effectively making each client read-only. Such a solution allowing for editing on a master device only is too limited. I believe it is possible to craft a client so that it can be trusted to protect data from loss/mangling, and that DAVdroid can be that client. I had started a more thorough analysis of the problem and it is not complete yet, so it is possible that the above still need refinement. For now, I will keep my clients segregated, preventing them from mangling each other’s data, and use DAVdroid only as backup vector for my Android devices. At some point, I hope DAVdroid will become the trusted master of my Card/Cal entries, saving me from the currently manual work of syncing Cards between devices.

Thanks,
Yuv

Thanks for taking the time to discuss this issue. I have read your explanation and understand the problem.

What do you think about talking about this in the issue trackers of the 3P clients? As far as I can see, it’s a problem on their side. I don’t want to be lazy and move the resposibility, but (for instance) Thunderbird and Evolution are open-source too and deserve contribution to make them behave more standards-compliant.

At the moment, threre are no plans for this feature (custom lines + mapping), so I’ll close this for now. I suggest to create enhancement requests for your others clients so they process and generate X-types. If there’s an incompatibility (because they generate X-types DAVdroid doesn’t understand or vice versa), please feel free to create a new issue or post here again.