TL;DR
I suggest a two-pronged solution:
-
Use custom lines like X-EVOLUTION-ABCD instead of TEL;TYPE=ABCD. Store all data redundantly in the same VCARD: once to custom X-DAVDROID-… lines and once to “standard” lines.
-
Map custom X-DAVDROID-… lines to the protocol’s “standard” lines and try to keep them in sync when the standard lines are changed by a third-party client (the “3P client”).
What Works for Me Now
Sorry I dropped the ball on this bug report and I will not have time for a meaningful contribution until at least June 16 (bar admission exam). Life goes on and I got things working “good enough for me” (i.e. DAVdroid is a great tool to backup my phone to a server and it is improving every month, but for reasons that are not DAVdroid’s fault my goal of automatic, seamless synchronization across clients and devices cannot be achieved because of what I believe is a fundamental flaw in the CardDAV/CalDAV (the “protocols”) specifications. The underlying assumption of the protocols is that all clients behave the same. Such uniformity is only possible in a controlled environment (walled garden). The protocols (or at least their implementations) fail to account for the heterogeneity of the real world. Naturally occurring variations between clients cause unresolved conflicts that result in a race condition of clients mangling each other’s entries.
The Case for Custom Lines:
On Android I can add a custom phone field that takes any user entry as field’s name. So if I add a custom phone number field “aunt” to my contact Joe on Android and then sync it, DAVdroid will do its job well and on the server I will see
TEL;TYPE=x-aunt:(333) 333-3333
In an ideal world the expectation is that 3P clients would respect the x- prefix. In the real world, after syncing with a 3P client, the entry was corrupted / useless:
TEL;TYPE=work:(333) 333-3333
While it is impossible to predict exactly what every 3P client will do or will not do, it is more likely that a 3P client respects the X- prefix at the beginning of a line than an x- prefix within a sub-field.
Use custom lines profusely to be reasonably sure that your data will not be mangled by 3P clients. The cost of this redundancy (double bandwidth and double storage usage) are more than justified by the many benefits. The biggest benefit in the current context is the prevention of data loss/mangling due to unpredictable 3P clients.
Dealing With the Redundancy
Redundant storage in X-DAVDROID- custom lines is easy. The challenge is to keep those lines synchronized with the “standard” lines. The solution requires some data, some sync logic, some heuristic, and some display logic.
Data
The custom line will have a field, mapping it to a standard line. E.g.
X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
TEL;TYPE=work:(333) 333-3333
Logic
When syncing, DAVdroid will first identify standard lines that are (1) new or (2) changed. The corresponding custom line will be either (1) created or (2) identified. If a corresponding custom line is identified, DAVdroid will either (2a) prompt the user to solve the conflict or (2b) automatically update the custom line. The choice between 2a and 2b is a user-set preference.
Heuristic
There can be one to many correspondences. e.g.
X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
TEL;TYPE=work:(333) 333-3333
TEL;TYPE=work:(444) 444-4444
In such a case, the correspondence could be established by looking at the value in the field. If the situation stays ambiguous, the paramount consideration should be to avoid data loss, i.e. present the user with a dialog and ask the user to resolve the conflict and decide which field should be updated and how.
The challenge is, again, what 3P clients do to the data and is beyond DAVdroid’s control. Consider the following situation:
(a) user enter record on the Android phone, for two phone numbers:
X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=x-aunt:(333) 333-3333
TEL;TYPE=work:(444) 444-4444
(b) the user sync with a 3P client that mangles the server data:
X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=work:(333) 333-3333
TEL;TYPE=work:(444) 444-4444
now DAVdroid is syncing. If no changes was done to the numbers that are used as reference, this is not a problem, but what if the 3P client has updated the actual number?
X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=work:(555) 555-5555
TEL;TYPE=work:(444) 444-4444
This case looks easy, because apparently the two lines to be synced can be unambiguously determined. Wrong. Even such easy situation can be ambiguous. Has the user updated the 333 numbers to 555, or has he added the 555 number and the misbehaving client has deleted the x- prefixed entry without warning? In such ambiguous situation, on sync the user should be presented with the choice of
(1) replace aunt number from (333)333-3333 to (555)555-5555
X-DAVDROID-TEL;TYPE=x-aunt:(555) 555-5555;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
TEL;TYPE=x-aunt:(555) 555-5555
TEL;TYPE=work:(444) 444-4444
or (2) restore aunt number and add the new work number
X-DAVDROID-TEL;TYPE=x-aunt:(333) 333-3333;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(444) 444-4444;MAP=TEL_TYPE_work
X-DAVDROID-TEL;TYPE=work:(555) 555-5555;MAP=TEL_TYPE_work
TEL;TYPE=x-aunt:(333) 333-3333
TEL;TYPE=work:(444) 444-4444
TEL;TYPE=work:(555) 555-5555
The worse outcome for the user is if information is silently discarded, hence it is important to protect the X- prefixed data and to make only changes that are explicitly approved by the user, even if that means bugging the user with questions on sync.
Display Logic
Once data integrity is dealt with, the last issue is what is being displayed /edited in the contacts application. I know too little about Android’s contacts app. Logic will have to be coded to make sure that the redundant entries are not displayed twice and that edits to standard fields by the contacts app are properly reflected in the custom field.
(Temporarily) Concluding Thoughts
CardDAV works well in a homogeneous single-client scenario but fails real world heterogeneous multiple-clients scenarios. The problem of data loss/corruption/mangling is a show stopper. I thought of solving it at the server, effectively making each client read-only. Such a solution allowing for editing on a master device only is too limited. I believe it is possible to craft a client so that it can be trusted to protect data from loss/mangling, and that DAVdroid can be that client. I had started a more thorough analysis of the problem and it is not complete yet, so it is possible that the above still need refinement. For now, I will keep my clients segregated, preventing them from mangling each other’s data, and use DAVdroid only as backup vector for my Android devices. At some point, I hope DAVdroid will become the trusted master of my Card/Cal entries, saving me from the currently manual work of syncing Cards between devices.
Thanks,
Yuv
