Contacts lose their `CATEGORIES`for reasons unclear

[ccoenen]

I started noticing this in the android 7 REV thread. I posted one of my contacts, who lost their CATEGORIES for no particular reason. I keep my VCARDS in a git repo, so this is easy to see (all future diffs have their personal info redacted!):

diff --git a/55cd82aa-5d0d-444e-b414-3905d2193d42.vcf b/55cd82aa-5d0d-444e-b414-3905d2193d42.vcf
index 3ae5862..1a0367f 100644
--- a/55cd82aa-5d0d-444e-b414-3905d2193d42.vcf
+++ b/55cd82aa-5d0d-444e-b414-3905d2193d42.vcf
@@ -1,13 +1,13 @@
 BEGIN:VCARD
-VERSION:3.0
-PRODID:-//Sabre//Sabre VObject 4.1.1//EN
+VERSION:4.0
+PRODID:+//IDN bitfire.at//DAVdroid/1.4.0.2-ose vcard4android ez-vcard/0.10.
+ 1
 UID:55cd82aa-5d0d-444e-b414-3905d2193d42
-CATEGORIES:work,friends
 FN:First Name Last Name
 N:Last Name;First Name;;;
 TEL;TYPE=cell:+49 123 4567
 EMAIL;TYPE=work:something@example.com
 ORG:some org
 BDAY:18000415
-REV:20170212T223553Z
-END:VCARD
\ No newline at end of file
+REV:20170215T095941Z
+END:VCARD

This just happened again, and this time I am certain that I did not accidentally click anything weird:

diff --git a/A.vcf b/A.vcf
index a5491c0..bca7a48 100644
--- a/A.vcf
+++ b/A.vcf
@@ -1,9 +1,9 @@
 BEGIN:VCARD
-VERSION:3.0
-PRODID:-//Sabre//Sabre VObject 4.1.1//EN
+VERSION:4.0
+PRODID:+//IDN bitfire.at//DAVdroid/1.4.0.2-ose vcard4android ez-vcard/0.10.
+ 1
 X-OPEN-XCHANGE-CTYPE:contact
 UID:113db4edb1
-CATEGORIES:activity-1,friends
 FN:redacted
 N:redacted;redacted;;;
 TEL;TYPE=cell:+49redacted
@@ -11,5 +11,5 @@ EMAIL;TYPE=home:redacted
 ADR;TYPE=home;LABEL=redactedlonglinelonglinelongline
  longlinecontinued
 BDAY:redacted
-REV:20170212T223742Z
-END:VCARD
\ No newline at end of file
+REV:20170218T093942Z
+END:VCARD

Since this is now the second contact to show this behaviour, I could find more commonalities between the two:

Both contacts changed recently because of a Nextcloud update to Contacts 1.5.3. This update contains a fix called “Detection of potentially broken vcards with auto correction features”, which is among other things removing a (probably faulty) backslash before a comma. So both of the contacts above have recently undergone a change like this:

--- a/55cd82aa-5d0d-444e-b414-3905d2193d42.vcf
+++ b/55cd82aa-5d0d-444e-b414-3905d2193d42.vcf
@@ -1,13 +1,13 @@
 BEGIN:VCARD
-VERSION:4.0
-PRODID:+//IDN bitfire.at//DAVdroid/1.3.6-ose vcard4android ez-vcard/0.10.1
+VERSION:3.0^M
+PRODID:-//Sabre//Sabre VObject 4.1.1//EN^M
 UID:55cd82aa-5d0d-444e-b414-3905d2193d42
-CATEGORIES:work\,friends
+CATEGORIES:work,friends
 FN:redacted

The important line being the last change from work\,friends to work,friends.

This is one of the related issues on this topic: https://github.com/nextcloud/contacts/issues/63

With the last update to contacts, they introduced a VCARD check & fix, and so far two of the affected contacts lost their categories when I used them afterwards. I do have a small number of contacts that should also show this behaviour, once I use them. I’d appreciate help with this bug, especially advice on how to get it fixed. I don’t know the specs well enough to even know which format is “right”, so I don’t know where to open a ticket right now.

[ccoenen]

It also sounds related to this topic: https://forums.bitfire.at/topic/1341/groups-categories-not-synced-anymore

[ccoenen]

I can easily tell with this command on my vcf backup:

git log -5 --color=always -GCATEGORIES -p | grep --color=never "commit\|CATEGORIES\|diff"
# (this is just here to remind myself later)

I identified 16 contacts that changed in the wake of the nextcloud/calendar upgrade. I checked them all, and here’s what I found:

• About half made the \, -> , conversion. ALL OF THOSE are now missing groups.
• The other half changed, but were already correctly formatted (only , or just one CATEGORY in the first place). THOSE still have their groups.

[rfc2822]

Thanks for your report. Can you please provide verbose steps to reproduce and DAVdroid debug info? Why has the VCard version changed from 3 to 4? Is DAVdroid still set to the CATEGORIES group method?

I’ll have to be able to reproduce the problem so that I can have a look at it.

[ccoenen]

@rfc2822 said in Contacts lose their `CATEGORIES`for reasons unclear:

Thanks for your report. Can you please provide verbose steps to reproduce and DAVdroid debug info?

I wish I could, I have no Idea how to re-introduce those faulty VCARDs back into the system (I presume they got there from earlier versions of nextcloud calendar).

If I were to guess the reproduction steps, thou would probably be

• have faulty (i.e. containing escaped comma) VCARDs on your server of choice
• SYNC (creating faulty entries on your android device)
• correct VCARDs on the server (i.e. remove comma)
• SYNC
• → wonder where your actual categories/groups went.

Why has the VCard version changed from 3 to 4?

As far as I can see, all VCARDS written by DAVdroid 1.3 and later are VERSION:4.0, so I do not fully understand the question.

Is DAVdroid still set to the CATEGORIES group method?

I don’t remember ever changing this setting (as far as I can tell your question was the first time I checked that deeply into settings). The chosen method is (and always has been) “Gruppen sind Kategorien der Kontakte” -> “groups are categories of contacts” (loosely translated from german translation)

I’ll have to be able to reproduce the problem so that I can have a look at it.

Yes, I can totally understand that. I will gladly help. There’s limits on what I can do, though. I don’t really have a debug setup, only my actual phone with my actual contacts.