Synchronization stuck if VCARD do not contain N, FN field (403 Forbidden, Synology)



  • Hi,

    It took two days, but finally I figured it out the root cause of the problem.

    Symptom: Synchronization fails with 403 Forbidden error. It do synchronize some contacts,
    until davdroid tries to synchronize a contact, where N or FN field are missing.
    Example:

    BEGIN:VCARD
    VERSION:2.1
    X-NICKNAME:Ryan Pulkovich
    X-JABBER:rapulkovich@gmail.com
    END:VCARD
    

    Example 2:

    BEGIN:VCARD
    VERSION:2.1
    EMAIL;INTERNET:renmio47@gmail.com
    END:VCARD
    

    The serverside is a Synology Carddav server, which is in reality a forked
    http://www.calendarserver.org/.

    Error message (on the server):

    2015-10-06 01:44:24+0200 [-] [caldav-0]  [-] [twistedcaldav.vcard#debug] vCard data had unfixable problems:
    2015-10-06 01:44:24+0200 [-] [caldav-0]           [VCARD] Missing required property: FN
    

    The actual bug is in "componentbase.py" file lines 194-212:

    for propname in self.propertyCardinality_1:
    195	            if self.countProperty(propname) != 1:
    196	                # Cannot fix a missing required property
    197	                logProblem = "[%s] Missing or too many required property: %s" % (self.getType(), propname,)
    198	                unfixed.append(logProblem)
    199	       
    200	        for propname in self.propertyCardinality_1_Fix_Empty:
    201	            if self.countProperty(propname) > 1:
    202	                # Cannot fix too many required property
    203	                logProblem = "[%s] Too many required property: %s" % (self.getType(), propname,)
    204	                unfixed.append(logProblem)
    205	            elif self.countProperty(propname) == 0:
    206	                # Possibly fix by adding empty property
    207	                logProblem = "[%s] Missing required property: %s" % (self.getType(), propname,)
    208	                if doFix:
    209	                    self.addProperty(PyCalendarProperty(propname, ""))
    210	                    fixed.append(logProblem)
    211	                else:
    212	                    unfixed.append(logProblem)
    

    After manualy adding fields: self.addProperty(PyCalendarProperty(propname, "UNKNOWN")), it does fail at another spot (vcard.py:571) too:

            fixed, unfixed = self._pycard.validate(doFix=doFix)
    

    I believe they intentionally fail if required vcard property is missing. So I think this should be fixed/workarounded on the client side (davdroid).

    I think adding some random string to the N and FN fields (eg: UNKNOWN-231, UNKNOWN-562), would make this bug go away. Also self evident on the phone, which contacts needs to be fixed.

    My exported vcard file contains 1000+ contacts, it is really hard to spot which contact is problematic. Also davdroid do not shows at exactly which contacts the synchronization fails (which would be a tremendous help in debugging).

    Please implement the above simple workaround.

    Best,
    Laszlo

    (Edit: formatted code)



  • A slightly unrelated question:
    Where can I read the full thread of issue #48? (vcard categories support)

    The github issues are disabled, and the forum do not have a copy of the issues.
    It is referenced basically everywhere (even on Android bug tracker), but can not be read, and not archived anywhere.


  • developer

    I will consider how to deal with contacts which don't have a name (i.e. not enough information to be represented in a VCard).

    BEGIN:VCARD
    VERSION:2.1
    EMAIL;INTERNET:renmio47@gmail.com
    END:VCARD
    

    May I ask what's the reason for such a contact? Shouldn't it have a name so that you know who it belongs to?

    I think if no FN name is present, it can be generated from a phone number or email address, or otherwise "—" can be used.

    However, for N, I see a problem: it must be present too, and it represents a structured name, and nothing else. That means if you upload a contact with only a phone number, and DAVdroid generates a "random" structured name, let's say "Unnamed Contact", and then you import the VCard again (either with DAVdroid or another client), the contact will have a structured name of "Contact, Unnamed". I don't know whether this makes sense.

    On the other side, one could argue that when DAVdroid is told to upload a contact to CardDAV (i.e. in VCard format), and VCard requires FN and N, it's thus entitled to generate all required values to do so, even if it doesn't make 100% sense.

    This topic was covered once for contacts with only a prefix/suffix, but it seems that sometimes users want to synchronizes contacts without any name component, too.

    Where can I read the full thread of issue #48? (vcard categories support)

    I guess you mean this one: https://forums.bitfire.at/topic/129/category-group-field-for-contacts-is-not-synchronized

    But you can just use the forum search.

    The github issues are disabled, and the forum do not have a copy of the issues.
    It is referenced basically everywhere (even on Android bug tracker), but can not be read, and not archived anywhere.

    All issues have been imported to the forum as threads. Unfortunately, the issue IDs are not matching the thread IDs.



  • @rfc2822

    May I ask what's the reason for such a contact? Shouldn't it have a name so that you know who it belongs to?

    Some android email client can autosuggest email address if it is in the contacts (like K-9 Mail for example).
    I don't know which program is responsible "contaminating" my phonebook, but it didn't cause any problem until now (maybe my old Nokia N900?)

    The android phonebook (contacts) do not show contacts which do not have a name field. So I was not fully aware of those contacts.

    But all those contacts are valid. I had conversation with each of them via email at least once.

    That is why I'm interested in categories, because I could make a distinction between real contacts with real phone number, and email contacts.

    I would prefer duplicate contacts instead of no synchronization at all.

    And I can confirm, after manually adding 458 N,FN fields, I synchronized 1149 contacts.
    Still some synchronization error occurs, I'm investigating.... (Could not access local content provider, android 4.4, sony xperia z1 compact)



  • @Laszlo said:

    Still some synchronization error occurs, I'm investigating.... (Could not access local content provider, android 4.4, sony xperia z1 compact)

    Ok, here is the conclusion:

    • 1149 contacts in total, bugfixed in exported vcard textfile (manually added 458 piece N,FN fields).
    • initial import is OK (sony xperia z1 compact, android 4.4)
    • initial upload to the carddav server is OK (Synology builtin carddav server, forked apple's calendarserver.org)
    • initial import to the new mobile is ok (Oneplus one, cyanogenmod 12)
    • subsequent syncing on source phone (android 4.4) results in

    Could not access local content provider

    after about 3-4 minutes of "work" (I suppose it is work, not much indicated).
    Also there are some request made to the server.

    I strongly suspect davdroid do not synchronize in "packets". I believe the correct way would be to synchronize in step of 100 contacts, correctly divided in separate syncing process, finalizing everything on android. Then restart with another 100 contacts.

    None of my contacts have profile picture bigger then 500kB. The whole exported .vcf file is 1.1MB with 1149 contacts in them, uncompressed (textfile).

    I hope this "analyse" helped a bit.


  • developer

    @Laszlo said:

    Could not access local content provider

    after about 3-4 minutes of "work" (I suppose it is work, not much indicated).
    Also there are some request made to the server.

    Can you maybe provide logs for what happens exactly? The "Couldn't access local content provider" just means that there was an error accessing (probably writing) the content provider (Contacts storage in your case). In other words, it's a kind of database error when writing a contact.

    I strongly suspect davdroid do not synchronize in "packets". I believe the correct way would be to synchronize in step of 100 contacts, correctly divided in separate syncing process, finalizing everything on android. Then restart with another 100 contacts.

    You can just fetch a list of all contacts from the server, and then process them one by one. DAVdroid

    1. uploads local changes first (one by one),
    2. fetches a list of contacts,
    3. determines which contacts have changed and thus have to be downloaded,
    4. downloads them in groups using multi-get,
    5. inserts/updates every multi-get result (one by one),
    6. deletes all local contacts that are not on the server anymore.


  • Found a new way triggering the bug:

    1. Go to Gmail's contact manager
    2. Click on export contacts (switch to old contact manager interface)
    3. choose my contacts -> More (dropdown) -> Export...
    4. Select The group (My contacts) 292
      vCard format (for importing into Apple Address Book or another application)

    Now you have an exported contacts.vcf file, with contacts which looks like this:

    BEGIN:VCARD
    VERSION:3.0
    FN:
    N:;;;;
    EMAIL;TYPE=INTERNET:a.novai@trebushid.hu
    NOTE:Utónév\: Áron\nVezetéknév\: Novai\nNév\: Novai Áron\n
    END:VCARD
    

    Please note, the localized NOTE field, where "Utónév" means first name, "Vezetéknév" means family name (hungarian language).

    And that is gmail standard export.
    On the server the exact same behaviour, when the N, FN field was completely missing (note, there is N, FN line in the exported vcard file):

    2015-10-07 16:47:44+0200 [-] [caldav-1]  [-] [twistedcaldav.vcard#debug] vCard data had unfixable problems:
    2015-10-07 16:47:44+0200 [-] [caldav-1]           [VCARD] Missing or too many required property: N, [[vCard Property: PRODID:DAVdroid/0.8.4.1 (ez-vcard/0.9.6)
    2015-10-07 16:47:44+0200 [-] [caldav-1]         ], [vCard Property: VERSION:3.0
    2015-10-07 16:47:44+0200 [-] [caldav-1]         ], [vCard Property: UID:b5d936ba-cd33-4431-af7b-ed77cdcc8d11
    2015-10-07 16:47:44+0200 [-] [caldav-1]         ], [vCard Property: REV:20151007T144743Z
    2015-10-07 16:47:44+0200 [-] [caldav-1]         ], [vCard Property: NOTE:Utónév: Áron\nVezetéknév: novai\nNév: Novai Áron
    2015-10-07 16:47:44+0200 [-] [caldav-1]         ], [vCard Property: EMAIL;TYPE=x-internet:a.novai@trebushid.hu
    2015-10-07 16:47:44+0200 [-] [caldav-1]         ]]
    2015-10-07 16:47:44+0200 [-] [caldav-1]           [VCARD] Missing required property: FN
    

    I modified on the server side, to print the complete VCARD info, to see what vcard causes the problem.

    I'm really happy to find a really common way (exporting email contacts from gmail interface) to trigger the bug. So it must be a widespread bug then! :)

    I tried to fix the problem by exporting them, fixing it in a text editor and reimporting them, but exporting shows me an error message: Cannot export more than 500 contacts.

    So once a "non-compliant" contact gets in the system breaks synchronization completely.... :-(


  • developer

    Please note, the localized NOTE field, where "Utónév" means first name, "Vezetéknév" means family name (hungarian language).

    And that is gmail standard export.

    Very strange. Empty FN and N values and instead putting the name into NOTE is not standard-compliant behaviour. Can you imagine how this can happen? Isn't that a severe bug in the Gmail export? Unfortunately, I don't have a GMail account, but maybe I can try with some friend's account.



  • Also I kind of stuck with this whole issue. I tried 20+ "contact manager" application from play store, none of them can export the whole contact database as VCARD format, most of them export only the name and phone number (like .csv file).

    I found on F-droid an "Export Contacts" application (http://bzr.ed.am/android/export-contacts),
    but it rewrites the VCARD to version 3.0, and most importantly all the photos BASE64 images are omitted from the export.

    If I delete the contacts from the phone, and try to sync with the server, it will delete all my contacts from the server too.

    Also exporting contacts from gmail.com was kind of necessary, otherwise autocomplete in K9-mail (or any other non-google email application) do not work.

    Update 1:
    I found the first application (MCBackup, full of ads), which exports all the contacts, but it replaces all the photos (BASE64 jpg originally) with much larger Base64 png one. Can not understand why it modify the photo in the first place.

    I think it is an impossible situation, I can not make a backup of my contacts anymore, no point updating them on the phone. I can only make a backup if I select exported contacts one-by-one (and wait for the animation to complete between two selection)....

    Update 2:
    OK, I literally tried more then 50 apps (54 to be exact) none of them export promptly. The photos are either missing or converted or only those contacts are exported which has a phone number (494), or some random number.



  • Reading DAVDroid sourcecode, I think these are the bits where the workaround should be put:
    at/bitfire/davdroid/syncadapter/SyncManager.java:139-161

    private int pushNew() throws URISyntaxException, LocalStorageException, IOException, HttpException {
    	int count = 0;
    	long[] newIDs = local.findNew();
    	Log.i(TAG, "Uploading " + newIDs.length + " new resource(s) (if not existing)");
    	try {
    		for (long id : newIDs)
    			try {
    				Resource res = local.findById(id, true);
    				String eTag = remote.add(res);
    				if (eTag != null)
    					local.updateETag(res, eTag);
    				local.clearDirty(res);
    				count++;
    			} catch (ConflictException|PreconditionFailedException e) {
                    Log.i(TAG, "Didn't overwrite existing resource with other content");
    			} catch (RecordNotFoundException e) {
    				Log.wtf(TAG, "Couldn't read new record", e);
    			}
    	} finally {
    		local.commit();
    	}
    	return count;
    }
    

    Here we should make an additional safety check, at this line:

    Resource res = local.findById(id, true);
    local.ensureMusthaveFields(res);
    

    And in at/bitfire/davdroid/resource/LocalCollection.java we should make a

    	public void ensureMusthaveFields(Resource res) throws LocalStorageException {
    

    method.

    I'm also seeing in that file a res.getName() method, but I can not find, if it is the actual N and FN fields...

    I never wrote Android program before, and I'm not really in java programming either.
    But it must be a relatively easy programming task, although I need some helping hand.
    Also no idea how to compile DAVdroid from source and how to try it on an actual device.

    This bug is so annoying!

    Laszlo


Log in to reply
 

Looks like your connection to Bitfire App Forums was lost, please wait while we try to reconnect.