Support for values with double-quotes and newlines (RFC 6868)

llucax

I'm getting a notification with this mesage "Error while uploading created/modified entries".

The server is OwnCloud 9.0.1.

Here is the debug log I get when I click in that message (e-mail, server and user anonymized):

SYNCHRONIZATION INFO
Synchronization phase: 4
Account name: user@example.com
Authority: com.android.contacts

EXCEPTION:
java.lang.IllegalArgumentException: [Error 12] Property "ADR" has a parameter named "LABEL" whose value contains one or more invalid characters.  The following characters are not permitted: "
	at ezvcard.Messages.getIllegalArgumentException(Messages.java:84)
	at ezvcard.io.text.VCardRawWriter.sanitizeParameterValue(VCardRawWriter.java:464)
	at ezvcard.io.text.VCardRawWriter.writeProperty(VCardRawWriter.java:399)
	at ezvcard.io.text.VCardWriter._write(VCardWriter.java:320)
	at ezvcard.io.StreamWriter.write(StreamWriter.java:70)
	at ezvcard.io.chain.ChainingTextWriter.go(ChainingTextWriter.java:207)
	at ezvcard.io.chain.ChainingTextWriter.go(ChainingTextWriter.java:152)
	at at.bitfire.vcard4android.Contact.toStream(Contact.java:444)
	at at.bitfire.davdroid.syncadapter.ContactsSyncManager.prepareUpload(ContactsSyncManager.java:135)
	at at.bitfire.davdroid.syncadapter.SyncManager.uploadDirty(SyncManager.java:305)
	at at.bitfire.davdroid.syncadapter.SyncManager.performSync(SyncManager.java:147)
	at at.bitfire.davdroid.syncadapter.ContactsSyncAdapterService$ContactsSyncAdapter.onPerformSync(ContactsSyncAdapterService.java:66)
	at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:272)

SOFTWARE INFORMATION
DAVdroid version: 1.0.8 (99) Tue Apr 26 23:36:31 GMT+02:00 2016
Installed from: com.android.vending
JB Workaround installed: no

CONFIGURATION
System-wide synchronization: automatically
Account: user@example.com
  Address book sync. interval: 240 min
  Calendar     sync. interval: 240 min
  OpenTasks    sync. interval: 240 min

SQLITE DUMP
android_metadata
	|  locale |
	|  en_GB |
----------
settings
	|  setting | value |
----------
services
	|  _id | accountName | service | principal |
	|  1 | user@example.com | carddav | https://owncloud.example.com/remote.php/dav/principals/users/user/ |
	|  2 | user@example.com | caldav | https://owncloud.example.com/remote.php/dav/principals/users/user/ |
----------
sqlite_sequence
	|  name | seq |
	|  services | 2 |
	|  homesets | 2 |
	|  collections | 9 |
----------
homesets
	|  _id | serviceID | url |
	|  1 | 1 | https://owncloud.example.com/remote.php/dav/addressbooks/users/user/ |
	|  2 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/ |
----------
collections
	|  _id | serviceID | url | readOnly | displayName | description | color | timezone | supportsVEVENT | supportsVTODO | sync |
	|  1 | 1 | https://owncloud.example.com/remote.php/dav/addressbooks/users/user/contacts/ | 0 | Contacts | Default Address Book | <null> | <null> | <null> | <null> | 1 |
	|  2 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/personal_shared_by_sea/ | 1 | Imported(sea) | Imported | -1146130 | <null> | 1 | 1 | 1 |
	|  3 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/meetings/ | 0 | Meetings | Meetings | -65536 | <null> | 1 | 1 | 1 |
	|  4 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/berlinpublicholidays/ | 0 | Berlin Public Holidays | Berlin Public Holidays | -8355840 | <null> | 1 | 1 | 1 |
	|  5 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/work/ | 0 | Work | Work | -6306073 | <null> | 1 | 1 | 1 |
	|  6 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/vacations/ | 0 | Travels | Travels | -256 | <null> | 1 | 1 | 1 |
	|  7 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/concerts/ | 0 | Events | Events | -23296 | <null> | 1 | 1 | 1 |
	|  8 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/contact_birthdays/ | 0 | Birthdays | <null> | <null> | <null> | 1 | 1 | 1 |
	|  9 | 2 | https://owncloud.example.com/remote.php/dav/calendars/user/defaultcalendar/ | 0 | Personal | Personal | -4989844 | <null> | 1 | 1 | 1 |
----------

SYSTEM INFORMATION
Android version: 6.0.1 (cm_jfltexx-userdebug 6.0.1 MHC19Q ea7448f43c test-keys)
Device: Samsung GT-I9505 (jfltexx)

llucax

BTW, this started recently, after I used some app to set contact photos, which might have broken some contact(s). The problem is I can't find a way to tell which contacts are the ones with problems, and I changed a lot of them.

Is there a way to get which contacts are problematic?

rfc2822

java.lang.IllegalArgumentException: [Error 12] Property "ADR" has a parameter named "LABEL" whose value contains one or more invalid characters. The following characters are not permitted: "

Seems like there's an address with " in it, and ez-vcard can't escape it. Is that possible?

Is there a way to get which contacts are problematic?

You can always turn on and inspect the logs.

llucax

@rfc2822 said in Error while uploading created/modified entries:

java.lang.IllegalArgumentException: [Error 12] Property "ADR" has a parameter named "LABEL" whose value contains one or more invalid characters. The following characters are not permitted: "

Seems like there's an address with " in it, and ez-vcard can't escape it. Is that possible?

I didn't edit any contacts adding a ", but as I said I used a program to set contact photos, which might or might not have messed my contacts...

Is there a way to get which contacts are problematic?

You can always turn on and inspect the logs.

Yeah, I've seen the logs there but there is nothing that looks useful. This is what I get before the error:

2016-05-01 14:33:25 2 [syncadapter.ContactsSyncManager] Server advertises VCard/4 support: true
2016-05-01 14:33:25 2 [syncadapter.SyncManager] Processing locally deleted entries
2016-05-01 14:33:25 2 [syncadapter.SyncManager] Locally preparing dirty entries
2016-05-01 14:33:25 2 [syncadapter.SyncManager] Uploading dirty entries
2016-05-01 14:33:25 2 [vcard4android.AndroidGroup] Querying content://com.android.contacts/groups/18
2016-05-01 14:33:25 2 [vcard4android.AndroidGroup] Querying content://com.android.contacts/groups/14
2016-05-01 14:33:25 2 [syncadapter.SyncManager] Unknown sync error
EXCEPTION java.lang.IllegalArgumentException: [Error 12] Property "ADR" has a parameter named "LABEL" whose value contains one or more invalid characters.  The following
characters are not permitted: "

Are these "groups/NN" of any use for me? Should I copy the whole log before the error in here?

Thanks!

rfc2822

2016-05-01 14:33:25 2 [syncadapter.SyncManager] Uploading dirty entries
EXCEPTION java.lang.IllegalArgumentException: [Error 12] Property "ADR" has a parameter named "LABEL" whose value contains one or more invalid characters. The following
characters are not permitted: "

Ok, there's no network traffic yet which could be seen on the logs, because the contact is only changed locally and converting it to a VCard fails because there's an invalid address.

Please check your created/updated addresses for " and remove this character.

I can confirm that this also happens here when using addresses containing ". I will check whether " is allowed in addresses and if not, whether DAVdroid could rewrite/filter it.

rfc2822

According to VCard 3/4, " must not appear in values, can not even be quoted. Quite strange …

I wonder whether DAVdroid should silently filter/replace " by '?

llucax

Please check your created/updated addresses for " and remove this character.

This is extremely complicated because I have hundreds of contacts and I didn't explicitly add any " character in any of them. Is it possible to make davdroid print some debug message about which contact is failing?

According to VCard 3/4, " must not appear in values, can not even be quoted. Quite strange …

I wonder whether DAVdroid should silently filter/replace " by '?

Yeah, I think that could be a good idea, since it's super weird that anything that works in the Android contact app triggers an error in davdroid that breaks the sync.

Thanks!

llucax

Ok, I went through ALL my contacts and found one with " in it. It's very weird because is a very old contract that was very recently imported from the server because I reflashed the phone recently, so the " imported by DAVdroid without any problems. I updated the picture of that contact, and when DAVdroid want to upload the updated contact, then I got the error. I don't know exactly where, but this really looks like a bug somewhere.

Thanks!

rfc2822

I updated the picture of that contact, and when DAVdroid want to upload the updated contact, then I got the error. I don't know exactly where, but this really looks like a bug somewhere.

Did you import it locally or over CardDAV? It shouldn't be possible to transmit " over CardDAV, if I understand it correctly.

llucax

Did you import it locally or over CardDAV? It shouldn't be possible to transmit " over CardDAV, if I understand it correctly.

I had my phone without any contacts, I installed and configured DAVdroid, I synced the phone and I got this contact with a " in it.

rfc2822

Did you have a look at the logs to find the contact? Would be interesting how to VCard had looked like.

llucax

@rfc2822 said in Error while uploading created/modified entries:

Did you have a look at the logs to find the contact?

No, I went through every contact manually. I posted the log in here, I could find any mention to the contact, I think it would be useful to print the contact info in the debug output when this happens.

Would be interesting how to VCard had looked like.

I saw no information in the logs but what I posted here.

rfc2822

@llucax Yes, when uploading, there's still no traffic and log. But when you download the contacts, there's HTTP traffic and you can see what has been downloaded — which would be quite interesting, because it should not be possible to encode " in VCards, so I wonder how you could download it as VCard without an error …

llucax

Well, this contact was downloaded ages ago, but I can change it in the server and see what happens. My suggestion still applies though, right now is impossible to see which contact failed, and you can reproduce this problem by just modifying a contact adding a " in the address (I just verified I van make it fail again by adding a " in the address of any contact), which is quite bad.

rfc2822

@llucax What happens when you change it on the server to something with "?

llucax

@rfc2822 said in Error while uploading created/modified entries:

@llucax What happens when you change it on the server to something with "?

Yes, it works perfectly. This is what DAVdroid receives in the address according to the log:

ADR;LABEL=Humberto Primo Dto. 18  \n ;TYPE=home:;;Humberto Primo Dt
 o. 18";;;;

llucax

The text that appears in the contact view is

Humberto Primo 639 Dto. 18"

rfc2822

@llucax Is it a VCard VERSION:3 or VERSION:4?

llucax

VERSION:4.0
PRODID:-//Sabre//Sabre VObject 3.5.0//EN

rfc2822

@llucax Thanks. So the whole problem can be split into two parts:

Your server sends an invalid VCard. According to RFC 6350, »property parameter values must not contain the DQUOTE (U+0022) character« (details in the RFC, see the definition of ADR LABEL as param-value, and param-value can't contain "). Because of this server bug, the address could be imported over CardDAV and cause further problems. Will you report that to OwnCloud? A possible solution could be to use RFC 6868, see below.
The second problem is that values with " can't be uploaded over CardDAV/VCard, although they may appear in Android's local contact storage. So, the user should either be notified about this (like you were), or DAVdroid should be able to deal with this in a standard-compliant way. I have just found that RFC 6868 provides a way to escape ", but I'm not sure how wide-spread this solution is. I'll investigate. In the meanwhile, DAVdroid won't synchronize " over VCard because it's not possible.