owncloud fails to sync contacts: "Received multi-get response without ETag"

[BigBoat]

has worked past year with no issues. after the last 2 play store updates no longer syncs contacts. Here is the debug log:

SYNCHRONIZATION INFO
Synchronization phase: 9
Account name: Public Contacts
Authority: com.android.contacts

STACK TRACE:
at.bitfire.dav4android.exception.DavException: Received multi-get response without ETag
at.bitfire.dav4android.exception.DavException: Received multi-get response without ETag
	at at.bitfire.davdroid.syncadapter.ContactsSyncManager.downloadRemote(ContactsSyncManager.java:185)
	at at.bitfire.davdroid.syncadapter.SyncManager.performSync(SyncManager.java:179)
	at at.bitfire.davdroid.syncadapter.ContactsSyncAdapterService$ContactsSyncAdapter.onPerformSync(ContactsSyncAdapterService.java:52)
	at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:259)

SOFTWARE INFORMATION
DAVdroid version: 0.9.0.4 (82) Fri Oct 23 21:08:05 EDT 2015
Installed from: Google Play Store
JB Workaround installed: no

CONFIGURATION
System-wide synchronization: automatically


SYSTEM INFORMATION
Android version: 5.0.2 (LRX22G.G925AUCU1AOE2)
Device: samsung / SAMSUNG-SM-G925A (zerolteatt)

Edit: prefixed/postfixed with ``` to make logs monospace

[rfc2822]

Hello,

Your server is defect. It advertises contacts, but doesn’t send ETags for them. ETags are required.

Please take verbose logs and contact your server manufacturer.

[BigBoat]

@rfc2822 said:

ETags are required.

I think you need to re-evaluate that response.
We have 3 other clients, 2 are on android and 1 is ios native that all sync just fine with owncloud. Even davdroid was syncing fine until the last update or two. I’d strongly recommend you don’t decide to break sync when an etag isn’t sent when it seems no other clients nor prior versions of davdroid required it. I looked at the rfc you sent and it’s use of the word “should” for etag makes me think it isn’t the same as “must”. What happens to all the other clients and prior versions of davdroid when they did not receive the e-tag? They were able to continue syncing somehow so it isn’t clear to me that an etag response is required vs something that “should” happen per the rfc. Regardless, i’ll look into getting e-tags sent from the server but I hope you can see that when everything is working, then davdroid has an update and can’t sync, telling someone their server is defect doesn’t make a ton of sense despite pointing a an RFC.

[rfc2822]

I’d strongly recommend you don’t decide to break sync when an etag isn’t sent when it seems no other clients nor prior versions of davdroid required it.

Basically: You’re right… see last paragraph.

I looked at the rfc you sent and it’s use of the word “should” for etag makes me think it isn’t the same as “must”.

I don’t read “should” here:

• The DAV:getetag property MUST be defined and set to a strong entity tag on all address object resources.
• A response to a GET request targeted at an address object resource MUST contain an ETag response header field indicating the current value of the strong entity tag of the address object resource.

Server MAY omit ETag on PUT responses, and DAVdroid (dav4android) handles that. But we’re not talking about PUT here, but about defined properties on address resources.

What happens to all the other clients and prior versions of davdroid when they did not receive the e-tag?

If you have a look at the response from your server, I guess (we don’t have logs here for exact analysis, so I have to guess) that some <response>s will have neither <resourcetype> nor <getetag>. That means, your server says that there are contacts available, but refuses to tell DAVdroid anything about some of them (not even their ETag, which would be required for synchronizing).

As far as I know, previous DAVdroid versions ignored those resources and tried to fetch them at every sync process again and again.

Regarding other clients: If they’re open-source, we can have a look at the source code.

They were able to continue syncing somehow so it isn’t clear to me that an etag response is required vs something that “should” happen per the rfc.

VCards MUST have an ETag in CardDAV (see above), and synchronizing without ETag doesn’t make any sense.

Regardless, i’ll look into getting e-tags sent from the server but I hope you can see that when everything is working, then davdroid has an update and can’t sync, telling someone their server is defect doesn’t make a ton of sense despite pointing a an RFC.

You’re right: probably it would be wise to make another workaround hack for OwnCloud (and maybe other servers) and to ignore such invalid responses.

But I don’t understand why every problem is automatically the fault of DAVdroid – did you even consider that your server may be broken?

By the way, somebody else reported this problem, too, and told me that the problem has vanished after updating to OwnCloud 8.1.3. Do you use a shared address book, too?

So please give more details about your OwnCloud version, etc. Logs would also be useful. Thanks

[BigBoat]

@rfc2822 said:

But I don’t understand why every problem is automatically the fault of DAVdroid – did you even consider that your server may be broken?

By the way, somebody else reported this problem, too, and told me that the problem has vanished after updating to OwnCloud 8.1.3. Do you use a shared address book, too?

I didn’t consider server broken in this case, though I do consider it in many cases. As I tried explaining above, the reason I didn’t consider the server here is because everything has been working. The only change was DavDroid received an update, then sync broke. Typically guy who last changed something, broke something. When we update the server code, and something happens, we look at the server. I understand what you are saying regarding following the RFC and not loving the idea of workaround hacks. It is just weird that the 3 other things we have syncing are doing fine and now just davdroid is throwing the error.
Yes we are on OwnCloud 8.1.3 and do use a shared address book. Its shared out by admin so all users can sync corporate contacts. I’ll be able to grab logs monday and begin seeing why e-tags aren’t making it, quick research shows it could be something with nginx, not even owncloud.

[BigBoat]

ah thought you wanted server logs, i see just the davdroid ones. makes sense, here they are http://pastebin.com/eA92tyYF

[rfc2822]

Typically guy who last changed something, broke something.

My approach to “broken (technical) things” is that everything tends to break all the time (by the law of entropy). Unless you put enormous efforts into maintaining a technical device, it will break sooner or later (in most cases, sooner). In this case, a part of the maintaining efforts is to find out why ETags don’t work, although they’re required by DAVdroid

When we update the server code, and something happens, we look at the server. I understand what you are saying regarding following the RFC and not loving the idea of workaround hacks. It is just weird that the 3 other things we have syncing are doing fine and now just davdroid is throwing the error.

To be honest, when I implemented the ETag check in the code, I didn’t think that it could be a problem. I just wanted to be sure that every resource has an ETag, so that the synchronization logic doesn’t to weird things, or crash with a NullPointerException because the ETag is null, or something like that. I thought (and still think) it was a good idea to force a well-defined sync state. Unfortunately, there are problems – now we have to solve them. Isn’t it a good thing to improve OwnCloud, so that CalDAV/CardDAV will work better in the future? (Of course, there’s still the possibility that the error is on DAVdroid’s side, even if I don’t think so in this special case.)

Yes we are on OwnCloud 8.1.3 and do use a shared address book. Its shared out by admin so all users can sync corporate contacts.

The other user had a shared calendar. He pointed me to https://github.com/owncloud/calendar/issues/832 and said that it affects only shared calendars. Maybe there’s a similar issue for shared address books.

I’ll be able to grab logs monday and begin seeing why e-tags aren’t making it, quick research shows it could be something with nginx, not even owncloud.

I don’t think so, because the questionable response is a 207 multi-get. nginx won’t intercept and change response bodies by itself.

ah thought you wanted server logs, i see just the davdroid ones. makes sense, here they are http://pastebin.com/eA92tyYF

Thanks. These are without verbose logs, aren’t they? With verbose logs, we would see the actual server response (and not only DAVdroid’s interpretation of it).

But as you can see:

[debug] Received <response> for https://owncloud.server.com/remote.php/carddav/addressbooks/user/contacts_shared_by_admin/05581f81-3e27-4370-8870-f7822b5879c0@owncloud.server.com.vcf, status: null, properties: [getetag(DAV:): null,
getcontenttype(DAV:): null,
address-data(urn:ietf:params:xml:ns:carddav): null]

For instance, this contact: /user/contacts_shared_by_admin/05581f81-3e27-4370-8870-f7822b5879c0@owncloud.server.com.vcf doesn’t have an ETag, nor Content-Type, nor any content (according to the server’s response).

[BigBoat]

here it is with verbose http://pastebin.com/rPPpBhSp
will investigate on the owncloud side as well.

[BigBoat]

Well investigation on owncloud side revealed that once I updated the contacts app itself to latest version we are now getting the e-tag.
http://pastebin.com/3xASnRrH

So as you correctly assumed, not a davdroid issue. I’d still recommend implementing some sort of “hack” as you say potentially throwing a warning but still allowing sync without e-tag. My reasoning is still that the other DAV clients were all still working and older DAVDroid were also working. It’s clear it was the server at fault here but its not clear the best client behavior is to hard fail the sync when other clients still work. thanks for the detailed responses, elaborating on “server is defect”.

[rfc2822]

I’m happy that it now works for you and will consider handling responses less strict (but I don’t know if and when, because that would change the sync logic again and at the moment, I don’t want to play around with it).