Dropping events during sync (from offline state)

[AutoImport-hhhon]

Dear DAVdroid Developers,

I noticed that after adding events offline, they get dropped during synchronization.

Thank you in advance for fixing the bug.

Best regards
Henryk

My system

• CM-11 (Android 4.4.2), without gapps
• DAVdroid (0.5.12-alpha f-droid version)
• Radicale (0.8)

How to Reproduce the Problem

1. Go offline (DAVdroid should not be able to connect to the server)
2. Open calendar app and add an event
3. In calendar menu press Refresh (DAVdroid is not able to sync,
   nothing really happens)
4. Go online (DAVdroid should be able to connect to the server)
5. Try to sync (e.g. by pressing the Refresh option in the calendar
   app)
6. The added event gets removed

When the event is added, when DAVdroid is able to connect to the server, there are no problems.

Server Logs

I looked at the server logs and there was a difference in the HTTP request headers.

Failed request contained: ‘HTTP_IF_MATCH’: ‘*’

The successful request contained: ‘HTTP_IF_NONE_MATCH’: ‘*’,

Both logs are below for reference.

Failed Request

PUT request at /hhhon/calendar.ics/20140408T073359Z-2568_dffdded70276e7b1.ics received
Request headers:
{'CONTENT_LENGTH': '434',
 'CONTENT_TYPE': 'text/calendar',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_AUTHORIZATION': 'Basic XXX',
 'HTTP_CONNECTION': 'Keep-Alive',
 'HTTP_HOST': 'localhost:5232',
 'HTTP_IF_MATCH': '*',
 'HTTP_USER_AGENT': 'DAVdroid/0.5.12-alpha',
 'PATH_INFO': '/hhhon/calendar.ics/20140408T073359Z-2568_dffdded70276e7b1.ics',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_HOST': 'localhost.my.domain',
 'REQUEST_METHOD': 'PUT',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'localhost.my.domain',
 'SERVER_PORT': '5232',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'WSGIServer/0.1 Python/2.7.6',
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x801c401e0>,
 'wsgi.file_wrapper': <class wsgiref.util.FileWrapper at 0x804430e88>,
 'wsgi.input': <socket._fileobject object at 0x80468cb50>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}
Sanitized path: /hhhon/calendar.ics/20140408T073359Z-2568_dffdded70276e7b1.ics
Request content:
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//bitfire web engineering//DAVdroid 0.5.12-alpha//EN
BEGIN:VEVENT
DTSTAMP:20140408T073554Z
UID:20140408T073359Z-2568@dffdded70276e7b1
DTSTART;VALUE=DATE:20140412
DTEND;VALUE=DATE:20140413
SUMMARY:aaa
STATUS:CONFIRMED
ORGANIZER:mailto:hhhon@localhost
LAST-MODIFIED:20140408T073554Z
BEGIN:VALARM
TRIGGER:-PT10M
ACTION:DISPLAY
DESCRIPTION:aaa
END:VALARM
END:VEVENT
END:VCALENDAR

hhhon has read access to collection hhhon/calendar.ics/
hhhon has write access to collection hhhon/calendar.ics/
Answer status: 412 Precondition Failed

Successful Request

PUT request at /hhhon/calendar.ics/20140408T074309Z-2568_dffdded70276e7b1.ics received
Request headers:
{'CONTENT_LENGTH': '434',
 'CONTENT_TYPE': 'text/calendar',
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_AUTHORIZATION': 'Basic XXX',
 'HTTP_CONNECTION': 'Keep-Alive',
 'HTTP_HOST': 'localhost:5232',
 'HTTP_IF_NONE_MATCH': '*',
 'HTTP_USER_AGENT': 'DAVdroid/0.5.12-alpha',
 'PATH_INFO': '/hhhon/calendar.ics/20140408T074309Z-2568_dffdded70276e7b1.ics',
 'QUERY_STRING': '',
 'REMOTE_ADDR': '127.0.0.1',
 'REMOTE_HOST': 'localhost.my.domain',
 'REQUEST_METHOD': 'PUT',
 'SCRIPT_NAME': '',
 'SERVER_NAME': 'localhost.my.domain',
 'SERVER_PORT': '5232',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SOFTWARE': 'WSGIServer/0.1 Python/2.7.6',
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x801c401e0>,
 'wsgi.file_wrapper': <class wsgiref.util.FileWrapper at 0x804430e88>,
 'wsgi.input': <socket._fileobject object at 0x80468cb50>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}
Sanitized path: /hhhon/calendar.ics/20140408T074309Z-2568_dffdded70276e7b1.ics
Request content:
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//bitfire web engineering//DAVdroid 0.5.12-alpha//EN
BEGIN:VEVENT
DTSTAMP:20140408T074309Z
UID:20140408T074309Z-2568@dffdded70276e7b1
DTSTART;VALUE=DATE:20140412
DTEND;VALUE=DATE:20140413
SUMMARY:aaa
STATUS:CONFIRMED
ORGANIZER:mailto:hhhon@localhost
LAST-MODIFIED:20140408T074309Z
BEGIN:VALARM
TRIGGER:-PT10M
ACTION:DISPLAY
DESCRIPTION:aaa
END:VALARM
END:VEVENT
END:VCALENDAR

hhhon has read access to collection hhhon/calendar.ics/
hhhon has write access to collection hhhon/calendar.ics/
Answer status: 201 Created

[devvv4ever]

Thank you for submitting the bug report. We’ll look into this soon and keep you updated if we’re able to reproduce and/or fix it.

[rfc2822]

I couldn’t reproduce this problem on two different devices. However, your logs might give some hints:

I looked at the server logs and there was a difference in the HTTP request headers.
Failed request contained: ‘HTTP_IF_MATCH’: ‘*’

This is used for updating contacts, and means “Only process this request when there’s a resource at the URL [with any (*) ETag]”.

The server responded with 412 to this request, meaning that the update failed because there was no resource that could be updated.

When we look at WebDavResource::put(), we see that If-Match is only set to * when there’s no ETag yet, which can only be the case

• for a new resource that hasn’t been uploaded yet or
• when the server doesn’t return an ETag (but ETag support is required for CardDAV and I don’t know any CardDAV server not supporting ETags).

The successful request contained: ‘HTTP_IF_NONE_MATCH’: ‘*’,

This is used for adding new contacts, meaning “Only process this request when there’s no resource at the URL [with any (*) ETag]”.

I don’t know what this all means because I can’t reproduce the problem with my own CyanogenMod 11 device (and I use Radicale 0.8, too).

Can you try the account on our server and see if the problem happens too? Please contact us at play@bitfire.at so that we can exchange test account details.

[AutoImport-hperrey]

Just to weight in: I do see the same problem with both new contacts and new calendar entries which disappear as soon as I sync them if they were created when my server has been unavailable.

This is on a Moto G with the default Android installation on 4.4.2 with DAVdroid (0.5.13-alpha from f-droid) syncing with a DAViCal server (updated to the current git master two weeks ago) running on my NAS which is only available in the local wifi – I see this problem therefore as soon as I add a contact/appointment on the go. Modifications to existing entries work perfectly fine, however.

I don’t have any server logs to show at the moment, but would be happy to provide more information – just let me know what you need.

[rfc2822]

For providing logs, please see here: https://github.com/rfc2822/davdroid/wiki/How-to-view-the-logs

As I can’t reproduce the bug, more logs would be critical.

[AutoImport-hperrey]

Alright, I’ll test this as soon as I can and will provide the logs – but just to rectify: I just realized I only had DAVdroid 0.5.12 installed until a minute ago; I’ll see if I can reproduce this after the update.

[AutoImport-hhhon]

Now, I use DAVdroid 0.5.13-alpha, which seem to have the same problem.

I noticed that in order to reproduce this bug, it is necessary to try to synchronize (Settings/Accounts sync option) while being offline (step 3 from the original post). Sometimes, just adding a new event while being offline does not reproduce the problem.

[rfc2822]

I have an idea what the reason could be: findNew() creates remote file name and UID when a record is synced the first time. When this first sync where findNew() returns the new items suceeds, everything is OK.

However, when the sync does not succeed, maybe the contacts data provider takes the records as “updated” (because of the generated UID and remote file name fields) instead of “new”. This would explain why the resource is uploaded as “updated” record instead of a “new” record, causing the upload to fail (because on the remote server, the record doesn’t exist yet, so the update fails).