FastMail sync not working

[AutoImport-p3lim]

I’m trying to sync my calendars to FastMail without success, using CalDAV-Sync (competitor) works fine.

[AutoImport-dper]

@p3lim More information please. For example, what version of DAVdroid, what version of FastMail, what kind of phone, since when has it not worked, has it ever worked before, what error message are you seeing.

[AutoImport-p3lim]

I was just going to test this app because it has support for both CalDAV and CardDAV.

DAVdroid version: 0.6.9, latest release here.
Android version: 4.4.4, custom ROM (AOSPA)
FastMail is a mail service and doesn’t really have any version.
It’s the first time trying this app so I don’t know if it’s ever worked in previous versions.

I get no errors, the app apparently syncs up with the provider correctly (I’m able to select calendars to display), but there are no events shown.

[devvv4ever]

Confirmed. Creating the calendar and syncing works, but events are not shown in the calendar. We’ll try to investigate.

[AutoImport-untitaker]

Could it be that you’re using the wrong kind of account? IIRC CalDAV is only allowed for certain accounts ATM.

[AutoImport-p3lim]

@untitaker As I mentioned in the OP, using a competitive application (CalDAV-Sync) works.

In any case, I have the “Enhanced” plan that supports CalDAV.

[rfc2822]

Can you please verify that FastMail works correctly? In my tests, both CardDAV as well as CalDAV return invalid and strange responses, so I guess that’s not a DAVdroid problem and I wonder why I should do debugging work for FastMail.

[AutoImport-untitaker]

In which way are the responses strange? Last week I tested its CalDAV server and it worked fine (not with DAVDroid, but vdirsyncer)

[rfc2822]

1. CardDAV:

12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "PROPFIND /dav/addressbooks HTTP/1.1[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Content-Type: text/xml; charset=UTF-8[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Accept: text/xml[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Depth: 0[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Content-Length: 88[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Host: carddav-d277161.messagingengine.com[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Connection: Keep-Alive[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "User-Agent: DAVdroid/0.6.9.2[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "Authorization: Basic XXXXXXX[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "[\r][\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "<propfind xmlns="DAV:">[\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "   <prop>[\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "      <current-user-principal/>[\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "   </prop>[\n]"
12-28 15:37:46.620  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 >> "</propfind>"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "HTTP/1.1 400 Bad Request[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "Server: nginx[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "Date: Sun, 28 Dec 2014 14:39:42 GMT[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "Content-Type: text/html[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "Content-Length: 166[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "Connection: close[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "<html>[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "<head><title>400 Bad Request</title></head>[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "<body bgcolor="white">[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "<center><h1>400 Bad Request</h1></center>[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "<hr><center>nginx</center>[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "</body>[\r][\n]"
12-28 15:37:46.625  14275-15320/at.bitfire.davdroid:sync D/Wire﹕ http-outgoing-17 << "</html>[\r][\n]"

Bad request? Why?

2. CalDAV: Yesterday, the responses contained emails with VEVENTs instead of VEVENTs. Today, it works again (and so does DAVdroid with FastMail). So this was a FastMail problem.

[AutoImport-p3lim]

It can’t be a FastMail problem, because the sync works perfectly with your competitor; CalDAV-Sync.

[rfc2822]

It can’t be a FastMail problem, because the sync works perfectly with your competitor; CalDAV-Sync.

This is an argumentum ab auctoritate. Please provide information on what is wrong in the PROPFIND request above and I’ll try to fix it.

[AutoImport-untitaker]

@rfc2822 At some point you’ll need to provide workarounds for broken servers. Apart from SabreDAV and iCloud I’ve hardly seen servers that could be described as RFC-compliant.

[AutoImport-p3lim]

Please provide information on what is wrong in the PROPFIND request above and I’ll try to fix it.

If I knew what that was and how to do that, I would, I’m just a consumer with a broken product in this matter.

[rfc2822]

@rfc2822 At some point you’ll need to provide workarounds for broken servers. Apart from SabreDAV and iCloud I’ve hardly seen servers that could be described as RFC-compliant.

There will be no workarounds. Servers which are not compliant have to be fixed and won’t be supported by DAVdroid, that’s just not our goal.

If I knew what that was and how to do that, I would, I’m just a consumer with a broken product in this matter.

Please report this issue to FastMail, including a link to this issue report with the logs posted above.

[rfc2822]

It can’t be a FastMail problem, because the sync works perfectly with your competitor; CalDAV-Sync.

By the way: at the moment, FastMail CalDAV (not CardDAV) works with DAVdroid. It didn’t work yesterday and the days before because FastMail was broken in this time.

[AutoImport-p3lim]

I’ll forward this to @robn, see what he has to say.

[AutoImport-untitaker]

Servers which are not compliant have to be fixed and won’t be supported by DAVdroid, that’s just not our target.

I do understand where you’re coming from. Fixing issues in upstream is always the best option, but especially with such overengineered (and therefore almost impossible to fully support) protocols, the idea that the client developer wouldn’t have to implement any workarounds is IMO wishful thinking.

Note that this is just about accepting patches to DAVDroid which would implement such workarounds, not about loading that work onto your back.

[rfc2822]

I do understand where you’re coming from. Fixing issues in upstream is always the best option, but especially with such overengineered (and therefore almost impossible to fully support) protocols, the idea that the client developer wouldn’t have to implement any workarounds is IMO wishful thinking.

Of course, but there’s a difference between not supporting some “required” exotic property or feature (in this case, DAVdroid should ignore the issue as long the feature is not essential) and, for instance, a non-working current-user-principal (and I’m not blaming FastMail here: everyone can make mistakes, and it is perfectly possible that it’s DAVdroid’s fault, but at the moment, logs indicate otherwise).

And I want to mention that we already support a quite large number of different servers and services and there’s not a single workaround for any of them.

But I guess we shouldn’t abuse the topic here In case that someone submits a patch, the discussion will take place there.

[AutoImport-robn]

I’m unable to reproduce this, using either the standard or discovered-domain modes:

$ curl -i -u robntest@fastmail.fm:******** -X PROPFIND -d '<propfind xmlns="DAV:"><prop><current-user-principal/></prop></propfind>' -H 'Content-type: text/xml' https://carddav.messagingengine.com/dav/addressbooks
HTTP/1.1 207 Multi-Status
Server: nginx
Date: Sun, 28 Dec 2014 21:37:37 GMT
Content-Type: application/xml; charset=utf-8
Content-Length: 1067
Connection: keep-alive
Vary: Accept-Encoding, Brief, Prefer

<?xml version="1.0" encoding="utf-8"?>
<multistatus xmlns="DAV:">
  <response>
    <href>/dav/addressbooks</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/dav/principals/user/robntest@fastmail.fm/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
  <response>
    <href>/dav/addressbooks/user/robntest@fastmail.fm/Default/</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/dav/principals/user/robntest@fastmail.fm/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
  <response>
    <href>/dav/addressbooks/user/robntest@fastmail.fm/Default/e6aa4532-076f-4ab8-add0-4e8afb636b7b.vcf</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/dav/principals/user/robntest@fastmail.fm/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

$ curl -i -u robntest:******** -X PROPFIND -d '<propfind xmlns="DAV:"><prop><current-user-principal/></prop></propfind>' -H 'Content-type: text/xml' https://carddav-d49.messagingengine.com/dav/addressbooks
HTTP/1.1 207 Multi-Status
Server: nginx
Date: Sun, 28 Dec 2014 21:36:07 GMT
Content-Type: application/xml; charset=utf-8
Content-Length: 1067
Connection: keep-alive
Vary: Accept-Encoding, Brief, Prefer

<?xml version="1.0" encoding="utf-8"?>
<multistatus xmlns="DAV:">
  <response>
    <href>/dav/addressbooks</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/dav/principals/user/robntest@fastmail.fm/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
  <response>
    <href>/dav/addressbooks/user/robntest@fastmail.fm/Default/</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/dav/principals/user/robntest@fastmail.fm/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
  <response>
    <href>/dav/addressbooks/user/robntest@fastmail.fm/Default/e6aa4532-076f-4ab8-add0-4e8afb636b7b.vcf</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/dav/principals/user/robntest@fastmail.fm/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

2. CalDAV: Yesterday, the responses contained emails with VEVENTs instead of VEVENTs. Today, it works again (and so does DAVdroid with FastMail). So this was a FastMail problem.

This was an outright bug that we (somehow) hadn’t noticed. BusyCal reported it to us and it was fixed within a couple of hours.

I wonder why I should do debugging work for FastMail.

You shouldn’t, but you should certainly tell us when there’s a problem! We care a lot about standards compliance, but of course *DAV is going to have ridiculous bugs because its a ridiculous protocol. If you see something obviously wrong, let us know and we’ll fix it. You can @-mention me here, or email robn@fastmail.com.

[rfc2822]

You shouldn’t, but you should certainly tell us when there’s a problem! We care a lot about standards compliance, but of course *DAV is going to have ridiculous bugs because its a ridiculous protocol. If you see something obviously wrong, let us know and we’ll fix it. You can @-mention me here, or email robn@fastmail.com.

@robn Thanks for your helpful reply. I know that implementing *DAV is quite cumbersome and error-prone.

At the moment, I get only a 401 Unauthorized for CardDAV, while the same credentials work with CalDAV:

curl -vX PROPFIND -u username@fastmail.com:password -d '<propfind xmlns="DAV:"><prop><current-user-principal/></prop></propfind>' -H 'Content-type: text/xml' https://caldav-d277161.messagingengine.com/dav/calendars
# works

curl -vX PROPFIND -u username@fastmail.com:password -d '<propfind xmlns="DAV:"><prop><current-user-principal/></prop></propfind>' -H 'Content-type: text/xml' https://carddav-d277161.messagingengine.com/dav/addressbooks
> PROPFIND /dav/addressbooks HTTP/1.1
> Authorization: Basic xxxxxxxxxxxxx
> User-Agent: curl/7.37.0
> Host: carddav-d277161.messagingengine.com
> Accept: */*
> Content-type: text/xml
> Content-Length: 72
> 
* upload completely sent off: 72 out of 72 bytes
< HTTP/1.1 401 Unauthorized
* Server nginx is not blacklisted
< Server: nginx
< Date: Sun, 28 Dec 2014 22:41:55 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: keep-alive
* Authentication problem. Ignoring this.
< www-authenticate: Basic realm="carddav-d277161.messagingengine.com"
< x-frontend: frontend1

Would it be possible to get a test account at FastMail for DAVdroid that is not time-limited? We would use it only for testing.