ampersands in caldav cause parsing error

CircleCode

I have 2 sogo accounts. One of them has an ampersand in username.
The one with ampersand fails to sync (couldn't parse multistatus XML element)

After reducing the case to the most minimal example, and after anonimization, here are the 2 xml files:

rfc2822

Hello,

Thanks for the report. As far as I know, ampersands have to be encoded as entities in XML, don’t they?

If you agree, can you please report this to SOGo and provide a link for reference here?

CircleCode

As far as I know, ampersands have to be encoded as entities in XML, don’t they?

Yes, and they are (open the attachment in a text editor to see this, it seems the forum opens them inline, thus decoding entities)

To avoid confusion, i include them inline:

Working

<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" 
  xmlns:a="urn:ietf:params:xml:ns:caldav">
  <D:response>
    <D:href>/SOGo/dav/foo@example.net/Calendar/personal/</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop></D:prop>
    </D:propstat>
    <D:propstat>
      <D:status>HTTP/1.1 404 Not Found</D:status>
      <D:prop>
        <a:calendar-data/>
      </D:prop>
    </D:propstat>
  </D:response>
  <D:response>
    <D:href>/SOGo/dav/foo@example.net/Calendar/personal/19A7-5CAA4180-15-5207B580.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <a:calendar-data>BEGIN:VCALENDAR
PRODID:-//Inverse inc./SOGo 4.0.7//EN VERSION:2.0 BEGIN:VTIMEZONE TZID:Europe/Paris X-LIC-LOCATION:Europe/Paris BEGIN:DAYLIGHT TZOFFSETFROM:+0100 TZOFFSETTO:+0200 TZNAME:CEST DTSTART:19700329T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:+0200 TZOFFSETTO:+0100 TZNAME:CET DTSTART:19701025T030000 RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:19A7-5CAA4180-15-5207B580 SUMMARY:foo CLASS:PUBLIC X-SOGO-SEND-APPOINTMENT-NOTIFICATIONS:NO TRANSP:OPAQUE DTSTART;TZID=Europe/Paris:20190408T100000 DTEND;TZID=Europe/Paris:20190408T104500
CREATED:20190407T182922Z
DTSTAMP:20190407T182922Z
LAST-MODIFIED:20190407T182922Z
END:VEVENT
END:VCALENDAR</a:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

Not working

<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" 
  xmlns:a="urn:ietf:params:xml:ns:caldav">
  <D:response>
    <D:href>/SOGo/dav/c&amp;m@example.net/Calendar/personal/</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop></D:prop>
    </D:propstat>
    <D:propstat>
      <D:status>HTTP/1.1 404 Not Found</D:status>
      <D:prop>
        <a:calendar-data/>
      </D:prop>
    </D:propstat>
  </D:response>
  <D:response>
    <D:href>/SOGo/dav/c&amp;m@example.net/Calendar/personal/3E5C-5CAB8E00-35-469F7780.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <a:calendar-data>BEGIN:VCALENDAR
PRODID:-//Inverse inc./SOGo 4.0.7//EN VERSION:2.0 BEGIN:VTIMEZONE TZID:Europe/Paris X-LIC-LOCATION:Europe/Paris BEGIN:DAYLIGHT TZOFFSETFROM:+0100 TZOFFSETTO:+0200 TZNAME:CEST DTSTART:19700329T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:+0200 TZOFFSETTO:+0100 TZNAME:CET DTSTART:19701025T030000 RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:3E5C-5CAB8E00-35-469F7780 SUMMARY:foo CLASS:PUBLIC X-SOGO-SEND-APPOINTMENT-NOTIFICATIONS:NO TRANSP:OPAQUE DTSTART;TZID=Europe/Paris:20190408T100000 DTEND;TZID=Europe/Paris:20190408T104500
CREATED:20190407T182922Z
DTSTAMP:20190407T182922Z
LAST-MODIFIED:20190407T182922Z
END:VEVENT
END:VCALENDAR</a:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

rfc2822

I see. Can you please provide the full DAVx⁵ exception info (which includes details about the exception, a stack trace etc)? This sounds really strange, because DAVx⁵ uses the Android XML parser and doesn’t parse by itself… Steps to reproduce would also be helpful.

CircleCode

Here is the log:

EXCEPTION:
at.bitfire.dav4jvm.exception.DavException: Couldn't parse multistatus XML element
at at.bitfire.dav4jvm.DavResource.processMultiStatus(DavResource.kt:494)
at at.bitfire.dav4jvm.DavResource.processMultiStatus(DavResource.kt:435)
at at.bitfire.dav4jvm.DavCalendar.calendarQuery(DavCalendar.kt:99)
at at.bitfire.davdroid.syncadapter.CalendarSyncManager$listAllRemote$2.invoke(CalendarSyncManager.kt:108)
at at.bitfire.davdroid.syncadapter.CalendarSyncManager$listAllRemote$2.invoke(CalendarSyncManager.kt:42)
at at.bitfire.davdroid.syncadapter.SyncManager.useRemote(SyncManager.kt:809)
at at.bitfire.davdroid.syncadapter.SyncManager.useRemoteCollection(SyncManager.kt:830)
at at.bitfire.davdroid.syncadapter.CalendarSyncManager.listAllRemote(CalendarSyncManager.kt:106)
at at.bitfire.davdroid.syncadapter.SyncManager$performSync$1$1.invoke(SyncManager.kt:142)
at at.bitfire.davdroid.syncadapter.SyncManager$performSync$1$1.invoke(SyncManager.kt:53)
at at.bitfire.davdroid.syncadapter.SyncManager.syncRemote(SyncManager.kt:455)
at at.bitfire.davdroid.syncadapter.SyncManager$performSync$1.invoke(SyncManager.kt:141)
at at.bitfire.davdroid.syncadapter.SyncManager$performSync$1.invoke(SyncManager.kt:53)
at at.bitfire.davdroid.syncadapter.SyncManager.unwrapExceptions(SyncManager.kt:835)
at at.bitfire.davdroid.syncadapter.SyncManager.performSync(SyncManager.kt:112)
at at.bitfire.davdroid.syncadapter.CalendarsSyncAdapterService$CalendarsSyncAdapter.sync(CalendarsSyncAdapterService.kt:55)
at at.bitfire.davdroid.syncadapter.SyncAdapterService$SyncAdapter.onPerformSync(SyncAdapterService.kt:73)
at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:321)
Caused by: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:TEXT @2:104 in okhttp3.ResponseBody$BomAwareReader@be74a6) 
at org.kxml2.io.KXmlParser.readEntity(KXmlParser.java:1220)
at org.kxml2.io.KXmlParser.readValue(KXmlParser.java:1402)
at org.kxml2.io.KXmlParser.next(KXmlParser.java:393)
at org.kxml2.io.KXmlParser.next(KXmlParser.java:313)
at org.kxml2.io.KXmlParser.nextText(KXmlParser.java:2077)
at at.bitfire.dav4jvm.Response$Companion.parse(Response.kt:113)
at at.bitfire.dav4jvm.DavResource$processMultiStatus$2.invoke(DavResource.kt:465)
at at.bitfire.dav4jvm.DavResource.processMultiStatus(DavResource.kt:484)
... 17 more

CircleCode

as an addition, the error

Caused by: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:TEXT @2:104 in okhttp3.ResponseBody$BomAwareReader@be74a6)

refers to the first letter in the xml entity:

<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" 
  xmlns:a="urn:ietf:params:xml:ns:caldav">
  <D:response>
    <D:href>/SOGo/dav/c&amp;m@example.net/Calendar/personal/</D:href>
                        ↑
                        This is 2:104 in raw output

@rfc2822 If required, I can give you a temporary access to this calendar

rfc2822

@CircleCode At the moment, we don’t have any left time. If you can keep the test account online some time, please create one and send it over forum chat. Thanks

CircleCode

for the reference, this may be related to https://sogo.nu/bugs/view.php?id=2479

As an addition, I am wondering if entities are expected at all in href (at xml level, they should be, of course, but at the caldav parser, I am wondering).
For example, when I share this calendar with a user not having an ampersand in its name, it becomes as following (note the replacements : & → _AM_, @ → _A_, . → _D_) :

<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" 
  xmlns:a="urn:ietf:params:xml:ns:caldav">
  <D:response>
    <D:href>/SOGo/dav/user@example.net/Calendar/c_AM_m_A_example_D_net_personal</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop></D:prop>
    </D:propstat>
    <D:propstat>
      <D:status>HTTP/1.1 404 Not Found</D:status>
      <D:prop>
        <a:calendar-data/>
      </D:prop>
    </D:propstat>
  </D:response>
  <D:response>
    <D:href>/SOGo/dav/user@example.net/Calendar/c_AM_m_A_example_D_net_personal/3E5C-5CAB8E00-35-469F7780.ics</D:href>
    <D:propstat>
      <D:status>HTTP/1.1 200 OK</D:status>
      <D:prop>
        <a:calendar-data>BEGIN:VCALENDAR
PRODID:-//Inverse inc./SOGo 4.0.7//EN VERSION:2.0 BEGIN:VTIMEZONE TZID:Europe/Paris X-LIC-LOCATION:Europe/Paris BEGIN:DAYLIGHT TZOFFSETFROM:+0100 TZOFFSETTO:+0200 TZNAME:CEST DTSTART:19700329T020000 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:+0200 TZOFFSETTO:+0100 TZNAME:CET DTSTART:19701025T030000 RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT UID:3E5C-5CAB8E00-35-469F7780 SUMMARY:Mat CLASS:PUBLIC X-SOGO-SEND-APPOINTMENT-NOTIFICATIONS:NO TRANSP:OPAQUE DTSTART;TZID=Europe/Paris:20190408T100000 DTEND;TZID=Europe/Paris:20190408T104500
CREATED:20190407T182922Z
DTSTAMP:20190407T182922Z
LAST-MODIFIED:20190407T182922Z
END:VEVENT
END:VCALENDAR</a:calendar-data>
      </D:prop>
    </D:propstat>
  </D:response>
</D:multistatus>

CircleCode

@rfc2822 I won’t be able to keep the test account online for time.

For reference:
I am using SoGo 4.0.7 on Gandi hosting.
I did nothing else than creating a mail with address c&m@example.net, and it should be sufficient to reproduce this issue.

rfc2822

Can you please send a test account nevertheless? I’ll have a look as soon as I find some time.

CircleCode

I did not find a way to send privte message, and I’d like to avoid posting credentials in the forum

CircleCode

may be related to https://sogo.nu/bugs/view.php?id=2479, but not sure

rfc2822

Thanks for the test account. When DAVx⁵ queries the resources, the server sends:

<?xml version="1.0" encoding="utf-8"?>
    <D:multistatus xmlns:a="urn:ietf:params:xml:ns:carddav" xmlns:D="DAV:"><D:response><D:href>/SOGo/dav/c%26m@example</D:href><D:propstat><D:status>HTTP/1.1 200 OK</D:status><D:prop><a:addressbook-home-set><D:href xmlns:D="DAV:">/SOGo/dav/c&m@example/Contacts/</D:href></a:addressbook-home-set><D:group-membership/></D:prop></D:propstat></D:response></D:multistatus>

which causes this DAVx⁵ error: Couldn't parse urn:ietf:params:xml:ns:carddavaddressbook-home-set EXCEPTION org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:TEXT @2:247 in okhttp3.ResponseBody$BomAwareReader@abfda3c)

As you can see, the & is not escaped in the href element, which is invalid XML and can’t be parsed. So, this seems to be a server problem and it seems to be related to the bug report you have posted.

If there are any specific indications that this could be a DAVx⁵ problem, please let us know.

ampersands in caldav cause parsing error

Similar topics

Resource file name being an encoded URL causes this URL to be accessed
DAVx⁵ • bug • • AutoImport-smoebody

Internal Server Error
DAVx⁵ • 3rd-party issue owncloud • • stefan

Birthdays are not correctly synced
DAVx⁵ • 3rd-party issue owncloud • • engerfranco

ampersands in caldav cause parsing error

Similar topics

Resource file name being an encoded URL causes this URL to be accessed DAVx⁵ • bug • • AutoImport-smoebody

Internal Server Error DAVx⁵ • 3rd-party issue owncloud • • stefan

Birthdays are not correctly synced DAVx⁵ • 3rd-party issue owncloud • • engerfranco

Resource file name being an encoded URL causes this URL to be accessed
DAVx⁵ • bug • • AutoImport-smoebody

Internal Server Error
DAVx⁵ • 3rd-party issue owncloud • • stefan

Birthdays are not correctly synced
DAVx⁵ • 3rd-party issue owncloud • • engerfranco