Duplicate calendar events


  • developer

    @cgogolin Can you

    1. unselect the calendar
    2. sync again (so that everthing is gone)
    3. then turn on verbose DAVdroid logging
    4. select the calendar again
    5. sync again and
    6. then turn off the DAVdroid logs and post it here? If you don’t want to anonymize your data, you can send it to play@bitfire.at (will be treated confidentially, PGP key on https://davdroid.bitfire.at) instead of posting it.


  • After the procedure You described other entries appear as duplicate (this was always the case, but I think I didn’t mention this before). One entry that is now duplicated appears in the dump as

    INSERT INTO "Events" VALUES(3161,'b133303f-f0d2-4cc9-a89b-cf4a49f5e003.ics',0,NULL,0,12,'XXX',NULL,NULL,NULL,NULL,NULL,0,1499709600000,1499727600000,'Europe/Madrid',NULL,0,0,0,0,0,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1499727600000,1,0,1,1,'xxx@xxx.net',NULL,0,'Europe/Madrid',NULL,NULL,'b133303f-f0d2-4cc9-a89b-cf4a49f5e003','8165fd52278e63746e3da1f257ab6a2e',NULL,'0',NULL,NULL,NULL,NULL,NULL,NULL,NULL);
    

    and then further down with the first entry in VALUES() equal to 3177 instead of 3161.

    The logging produced three pairs of log files, davdroid-26440-20170828-114615.txt, davdroid-26476-20170828-114615.txt, and davdroid-27951-20170828-115853.txt, each with an accompanying and empty .txt.lck file.

    The first file, davdroid-26440-20170828-114615.txt, simply contains:

    2017-08-28 11:46:18 1789 [AccountSettings] Account xxx@xxx.net has version 6, current version: 6
    2017-08-28 11:59:25 1 [log.Logger] Verbose logging: false
    

    The last file, davdroid-27951-20170828-115853.txt, is attached (anonymized)
    as 0_1503916650335_davdroid-27951-20170828-115853.txt .

    The second file is huge (8.6MB) and contains all(?) the clear text content of the calendar entries. I am not comfortable sharing this with anyone and not sure how to a anonymize it. The duplicate entry quoted above appears in three places in that file, here

    END:VEVENT
    END:VCALENDAR
    </cal:calendar-data></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/dav/calendars/user1/username2_username1_shared_by_user2/b133303f-f0d2-4cc9-a89b-cf4a49f5e003.ics</d:href><d:propstat><d:prop><d:getcontenttype>text/calendar; charset=utf-8; component=vevent</d:getcontenttype><d:getetag>&quot;8165fd52278e63746e3da1f257ab6a2e&quot;</d:getetag><cal:calendar-data>BEGIN:VCALENDAR
    PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
    VERSION:2.0
    BEGIN:VTIMEZONE
    TZID:Europe/Madrid
    BEGIN:DAYLIGHT
    TZOFFSETFROM:+0100
    TZOFFSETTO:+0200
    TZNAME:CEST
    DTSTART:19700329T020000
    RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
    END:DAYLIGHT
    BEGIN:STANDARD
    TZOFFSETFROM:+0200
    TZOFFSETTO:+0100
    TZNAME:CET
    DTSTART:19701025T030000
    RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
    END:STANDARD
    END:VTIMEZONE
    BEGIN:VEVENT
    CREATED:20170713T103641Z
    LAST-MODIFIED:20170713T103720Z
    DTSTAMP:20170713T103720Z
    UID:b133303f-f0d2-4cc9-a89b-cf4a49f5e003
    SUMMARY:XXX
    DTSTART;TZID=Europe/Madrid:20170710T200000
    DTEND;TZID=Europe/Madrid:20170711T010000
    TRANSP:OPAQUE
    END:VEVENT
    END:VCALENDAR
    </cal:calendar-data></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/owncloud/remote.php/dav/calendars/user1/username2_username1_shared_by_user2/df4c82ed-d3fd-4845-8b38-e235036ec850.ics</d:href><d:propstat><d:prop><d:getcontenttype>text/calendar; charset=utf-8; component=vevent</d:getcontenttype><d:getetag>&quot;34d173fb52668249e9234571382be094&quot;</d:getetag><cal:calendar-data>BEGIN:VCALENDAR
    PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
    VERSION:2.0
    BEGIN:VEVENT
    

    here

    2017-08-28 11:49:40 1807 [net.fortuna.ical4j.data.CalendarParserImpl:464] [58]
    2017-08-28 11:49:40 1807 [net.fortuna.ical4j.data.CalendarParserImpl:464] [-3]
    2017-08-28 11:49:40 1807 [net.fortuna.ical4j.data.CalendarParserImpl:464] [10]
    2017-08-28 11:49:40 1809 [dav4android.DavResource] Received <response> for https://xxx.xxx.org:8080/owncloud/remote.php/dav/calendars/user1/username2_username1_shared_by_user2/b133303f-f0d2-4cc9-a89b-cf4a49f5e003.ics
    	PARAMETER #0 = [calendar-data(urn:ietf:params:xml:ns:caldav): CalendarData(iCalendar=BEGIN:VCALENDAR
    PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
    VERSION:2.0
    BEGIN:VTIMEZONE
    TZID:Europe/Madrid
    BEGIN:DAYLIGHT
    TZOFFSETFROM:+0100
    TZOFFSETTO:+0200
    TZNAME:CEST
    DTSTART:19700329T020000
    RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
    END:DAYLIGHT
    BEGIN:STANDARD
    TZOFFSETFROM:+0200
    TZOFFSETTO:+0100
    TZNAME:CET
    DTSTART:19701025T030000
    RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
    END:STANDARD
    END:VTIMEZONE
    BEGIN:VEVENT
    CREATED:20170713T103641Z
    LAST-MODIFIED:20170713T103720Z
    DTSTAMP:20170713T103720Z
    UID:b133303f-f0d2-4cc9-a89b-cf4a49f5e003
    SUMMARY:XXX
    DTSTART;TZID=Europe/Madrid:20170710T200000
    DTEND;TZID=Europe/Madrid:20170711T010000
    TRANSP:OPAQUE
    END:VEVENT
    END:VCALENDAR
    ), getcontenttype(DAV:): GetContentType(type=text/calendar; charset=utf-8; component=vevent), getetag(DAV:): 8165fd52278e63746e3da1f257ab6a2e]
    2017-08-28 11:49:40 1807 [net.fortuna.ical4j.data.CalendarParserImpl:464] [-3]
    2017-08-28 11:49:40 1807 [net.fortuna.ical4j.data.CalendarParserImpl$PropertyParser:240] Property [CREATED]
    

    and here

    2017-08-28 11:49:38 1807 [net.fortuna.ical4j.data.CalendarParserImpl:464] [58]
    2017-08-28 11:49:38 1807 [net.fortuna.ical4j.data.CalendarParserImpl:464] [-3]
    2017-08-28 11:49:38 1807 [net.fortuna.ical4j.data.CalendarParserImpl:520] [VCALENDAR]
    2017-08-28 11:49:38 1809 [dav4android.DavResource] Received <response> for https://xxx.xxx.org:8080/owncloud/remote.php/dav/calendars/user1/username2_username1_shared_by_user2/20170629T094205Z-8f48612dd2dfaecf.ics
    	PARAMETER #0 = [getetag(DAV:): dcef88b8223886602f2bef7e3ec3730a]
    2017-08-28 11:49:38 1807 [ical4android.Event] Assigning exceptions to master events
    2017-08-28 11:49:38 1807 [syncadapter.CalendarSyncManager] Adding b133303f-f0d2-4cc9-a89b-cf4a49f5e003.ics to local calendar
    2017-08-28 11:49:38 1807 [ical4android.AndroidEvent] Built event object
    	PARAMETER #0 = mType: 1, mUri: content://com.android.calendar/events?account_name=xxx%40xxx.net&account_type=bitfire.at.davdroid&caller_is_syncadapter=true&account_name=xxx%40xxx.net&account_type=bitfire.at.davdroid&caller_is_syncadapter=true, mSelection: null, mExpectedCount: null, mYieldAllowed: false, mValues: eventTimezone=Europe/Madrid title=XXX availability=0 allDay=0 hasAttendeeData=1 dtend=1499727600000 dtstart=1499709600000 calendar_id=12 eventEndTimezone=Europe/Madrid, mValuesBackReferences: null, mSelectionArgsBackReferences: null
    2017-08-28 11:49:38 1809 [dav4android.DavResource] Received <response> for https://xxx.xxx.org:8080/owncloud/remote.php/dav/calendars/user1/username2_username1_shared_by_user2/20170703T113241Z-8f48612dd2dfaecf.ics
    	PARAMETER #0 = [getetag(DAV:): 9b6762ffddff825aa2681d464dddb6e8]
    

    I am happy to send you more snippets from that file, if you tell me what you need.


  • developer

    @cgogolin Looks good to me and I can’t see why this event is added again. However, I’d really need to look at the whole sync process. Maybe you can just replace all SUMMARY, LOCATION, DESCRIPTION, DTSTART, DTEND values with XXX to anonymize it and send it to play@bitfire.at?



  • @rfc2822 The problem is that many of these fields have values that extend over several lines. For the fields you mentioned I can remove with emacs regexp-replace and regular experssions of the type

    DESCRIPTION:.*\(
     .*\)*
    

    because all the following lines start in with a white space. But the descriptions also appear in lines such as

    2017-08-28 11:52:51 1809 [ical4android.AndroidEvent] Built event object
    	PARAMETER #0 = mType: 1, mUri: content://com.android.calendar/events?account_name=xxx%40xxx.net[...]description=THE DESCRIPTION
    HERE EXTENDS
    OVER SEVERAL LINES title=[...]
    

    Here I get a “Stack overflow in regexp matcher” if I try to match them with a regexp of the type

    description=\(?:.\|
    \)+title=
    

    I know I am asking for much, but do you have any idea how to remove them? At the same time it is not really my fault that the logs are printed in a format that is so difficult o annonymize… 😉



  • I used grep to extract all lines that contain the string b133303f-f0d2-4cc9-a89b-cf4a49f5e003.ics from davdroid-26476-20170828-114615.txt. Maybe that contains already enough information for you to see what is going on? 0_1503928139665_davdroid-26476-20170828-114615.txt.greped


  • developer

    @cgogolin No, I’d need the context. It’s already hard enough to look through real logs, but only a few grepped lines for a bug that I can’t reproduce … maybe we can wait until this problem occurs more often so that it can finally be reproduced.



  • In the end I removed all the title= and description= parts manually. I just sent you the file via email to play@bitfire.at. Hope you will see something in there…


  • developer

    @cgogolin Thanks for the logs. Unfortunately, I just can see that at some time, there are some new entries which appear on the server (e.g. https://xxx.xxx.org:8080/owncloud/remote.php/dav/calendars/user1/username2_username1_shared_by_user2/540c319d-3a76-4726-a013-b2b479220051.ics) and then are downloaded and added to the calendar by DAVdroid. Don’t know if those events are the “duplicated” ones or just newly added events, but the whole DAVdroid processing seems correct to me.

    Maybe it’s related to the shared calendars… are you using the latest version of OwnCloud and it’s calendar app?



  • Thanks for your analysis. The entry 540c319d-3a76-4726-a013-b2b479220051.ics you mention is in the database dump only once and appears as

    INSERT INTO "Events" VALUES(3135,'540c319d-3a76-4726-a013-b2b479220051.ics',0,NULL,0,12,'XXX ',NULL,NULL,NULL,NULL,1,0,1506272400000,1506286800000,'Europe/Madrid',NULL,0,0,0,1,0,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,1506286800000,1,0,1,1,'xxx@xxx.net',NULL,0,'Europe/Madrid',NULL,NULL,'540c319d-3a76-4726-a013-b2b479220051','f006d92e71f6a52db9ec59b99089428a',NULL,'0',NULL,NULL,NULL,NULL,NULL,NULL,NULL);
    

    on the contrary, the duplicate entry I described above b133303f-f0d2-4cc9-a89b-cf4a49f5e003 is not among those that appear after a New resources have been found on the server entry in the log file. I don’t know where these New resources come from, but I would conjecture that they have nothing to do with the duplicate entry issue.

    The shared calendar was just an example. The same thing happens also for calendars only synced between the phone, ownclound, and thunderbird/lightning with sogo connector. I am using an up-to-date owncloud. Since the dav backend has been moved into the core I am no longer using the calendar app there.

    I have now also verified that the duplicate entries are only once in the owncloud database, which also explains why they only appear once in thunderbird and on the other phone that syncs the very same shared calendar via a different owncloud account. This is also consistent with the fact that every time I do a re-sync (with the procedure described above) other entries end up appearing twice. As a side remark: I have never seen an entry more than once.

    To me it really looks like a problem between davdroid and the android calendar database.


  • developer

    @cgogolin Well as I see it, DAVdroid doesn’t insert those events, so maybe something else inserts them. This could explain why it’s not reproducible here… Are there any other apps installed which could edit/insert events?

    If you have another suggestion on what I could do, please let me know.



  • I just checked with https://play.google.com/store/apps/details?id=org.androidsoft.app.permission which apps have calendar read or write permissions. These are: DAVdroid, Tasker, and Signal. All three are also installed on the other device that does not show the duplicate entries. Tasker has a few profiles to switch wifi on and off, but nothing with relation to calendar entries. I really don’t understand what is going on… This is one of the issues I wouldn’t even have dared to report, but then I saw that some else had the same problem so I had hope I could help resolve it…
    If you have any suggestions for what I can do, please let me know. I keep you posted in case we observe anything else that might be useful for you.



  • The last DAVdroid update on Google Play states under WHAT’S NEW

    prevent multiple syncs to be run at the same time for the same account and authority (hopefully fixes duplicate entries – if you have duplicate entries, please uninstall DAVdroid and try with 1.9.1 again)

    and indeed, the problems reported above seem to have disappeared after I uninstalled and reinstalled the new version. Thanks for fixing it and making this App even more awesome!


Log in to reply
 

Similar topics