Transaction too large on calendar

[AutoImport-nijel]

When trying to sync calendar, I get following error:

D/davdroid.LocalCollection(14837): Committing 1949 operations
E/JavaBinder(14837): !!! FAILED BINDER TRANSACTION !!!
E/davdroid.DavSyncAdapter(14837): Local storage (content provider) exception
E/davdroid.DavSyncAdapter(14837): at.bitfire.davdroid.resource.LocalStorageException: Couldn't access local content provider
E/davdroid.DavSyncAdapter(14837): 	at at.bitfire.davdroid.resource.LocalCollection.commit(LocalCollection.java:314)
E/davdroid.DavSyncAdapter(14837): 	at at.bitfire.davdroid.syncadapter.SyncManager.pullNew(SyncManager.java:193)
E/davdroid.DavSyncAdapter(14837): 	at at.bitfire.davdroid.syncadapter.SyncManager.synchronize(SyncManager.java:87)
E/davdroid.DavSyncAdapter(14837): 	at at.bitfire.davdroid.syncadapter.DavSyncAdapter.onPerformSync(DavSyncAdapter.java:129)
E/davdroid.DavSyncAdapter(14837): 	at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:259)
E/davdroid.DavSyncAdapter(14837): Caused by: android.os.TransactionTooLargeException
E/davdroid.DavSyncAdapter(14837): 	at android.os.BinderProxy.transact(Native Method)
E/davdroid.DavSyncAdapter(14837): 	at android.content.ContentProviderProxy.applyBatch(ContentProviderNative.java:521)
E/davdroid.DavSyncAdapter(14837): 	at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:377)
E/davdroid.DavSyncAdapter(14837): 	at at.bitfire.davdroid.resource.LocalCollection.commit(LocalCollection.java:311)
E/davdroid.DavSyncAdapter(14837): 	... 4 more
I/davdroid.DavSyncAdapter(14837): Sync complete for com.android.calendar
D/davdroid.DavSyncAdapter(14837): Closing httpClient

The problem here is probably that the event has 1948 attendees, but that’s not something I can control…

[rfc2822]

The problem here is probably that the event has 1948 attendees, but that’s not something I can control…

Indeed, this is a problem. Transactions can only have up to 1 MB (hard limit by Android), and each update must be wrapped into a transaction, so I’ll label this as 3rd-party bug.

[AutoImport-nijel]

Isn’t there possibility to split these huge entries into more transactions?

[rfc2822]

Isn’t there possibility to split these huge entries into more transactions?

No, transactions are used to make database access atomic. They shall ensure that either all or no rows are written for a specific update. It wouldn’t be very useful if, for instance, only 500 attendees would be added and then at the next transaction an error would occur and leave the (already written) event in an unspecified state.

[rfc2822]

Isn’t there possibility to split these huge entries into more transactions?

I don’t like the idea (see above for the reason), but at least the Exchange sync adapter splits transactions as soon as a TransactionTooLargeException is caught.

This would solve issues with too many operations per transaction, but not issues where a single operation is too large (see #230).

[AutoImport-nijel]

I understand that splitting the transaction is not a perfect solution, but I still think that it’s better that completely failing the synchronization what happens right now.

[rfc2822]

I understand that splitting the transaction is not a perfect solution, but I still think that it’s better that completely failing the synchronization what happens right now.

As we all know, transactions are used to ensure consistency (the C from ACID). Splitting transactions might solve some TransactionTooLargeException issues, but it will cause other problems by bringing the database into inconsistent states. I can already see the “Only 1843 of my 1948 attendees are synced” issue

[AutoImport-lmamane]

To write contacts in several transactions (to avoid TransactionTooLarge), but avoid the (or only some of the?) issues with “database in inconsistent state”, such as “Only 1843 of my 1948 attendees are synced”, we could:

1. Use one of the “private SYNC” fields given to us by the contacts database to mark partially synced contacts as such. On these contacts, we redo the full sync, since we don’t know where the sync was interrupted. This assumes the sync procedure is idempotent. I expect it is.

2. Remains the issue of what happens if the user modifies this contact while it is in this “partially-synced” state.

3. If we write back the contact as modified by the user, we have data loss (we throw away some unsynced data from the server).

4. If we overwrite the contact by the server version, we throw away the user’s changes.

5. None of these is ideal, but intuitively, I prefer 2.

6. Actually, this is similar to the (already existing) situation that a contact has been modified both on the server and locally. What does davical do in this situation?

7. If we could “freeze” this contact so that it is not editable (until fully synced), then it would avoid this issue.

8. You see any other issue with this approach?

[rfc2822]

Splitting transactions > 1 MB and implementing meta-database logic is not within the scope of DAVdroid. Mobile devices have still limited resources, and that’s why the transaction buffer is limited.

I’ll close this issue for now, please follow up at https://code.google.com/p/android/issues/detail?id=73499 and comment there if that is an issue for you.

[rfc2822]

Maybe we should really split oversized transactions. The loss of atomicity would be bearable because

1. this would only affect some oversized transactions,
2. possible data loss, i.e. incomplete records (when some of the splitted transactions fail) would only occur for these transactions.

Basically, the trade would be “oversized transactions basically working, but with loss of atomicity and possibly incomplete records in case of an error” vs. “oversized transactions are rejected; no data loss but no chance to synchronize such records”.

I’ll flag this as a possible enhancement.

[groove]

I think that would be a good enhancement. Better to have incomplete records and know you have an meeting then having no record and miss the meeting.

[rfc2822]

This has been implemented using a completely rewritten BatchOperation. All queuing operations had to be changed.

• implementation in ical4android
• implementation in vcard4android
• implementation in DAVdroid

We will release a new version of DAVdroid (1.2.1) as soon as we have finished our tests.