TransactionTooLargeException when syncing contacts with high-res images

[AutoImport-rigid]

I’m syncing vcards that include large photos and I get:

E/davdroid.DavSyncAdapter(26828): Local storage (content provider) exception
E/davdroid.DavSyncAdapter(26828): at.bitfire.davdroid.resource.LocalStorageException: Couldn't access local content provider
E/davdroid.DavSyncAdapter(26828): 	at at.bitfire.davdroid.resource.LocalCollection.commit(LocalCollection.java:235)
E/davdroid.DavSyncAdapter(26828): 	at at.bitfire.davdroid.syncadapter.SyncManager.pullChanged(SyncManager.java:206)
E/davdroid.DavSyncAdapter(26828): 	at at.bitfire.davdroid.syncadapter.SyncManager.synchronize(SyncManager.java:90)
E/davdroid.DavSyncAdapter(26828): 	at at.bitfire.davdroid.syncadapter.DavSyncAdapter.onPerformSync(DavSyncAdapter.java:101)
E/davdroid.DavSyncAdapter(26828): 	at android.content.AbstractThreadedSyncAdapter$SyncThread.run(AbstractThreadedSyncAdapter.java:259)
E/davdroid.DavSyncAdapter(26828): Caused by: android.os.TransactionTooLargeException
E/davdroid.DavSyncAdapter(26828): 	at android.os.BinderProxy.transact(Native Method)
E/davdroid.DavSyncAdapter(26828): 	at android.content.ContentProviderProxy.applyBatch(ContentProviderNative.java:517)
E/davdroid.DavSyncAdapter(26828): 	at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:377)
E/davdroid.DavSyncAdapter(26828): 	at at.bitfire.davdroid.resource.LocalCollection.commit(LocalCollection.java:232)
E/davdroid.DavSyncAdapter(26828): 	... 4 more
D/SyncManager( 2360): failed sync operation CardDAV u0 (bitfire.at.davdroid), com.android.contacts, USER, latestRunTime 114674148, reason: 1000, SyncResult: databaseError: true stats []
D/SyncManager( 2360): not retrying sync operation because the error is a hard error: CardDAV u0 (bitfire.at.davdroid), com.android.contacts, USER, latestRunTime 114696545, reason: 1000
D/AccountTypeManager(25361): Registering external account type=bitfire.at.davdroid, packageName=at.bitfire.davdroid

I’m not 100% sure that this is caused by a large photo. I’m using baikal as server and CardDavMate as web frontend (where the photos came from).

[rfc2822]

I guess it was. At the moment, I can only suggest to use smaller photos. It’s very complicated to write the photos to a file (actually, it’s easy, but then Android resizes the photo, which will mark the contact that has just been updated as “dirty”, which will cause a re-sync, which causes an endless loop), so at the moment, DAVdroid only uses the database BLOB to store the photo. When a transaction (with the INSERT/UPDATE of the binary BLOB) is > 1 MB, a TransactionTooLargeException will be thrown.

If you have suggestions on how to fix this, please post them here.

[AutoImport-rigid]

It would be nice, if there would be a way to identify the faulty contact. (If a notification with detailed error-message isn’t possible, at least using adb logcat would be helpful). Then the error could at least be worked around manually.
In addition to that, I don’t understand the “endless loop issue”. As I understand, the photo that is resized by android would be synced back to the server which just stores the resized version. If the server won’t touch the photo again, the circle would end here, wouldn’t it?
I suppose, the 1MB limit can’t be adjusted anywhere, can it?

Update: Personally I don’t think storing the photo outside of the database would be a clean solution (if that was what you meant with “write the photos to a file”). It maybe would be, if one could configure it (e.g. choose between limited blob size and external files). Davdroid could scale the image down in case the blob is too large, marks it as dirty and sync it back to the server.
Wouldn’t the problem also arise when a vcard grows in general? (even without photo). E.g. A vcard that is 0.999MB large, stored in the database and then gets edited (e.g. a long note text added) so it exceeds the 1MB limit. The only solution I could imagine would be, to trade in image size/quality of the photo then.

[rfc2822]

In addition to that, I don’t understand the “endless loop issue”. As I understand, the photo that is resized by android would be synced back to the server which just stores the resized version. If the server won’t touch the photo again, the circle would end here, wouldn’t it?

1. The high-res photo >1MB is downloaded from the server and parsed by ez-vcard, which returns a byte[].
2. This large byte[] can’t be stored into the contact’s thumbnail BLOB (transaction too large).
3. So, the photo must be written as a file using the ContactsContract.RawContacts.DisplayPhoto CONTENT_DIRECTORY.
   3a. In order to write this file, we need a raw contact ID before, so the INSERT has to be done before and
   3b. in a second pass, the photo file for this raw contact ID will be written.
4. Sync is now complete, the contact’s DIRTY flag is removed.

But: 3b is not blocking, so Android modifies the photo in the background and after a few seconds, when step 4 is already finished, it writes the modified (resized etc.) photo for the contact and marks the contact as dirty again.

So, at the next sync, the contact is dirty and it will all start from 1…

Normally, there’s a “IsSyncAdater” that can be set and that should prevent marking records as DIRTY when changing them, but this doesn’t seem to work for the photo-writing class.

I suppose, the 1MB limit can’t be adjusted anywhere, can it?

Exactly As far as I have understood it, the limit is hard-coded.

[AutoImport-rigid]

Ah, i see. Then I could only imagine davdroid manipulating the blob, trading in image quality of the photo for size (and even removing the photo if someone really uses ~1MB of lossless data in a contact like I mentioned in the update of my previous comment.

[AutoImport-sarumont]

@rfc2822 Have you tested the approach in 3 to confirm it doesn’t respect ContactsContract.CALLER_IS_SYNCADAPTER? I wouldn’t be surprised to see that there’s an underlying Android bug here. I’m happy to dig in and give the approach a(nother) try.

What about using the non-batch insert [ContentProvider.insert()](https://developer.android.com/reference/android/content/ContentResolver.html#insert(android.net.Uri, android.content.ContentValues))? I’m not sure if that goes through the Binder framework as well. Probably does, but who knows…

What sort of workarounds are acceptable? At the minimum, I think logging the contact’s info would be useful here. Even better, I think, would be collecting these exceptions but allowing the sync to finish and throwing something like a partial-sync exception when all Resources have been processed (all which can, at least).

[rfc2822]

@rfc2822 Have you tested the approach in 3 to confirm it doesn’t respect ContactsContract.CALLER_IS_SYNCADAPTER? I wouldn’t be surprised to see that there’s an underlying Android bug here. I’m happy to dig in and give the approach a(nother) try.

I have tried it, but I couldn’t get it working in a reasonable amount of time. I would be very happy if you could try again and maybe get it working. I also guess that this is an Android bug, because CALLER_IS_SYNCADAPTER should be respected.

What about using the non-batch insert ContentProvider.insert()? I’m not sure if that goes through the Binder framework as well. Probably does, but who knows…

The insert is not the problem; CALLER_IS_SYNCADAPTER is respected here. The problem arises from Android’s image processing of the large image file – it processes and resizes the image for the preview which takes some time and then updates the record, marking it dirty again. A correct approach would require to prevent Android from marking the record dirty when it sets the thumbnail blob.

[AutoImport-manwesulimo2004]

Sorry, I don’t have anything constructive to add. Just wanted to let you know I’m affected by this issue too and would love to see it fixed.

And til then: Any suggestions on how to figure out which contact is causing the issue?

[rfc2822]

I had another look into the ContactsProvider2 (Android 4.4.2) which has 8886 lines of code (very funny to read).

As far as I understand it, a high-res photo should be put into an asset using openAssetFile which calls openAssetFileLocal which calls openAssetFileInner which calls (for RawContact photos) openDisplayPhotoForWrite which creates the file descriptor and launches a new, asynchronuous PipeMonitor which will resize the photo using PhotoProcessor and update the database asynchronuously. However, the asynchronuous processing causes the DIRTY flag of the raw contact to be set as soon as the photo is processed because CALLER_IS_SYNCADAPTER is ignored (it’s simply not evaluated anywhere).

So, writing high-res photos using file assets (required for photos > ~0,8 MB) would be impossible without serious hacks and dirty work-arounds.

Do you agree? Just look for the method names in the source code.

[AutoImport-mnalis]

if there was delay long enough after 3b (allowing async processing to finish before 4), would the bug go away?

If so, would it be possible (if “image_quirks” flag was set somewhere in preferences) to sleep a little after contact update in 3b, and before marking contact as not dirty in step 4? Or maybe there is a way to check if android process is finished modifying image, and wait for its completion (or some hard timeout, whatever comes first - just in case) ?

It does look like that is bug in Android, but it is a real life problem

[rfc2822]

if there was delay long enough after 3b (allowing async processing to finish before 4), would the bug go away?

No, the contact would be set to DIRTY as soon as the async processing finishes, causing the contact to be sync’ed again.

[AutoImport-mnalis]

Yes, I understand that. I meant that we sleep in the middle of processing contact until async processing finishes (and contact is (re-)marked DIRTY) and only after that we finish that contact and mark it as CLEAN.

So something like:

davdroid (process1)	image processor (process 2)
contact sync start	(nothing)
start merging data, contact marked as DIRTY	(nothing)
store large picture	Android notices picture update and starts image processor asynchronuously
sleep	picture is being reprocessed
sleep	picture is still being reprocessed
sleep	picture is finished being reprocessed
sleep	contact is re-marked DIRTY and process2 terminates
sleep	(nothing)
contact mark as CLEAN	(nothing)
finish processing contact and move to next one	(nothing)

If that is possible to do, then contact would be CLEAN at the end.
As opposed to current situation which is (if I understand correctly):

davdroid (process1)	image processor (process 2)
contact sync start	(nothing)
start merging data, contact marked as DIRTY	(nothing)
store large picture	Android notices picture update and starts image processor asynchronuously
contact mark as CLEAN	picture is being reprocessed
finish processing contact and move to next one	picture is still being reprocessed
…	picture is finished being reprocessed
…	contact is re-marked DIRTY and process2 terminates

where contact ends up being DIRTY at the end.

Or you mean that async processing doesn’t even start until we mark contact not-dirty and move to next contact?

[AutoImport-acabal]

Looks like I’m running in to this issue as well. Running stock Android 5.1 on a Nexus 6 and Baikal 0.2.7 as the server.

I was trying to use CardDAV-sync, and it would silently not sync some contacts, seemingly at random. Sometimes it wouldn’t even import them initially. So I switched to Davdroid, but initial sync failed with “LocalStorageException: Couldn’t access local content provider”. Trying to sync repeatedly seemed to add more of the contacts that the initial sync missed, but not all of them, and the exception keeps getting thrown.

After scouring the internet I think this issue might be the problem–a lot of my contacts have hi-res photos. But on my Nexus 6 lo-res photos look really crappy, so it’s not a very nice workaround to use lo-res photos.

Is there any progress on this? I assume contacts synced from a Google account manage to have hi-res photos… how does that app get around this issue?

[rfc2822]

Is there any progress on this?

No, as far as I know, there’s still no idea how to fix this.

I assume contacts synced from a Google account manage to have hi-res photos… how does that app get around this issue?

I guess we will never know because these proprietary Google apps are not open-source (which is the reason why I don’t have them on my mobile phone).

By the way, https://code.google.com/p/android/issues/detail?id=73499 has just been closed as “obsolete” without giving any reason or explanation.