TransactionTooLargeException when syncing contacts with high-res images

[AutoImport-rigid]

Ah, i see. Then I could only imagine davdroid manipulating the blob, trading in image quality of the photo for size (and even removing the photo if someone really uses ~1MB of lossless data in a contact like I mentioned in the update of my previous comment.

[AutoImport-sarumont]

@rfc2822 Have you tested the approach in 3 to confirm it doesn’t respect ContactsContract.CALLER_IS_SYNCADAPTER? I wouldn’t be surprised to see that there’s an underlying Android bug here. I’m happy to dig in and give the approach a(nother) try.

What about using the non-batch insert [ContentProvider.insert()](https://developer.android.com/reference/android/content/ContentResolver.html#insert(android.net.Uri, android.content.ContentValues))? I’m not sure if that goes through the Binder framework as well. Probably does, but who knows…

What sort of workarounds are acceptable? At the minimum, I think logging the contact’s info would be useful here. Even better, I think, would be collecting these exceptions but allowing the sync to finish and throwing something like a partial-sync exception when all Resources have been processed (all which can, at least).

[rfc2822]

@rfc2822 Have you tested the approach in 3 to confirm it doesn’t respect ContactsContract.CALLER_IS_SYNCADAPTER? I wouldn’t be surprised to see that there’s an underlying Android bug here. I’m happy to dig in and give the approach a(nother) try.

I have tried it, but I couldn’t get it working in a reasonable amount of time. I would be very happy if you could try again and maybe get it working. I also guess that this is an Android bug, because CALLER_IS_SYNCADAPTER should be respected.

What about using the non-batch insert ContentProvider.insert()? I’m not sure if that goes through the Binder framework as well. Probably does, but who knows…

The insert is not the problem; CALLER_IS_SYNCADAPTER is respected here. The problem arises from Android’s image processing of the large image file – it processes and resizes the image for the preview which takes some time and then updates the record, marking it dirty again. A correct approach would require to prevent Android from marking the record dirty when it sets the thumbnail blob.

[AutoImport-manwesulimo2004]

Sorry, I don’t have anything constructive to add. Just wanted to let you know I’m affected by this issue too and would love to see it fixed.

And til then: Any suggestions on how to figure out which contact is causing the issue?

[rfc2822]

I had another look into the ContactsProvider2 (Android 4.4.2) which has 8886 lines of code (very funny to read).

As far as I understand it, a high-res photo should be put into an asset using openAssetFile which calls openAssetFileLocal which calls openAssetFileInner which calls (for RawContact photos) openDisplayPhotoForWrite which creates the file descriptor and launches a new, asynchronuous PipeMonitor which will resize the photo using PhotoProcessor and update the database asynchronuously. However, the asynchronuous processing causes the DIRTY flag of the raw contact to be set as soon as the photo is processed because CALLER_IS_SYNCADAPTER is ignored (it’s simply not evaluated anywhere).

So, writing high-res photos using file assets (required for photos > ~0,8 MB) would be impossible without serious hacks and dirty work-arounds.

Do you agree? Just look for the method names in the source code.

[AutoImport-mnalis]

if there was delay long enough after 3b (allowing async processing to finish before 4), would the bug go away?

If so, would it be possible (if “image_quirks” flag was set somewhere in preferences) to sleep a little after contact update in 3b, and before marking contact as not dirty in step 4? Or maybe there is a way to check if android process is finished modifying image, and wait for its completion (or some hard timeout, whatever comes first - just in case) ?

It does look like that is bug in Android, but it is a real life problem

[rfc2822]

if there was delay long enough after 3b (allowing async processing to finish before 4), would the bug go away?

No, the contact would be set to DIRTY as soon as the async processing finishes, causing the contact to be sync’ed again.

[AutoImport-mnalis]

Yes, I understand that. I meant that we sleep in the middle of processing contact until async processing finishes (and contact is (re-)marked DIRTY) and only after that we finish that contact and mark it as CLEAN.

So something like:

davdroid (process1)	image processor (process 2)
contact sync start	(nothing)
start merging data, contact marked as DIRTY	(nothing)
store large picture	Android notices picture update and starts image processor asynchronuously
sleep	picture is being reprocessed
sleep	picture is still being reprocessed
sleep	picture is finished being reprocessed
sleep	contact is re-marked DIRTY and process2 terminates
sleep	(nothing)
contact mark as CLEAN	(nothing)
finish processing contact and move to next one	(nothing)

If that is possible to do, then contact would be CLEAN at the end.
As opposed to current situation which is (if I understand correctly):

davdroid (process1)	image processor (process 2)
contact sync start	(nothing)
start merging data, contact marked as DIRTY	(nothing)
store large picture	Android notices picture update and starts image processor asynchronuously
contact mark as CLEAN	picture is being reprocessed
finish processing contact and move to next one	picture is still being reprocessed
…	picture is finished being reprocessed
…	contact is re-marked DIRTY and process2 terminates

where contact ends up being DIRTY at the end.

Or you mean that async processing doesn’t even start until we mark contact not-dirty and move to next contact?

[AutoImport-acabal]

Looks like I’m running in to this issue as well. Running stock Android 5.1 on a Nexus 6 and Baikal 0.2.7 as the server.

I was trying to use CardDAV-sync, and it would silently not sync some contacts, seemingly at random. Sometimes it wouldn’t even import them initially. So I switched to Davdroid, but initial sync failed with “LocalStorageException: Couldn’t access local content provider”. Trying to sync repeatedly seemed to add more of the contacts that the initial sync missed, but not all of them, and the exception keeps getting thrown.

After scouring the internet I think this issue might be the problem–a lot of my contacts have hi-res photos. But on my Nexus 6 lo-res photos look really crappy, so it’s not a very nice workaround to use lo-res photos.

Is there any progress on this? I assume contacts synced from a Google account manage to have hi-res photos… how does that app get around this issue?

[rfc2822]

Is there any progress on this?

No, as far as I know, there’s still no idea how to fix this.

I assume contacts synced from a Google account manage to have hi-res photos… how does that app get around this issue?

I guess we will never know because these proprietary Google apps are not open-source (which is the reason why I don’t have them on my mobile phone).

By the way, https://code.google.com/p/android/issues/detail?id=73499 has just been closed as “obsolete” without giving any reason or explanation.