TransactionTooLargeException when "Removing non-dirty resources that are not present remotely anymore"

[AutoImport-lmamane]

I’m synchronising an AddressBook that has > 18000 entries (albeit many are duplicate / triplicate / … that are merged on display by Android’s contact app).

At the step “Removing non-dirty resources that are not present remotely anymore”, the synchronisation fails because of TransactionTooLargeException. I first thought that the actual delete was giving a TransactionTooLarge exception, so I wrote the attached patch to make the deletes in smaller steps (one at a time).

However, now, I see that the mere query of list of entries to delete fails (!) with TransactionTooLargeException.

(Obviously the attached patch is not OK for production, it hides / ignores the TransactionTooLargeException and feigns success.)

[AutoImport-lmamane]

It seems that github lets me attach only images… Will upload to some git repo so that you can see.

[AutoImport-lmamane]

Uploaded my debugging change to git://anonscm.debian.org/users/lmamane/davdroid.git

The result with that is that in src/at/bitfire/davdroid/resource/LocalAddressBook.java, member function deleteAllExceptRemoteNames, the try/catch that tries to set cursor with

   providerClient.query(entriesURI(),
				new String[] { entryColumnID() },
				where, null, null);

catches a TransactionTooLargeException

[rfc2822]

The suspect SQL query is built here. If there are 18,000 entries that should be kept, the query will contain 18,000 entries in the source_id NOT IN (1,2,3,4,5,6....,18000) clause. Apparently this is too large for Android.

Maybe you just expect too much from your mobile phone – handling address books with 18,000 entries doesn’t seem to be what Android developers had in mind when setting the hard-coded 1 MB transaction limit.

I’ll close this now because it’s an Android issue, but you may follow up in #274 if you’re interested in discussing about how TransactionTooLargeExceptions should be handled.

[AutoImport-lmamane]

From a naive POV, no, I don’t expect too much from my Android device. It has orders of magnitude more resources (CPU, memory) more than old Psion or less-old PalmPilot PDAs with which I never ran into this kind of issues.

More topically, “other” implementation of CardDAV for Android (albeit non FLOSS) handles this dataset without such problems, so there is no absolute reason that davdroid cannot on the same device. Not sure how DMFS CardDAV-sync handles it. Does it sacrifice some transaction safety (e.g., as you wrote on another bug, the “C” in ACID)? Maybe. But it works.

I’m not familiar with the Android database being here used. But in a “desktop SQL” frame of mind, to avoid making a “too long” SQL string, we could do something like:

sql: CREATE TEMPORARY TABLE remoteIds (ID INTEGER PRIMARY KEY);
java:
    for (Resource remoteResource : remoteResources) {
         // add remoteResource to remoteIds
    }
sql: DELETE FROM tblLocalCollection WHERE SOURCE_ID NOT IN (SELECT ID FROM remoteIDs)
sql: DROP TABLE remoteIds

Or we could add to “tblLocalCollection” a column “lastSync”, and during the sync, for each remoteResource update “lastSync” to the unique ID of the currently running sync, and then do

sql: DELETE FROM tblLocalCollection WHERE SOURCE_ID <> :current_sync_id

Could we do something like that?

[rfc2822]

From a naive POV, no, I don’t expect too much from my Android device. It has orders of magnitude more resources (CPU, memory) more than old Psion or less-old PalmPilot PDAs with which I never ran into this kind of issues.

Ok, but why is there a hard limit of 1 MB for transactions? I think it was feasible for early Android hardware but it hasn’t been updated.

Could we do something like that?

No, the database is hidden behind the ContactsProvider API. However, the process could be split up into a) detecting which entries should be deleted (by building the set difference in DAVdroid, not in the database) and b) deleting it (batch-wise).

I’m thinking about refactoring the sync. collections logic anyway, so let’s have a look.

[AutoImport-lmamane]

A way to avoid the issue about entirely on “good modern servers” would be to implement Collection Synchronization for WebDAV.

[AutoImport-lmamane]

Something like http://anonscm.debian.org/cgit/users/lmamane/davdroid.git/?h=issue305 ? Need to do the same with LocalCalendar, obviously.

[AutoImport-lmamane]

Any comment about my patch? I use it in production, seems to behave OK.

[AutoImport-untitaker]

I refiled the original Android bugreport since Google seems to have autoclosed it.

https://code.google.com/p/android/issues/detail?id=182292

[AutoImport-virl]

@untitaker just stumbled upon this discussion. Android is such a crap.

1MB shared limit for all IPC communication of an app? Is it a joke?

Starred your bug report.

[AutoImport-virl]

@untitaker Also let’s write this bug description to author of Binder — Dianne Kyra Hackborn, who works at Google:
hackbod@angryredplanet.com
https://plus.google.com/105051985738280261832/about

[AutoImport-untitaker]

No, absolutely not. That crosses a line.

[AutoImport-virl]

@untitaker why? She is author of Binder subsystem. Google doesn’t fix Android bugs FOR YEARS.

[AutoImport-untitaker]

Because that’s harassment.

[AutoImport-virl]

@untitaker reporting a bug to the author of subsystem is harassment?

[AutoImport-untitaker]

If that author is not solely responsible for maintaining it, yes.

[AutoImport-virl]

@untitaker who is responsible then? Previous Issue about that bug on Google Code was closed as Obsolete.

[AutoImport-untitaker]

Do not contact people about their work at a private address. Show some respect.

[AutoImport-virl]

@untitaker Okay, who I must contact for this bug to be fixed?
My app randomly crashes because of it.