I have a similar problem, but it’s not a matter of calendar size.
I sync owncloud calendars with a lot of devices, but only one has repetitive issues. In detail I can’t get all of my three calendars when syncing the first time (in most cases). But at the latest after syncing some more times all calendars are gone completely, so in the calendar app the “syncing calendars” window is blank.
The weird thing is that my tablet (same Android 6.0 version) doesn’t have those problems at all. Also none of the other used devices have syncing problems. I’ve also tried several different calendar apps…
Android 6.0 MRA58N
LGE Nexus 5
DAVdroid version 0.9.x
Here is a log file of one first sync: