I just installed DAVdroid version 1.7.2-ose from F-Droid on OnePlus 5 (OxygenOS 4.5.10 / Android 7.1.1, security patch level 1 July 2017) and tried to add an account with a non-default port in the URL. The server Apache is running ownCloud, the URL I used for configuring DAVdroid was like this: https://server.name.here.invalid:12345/ocloud/remote.php/carddav/
The server is running a generally non-trusted cert, so I was expecting a cert dialog, which I got. But the certificate shown in the dialog is not from the server software running on the specified 12345 port. Instead the default certificate used on port 443 is shown. When I looked at the network traffic, I initially see traffic to port 12345 and right after I see traffic to 443.
My guess is that the port 12345 is dropped at some point from the URL actually used. Possibly when the suspect URL/host is passed to a cert manager when it is tasked to show the user the untrusted certificate for acceptance.
In any case there should not be traffic to other ports when I manually specify a port in the URL. The server httpd logs do not show any http queries, so the requests are apparently aborted by DAVdroid before http is even started inside TLS, which is appropriate, given the cert is still untrusted at that point.