Thanks for the feedback @rfc2822! I tried over 4G (previously I was using WiFi) without success, but now I can see that the FastMail return 401 Unauthorized for my user. Alternatively, I tried connecting to caldav.fastmail.com as they described in on their side, but that failed with 401, too.
This looks more like a FastMail issue, or?
Ignore what I wrote. 🙂 I was missing a letter from the FastMail generated app password. My bad! It works now with the https://caldav.fastmail.com/dav/ url.