Seems like your user name / password is incorrect.
Did you try as described on https://www.davdroid.com/tested-with/fastmail/?
Did you use your full email address (without special characters like " " from copy/paste) as user name?
Did you use the generated app password (without special characters like " " from copy/paste) as password?
Is there any other way how we can help you? This is most probably not a DAVdroid problem.
Thanks for the feedback @rfc2822! I tried over 4G (previously I was using WiFi) without success, but now I can see that the FastMail return 401 Unauthorized for my user. Alternatively, I tried connecting to caldav.fastmail.com as they described in on their side, but that failed with 401, too.