If an SRV record is not found, the client will need to prompt the user to enter the FQDN and port number information directly or use some other heuristic, for example, using the extracted “domain” as the FQDN and default HTTPS or HTTP port numbers. In this situation, clients MUST first attempt an HTTP connection with TLS.
But this is part of step 2. Determination of service FQDN and port number.
But we are at step 3. Determination of initial “context path”.
and there it is:
When an initial “context path” has not been determined from a TXT record, the initial “context path” is taken to be “/.well-known/caldav” (for CalDAV) or “/.well-known/carddav” (for CardDAV).
Which is the situation we have: No SRV and no TXT record.
I find it overly strict to read this in a way, that a vaild SRV record is a precondition to trying to use wll-known paths.