One of the problems with DNSSEC is that it causes the size of DNS responses to increase substantially, which becomes a DDoS vector. The response to this is nominally to use response rate limiting, but the more people who use DNSSEC the less effective that is because it means more DNS servers the attacker can use to reflect their attack through without reaching the rate limit for any one. (RRL is also not ideal because a large DNS cache can hit the rate limit by just making a large number of legitimate queries.) And the more servers that use DNSSEC the harder to block a DDoS attack because it comes from a larger number of unique addresses.
And now TXT records used for domain validation are becoming quite large as well. You add a ~hundred byte TXT record for each service you want to validate your domain against, but they all want the entry to go in the same place (at the root), so you do the DNS TXT query for the root and you get a big response. Adding DNSSEC on top of that might even blow the practical limit for UDP EDNS0 responses sometimes, never mind the DDoS potential.
There may be a solution for that one -- specify a location for each TXT record instead of everyone using the root, so Facebook uses _facebook-domain-verification.[example.com], Google uses _google-site-verification.[example.com], etc. Then you wouldn't get a dozen large TXT records in response to a single query because they're each at a different name. But that's not currently what companies are doing, and a purist wouldn't like it because in principle controlling _google-site-verification.example.com doesn't mean you control all of example.com.
For at least some of this, TCP could be a decent solution. For CAA queries, for example, no one really cares how fast the query is, so a server could plausibly refuse to answer over UDP.
That works for the thing expecting the large response, but requires an unusual custom configuration on the server (refuse UDP queries for specific record types but not others), and doesn't really help other clients that may make queries that yield large responses unexpectedly.
For example, the large TXT records can prevent mail delivery from some versions of qmail, including the one currently packaged with Debian stable, because it makes an ANY query for the domain (to avoid separate queries for MX, A and CNAME) but only supports UDP and only supports responses up to the 512 octets specified in RFC1035. Then it gets a truncated response when there are >512 octets of TXT records and considers it a name resolution failure.
Making the ANY query is an unusual quirk in qmail, but it could have just as easily been an actual TXT query looking for the SPF record etc.
And now TXT records used for domain validation are becoming quite large as well. You add a ~hundred byte TXT record for each service you want to validate your domain against, but they all want the entry to go in the same place (at the root), so you do the DNS TXT query for the root and you get a big response. Adding DNSSEC on top of that might even blow the practical limit for UDP EDNS0 responses sometimes, never mind the DDoS potential.
There may be a solution for that one -- specify a location for each TXT record instead of everyone using the root, so Facebook uses _facebook-domain-verification.[example.com], Google uses _google-site-verification.[example.com], etc. Then you wouldn't get a dozen large TXT records in response to a single query because they're each at a different name. But that's not currently what companies are doing, and a purist wouldn't like it because in principle controlling _google-site-verification.example.com doesn't mean you control all of example.com.