Musl also includes res_send() etc., which can be used for MX records, and e.g. Qmail did use that back then. Here's the commit adding TCP support to those functions in Musl.
AOL was the specific case we ran into that at the time (ca. 2000) returned either MX or A records (can't remember which caused the problem) that required more than 512 bytes.
The point wasn't that this would have helped Qmail - Musl didn't exist back then for that matter -, but that running into the 512 byte limit and needing TCP fallback was already a thing. Bumping a buffer size is far less effort than having to replace the resolver.
gethostbyname() only returns an address, it can't be used to e.g. query for MX records. You also lose control over retries to different addresses if you use gethostbyname(), which some mail server software will also care about.
However as I noted in my other reply, musl also implements the res_* functions, like res_send() etc., and those can be used to address both.
