I finally pulled IPv6 resolvers out of my config for this very reason. I was getting 4+ second lookups against both Comcast and Google v6 resolvers even though direct v6 sockets worked instantly; made worse because HSTS apparently makes Chrome put "establishing secure connection" in the status bar while resolving (maybe?), so I ended up down a rabbit hole of identifying reasons for phantom TLS stalls. Finally cut down to just v4 resolvers and it's a brand new computer.
That is pretty much what I saw. I was somewhat worried if it was the AT&T modem config or OSX, but nothing I tried fixed it (including explicitly setting DNS sources and so on).
