I've been working on little temp/humidity/air qual sensors on ESP32, and I've been feeling the MQTT+TLS pain as well. I was thinking I might create a separate wifi network just for these devices, with a rule on the router that only allows MQTT from the sensor network to the main network, and ssh from the main network to the sensor network. Then I can just get rid of TLS. Sure, the MQTT will flow in cleartext from the AP, through the switch, and to the MQTT broker machine, but I think that's fine.
What are your issues with MQTT+TLS in practice? If you can keep the TCP connection open then you shouldn't have too much overhead, should you?
You may want to look into the Matter standard, which already takes security into account and provides some non-certificate based security options. Such devices should integrate well with current and upcoming home automation products from tech giants like Google and Apple. Implementing it from scratch seems very very complicated, but it seems there are premade examples available for ESP32 and other such devices. As an added bonus, Matter can work over other physical protocols such as Bluetooth LE and 802.15.4 + Thread which may take some of the load of your WiFi routing.
