Agreed. I don't want a smart TV or a media-player box, I want a smartphone/tablet/PC and the ability to easily get its content onto a TV.
And if you're inclined to tinker, the Chromecast is a ton of fun: it's actually running a full browser engine, and you can send it a page to view. That page can then use things like WebRTC streaming.
Two things I would like to put forward in counterpoint:
1. Communal browsing of content - you sometimes want to browse available content on the big screen so everyone can join in. That means menus and UIs on the TV not the device
2. Controls with physical buttons have two advantages. A well-designed remote is much easier to learn and use without needing to look at it. Secondly - games controlled via a touchscreen when you're actually looking at the TV are a fairly unpleasant experience.
I do want a smart TV with a physical controller. I also want Chromecast/Airplay etc. Different beasts.
And if you're inclined to tinker, the Chromecast is a ton of fun: it's actually running a full browser engine, and you can send it a page to view. That page can then use things like WebRTC streaming.