What I find most interesting is that I'm quite eager and happy to believe that Samsung is spying on us, while you seem to be eager and happy to assert that they wouldn't do such an evil thing.
What's different about you and me? Why do you want to trust Samsung, but I don't?
Just to be clear: I think that currently, these TVs do not record audio and send it over the internet except when voice control is specifically activated (and not just enabled). I'm not saying they don't have that ability, or that they never will do that, or that they're not spying on people in other ways.
To answer your question: If they were, I'm pretty sure we'd have found out by now. I don't think they'd be able to hide the fact that they're sending audio data over the internet. If you connect your TV to a wireless router that you control, you hopefully can't tell exactly what data they're sending to the internet, but you can probably distinguish between "currently sending audio data" and "not currently sending audio data".
(Also, are capped internet plans still a thing? Someone who gets a smart TV and then finds their entire data quota used up in a week would probably notice.)
And it's somewhat plausible to me that this kind of news would break and I'd miss it. But then I'd expect you to be posting articles about how it actually does happen, instead of one which says that they probably aren't listening in on things you say when you're not using the remote, but here are some things you can do just in case.
What's different about you and me? Why do you want to trust Samsung, but I don't?