And it’s hard to see any benefits of them collecting all this data. If I remember correctly, there was a rumor that their telemetry told them that nobody used the Start Menu, leading to the Windows 8 debacle! Have they pushed any new features lately that have been directly attributed to data collection?
Almost everything? Insider Blogs continue to make it pretty clear that in the Windows 10 "Windows as a Service" world, almost everything is subject to telemetry based development/focus/resourcing. Insider Builds themselves often have A/B testing experiments in play.
Even things like the giant Console rewrite and WSL apparently owe just as much to telemetry meriting their resources as the (return) of Azure and DevDiv's "developer-first focus" to Windows (after all these years).
(A fun irony of the "Start Menu debacle" to which you refer was more of a sample size problem than a telemetry problem. With Windows Vista-8 keeping telemetry primarily opt-in, they mostly only had interesting subsets of newbie/casual users that clicked yes during setup and power users with vested telemetry interests. Newbie/casual users tend to launch apps "Search only" or "Desktop only" keeping all their shortcuts on the desktop. Power Users tend to use alternate launchers or become increasingly "Search only" over time, especially if you count Win+R old timers, which Windows did. Microsoft had a good argument that trying to make the telemetry opt-out would avoid such biased sample sets going forward.)
I'm used to this on web and mobile - because arguably their apps are their private property which I'm using, but there's something incredibly uncomfortable about my operating system running an experiment on me. I get that Windows is also Microsoft's property, but it's the lowest level in the stack, which makes my hardware more theirs than mine if they're able to do what they want with it.
I'm not even against telemetry as long as it's transparent.
So far as I know, the A/B experiments haven't happened in "RTM" (or "RTW" as more accurate) builds, only Insider builds (which are opt-in). The Insider blogs have also been generally transparent in which tests were performed, at least after the fact, but big ones were also announced well in advance to avoid confusing users.
The biggest/wildest example that was A/B tested to date was Sets. Sets was a feature/system where nearly every window in Windows was heterogeneous tab-capable. (Illustrative example: Open a Word document, add a tab for some OneNote notes, add a browser tab for some research. Save it all as a Set to the Windows Timeline as Set, open it all back up at once.) But ultimately Sets never made the cut out of testing (too many compatibility issues, not enough telemetry usage, too confusing a UX sometimes, not enough apps in the wild with Windows Timeline support, etc) and it sounds like the project is now dead.
(Supposedly it was partly dropped too because apparently they were piggy backing on Edge's UWP tab control and with Edge going "Edgmium" that tab control was considered less critical infrastructure to any team and less of an "in-box" control to piggy back off of. Microsoft Terminal got a version of the UWP tab control pushed out into the open source UWP control toolkit, but probably not in time to be reconsidered as an in-box control for Windows.)
(I was sad I never ended up in the group with Sets turned on. I wanted to play with it.)
(ETA: Also, I really hope as Edgmium gets closer to launch it uses a XAML Island to pick back up the UWP tab control where possible. That would be one great way to feel a bit more Edge and a bit less Chrome knock-off.)
Microsoft wants to be like Facebook, where they can make each and every pixel on screen a part of an A/B test, and gather ineffable amounts of telemetry based on that information, which they can then magically turn into amounts of money that would embarrass even God.
They fail to understand that "Be Like Facebook" is not a positive aphorism or outcome.
