Those ChatGPT posts where someone gets it to pretend it's a bash shell come to mind, but you are right: this is an incredibly difficult problem domain and you start hitting roadblocks very quickly as soon as you move from cool demo to tool intended for real people to use.
One thing I would add to the conversation is that one of the most potent ways to move this discussion forward is to create technical demonstrations of how this sort of interface could work, presented as video. It's completely unimportant if the functionality is actually working, so long as you disclose this up front.
The goal is to give people with less imagination and hopefully more technical acumen an opportunity to roll up their sleeves and maybe work on making it real.
I was one of the backers of https://www.kickstarter.com/projects/ibdknox/light-table/des... and have wanted to see progress in this domain for a long time.
One thing I would add to the conversation is that one of the most potent ways to move this discussion forward is to create technical demonstrations of how this sort of interface could work, presented as video. It's completely unimportant if the functionality is actually working, so long as you disclose this up front.
The goal is to give people with less imagination and hopefully more technical acumen an opportunity to roll up their sleeves and maybe work on making it real.