I agree, of sorts. The problem being that we have developed a culture of making user-facing user interface and APIs the same in Unix-y tools. I think "everyone" recognises the problem, but we may have different preferences about what the default should be. Personally, I'd prefer a default way of indicating "I want API behaviour now" but defaulting to more human-friendly output, because even in most cases like "tool | grep xyz" from the command line I'm still more likely to want colours etc. to stay intact through the pipeline. Most of the time that's fine, but you're of course right it will give unexpected results some time.
I think my ideal expected behaviour (which would still not be perfect) would be something like:
* Tools defaulting to unescaped output in scripts, but with a standard short option and/or ENV var to trigger colour output.
* Tools default to colour/escaped output when run in an interactive shell even if in a pipe.
* Tools being escape sequence aware, maybe with switches to turn that behaviour off if you genuinely e.g. do want to grep for sequences that may include escapes and you want them considered.
But I'm not sure there is a good solution to this other than decoupling UI and API and having different defaults for tools that are expected to be "user facing" vs treated as API. I have an "ls" replacement on my system, for example, which changes formatting and adds more colour to my ls output, and it's obviously not named ls because the amount of stuff that breaks if "ls" isn't reliably the same as always is significant. It's still not great to have to separate this given that part of the ease of composing pipelines etc. is familiarity with the output, but maybe if coupling that with reasonably standard switches to turn on/off machine-friendly output.
I can imagine that a world like that could work (with some care to make it less error-prone for common command-line scripting, since command lines can sometimes be small scripts and evolve into large scripts). But I think it's too late to do that for CLI tools without causing widespread breakage.
I think we might be able to do better in something that isn't a traditional shell, and that uses ptys instead of pipes, together with builtins that replace standard UNIX tools with escape-aware tools.
Yeah, I think it at least means abandoning any idea of changing the API of any of the standard tools. Whether you silently replace them in a custom shell or provide alternative named replacement.
I'm halfway tempted to replace my shell with one that is more integrated with my terminal and do something like the last bit you suggested, given it can be very trivial[1] if you explicitly make the choice that for any scripting you'll use a "regular" shell.
[1] there is, in fact, a tiny single-file Ruby shell that I might be tempted to extend.
I think my ideal expected behaviour (which would still not be perfect) would be something like:
* Tools defaulting to unescaped output in scripts, but with a standard short option and/or ENV var to trigger colour output.
* Tools default to colour/escaped output when run in an interactive shell even if in a pipe.
* Tools being escape sequence aware, maybe with switches to turn that behaviour off if you genuinely e.g. do want to grep for sequences that may include escapes and you want them considered.
But I'm not sure there is a good solution to this other than decoupling UI and API and having different defaults for tools that are expected to be "user facing" vs treated as API. I have an "ls" replacement on my system, for example, which changes formatting and adds more colour to my ls output, and it's obviously not named ls because the amount of stuff that breaks if "ls" isn't reliably the same as always is significant. It's still not great to have to separate this given that part of the ease of composing pipelines etc. is familiarity with the output, but maybe if coupling that with reasonably standard switches to turn on/off machine-friendly output.