Removing SSH should be the goal though. If you follow the old Visible Ops book you also "Electrify the Fence" and introduce accountability, etc. If your goal is to see what a process is doing introduce tracing. If you need to add a "println()" then push that out as a change because the environment is changing from your altering of it. Because the tool doesn't exist yet that you need to SSH into a box doesn't mean it shouldn't - you have to instrument the tooling to prevent you from needing this adhoc ability. Admittedly it scares me still but ideally the end game is to never need to or have the ability to do so through a tool which has all the things you are looking for without allowing a human to be too human and miss a semi-colon.
No, when you actually need to debug in production that's usually not what you want. Changing or restarting the software you are debugging might well make the behaviour you want to understand go away.
> introduce tracing
Yeah, well, that's basically "logging in". Just over less mature and likely less secure protocol than SSH.
You don't need ptrace and tcpdump to debug software. It's just that it can shave a few weeks off your time when you need to reproduce something in the more tricky cases.
These discussions tend to surface in the context of containers but that's all very irrelevant. You need to debug software isn't affected by the way you package it.
You need to be able to troubleshoot things in production though.
Perhaps whenever a developer wants to troubleshoot the orchestrator could make a clone of the container. The clone continuously receives the same input as the original and gets to believe that it affects the back end in the same way. That way the developer can dissect the container without impacting real data.
