This seems like a meaningless argument - we use fly-by-wire systems all the time and your point is true for most of them. Should we be suspicious of electronic throttle because it could theoretically hit the gas at a crosswalk when you tried to stop?
Should we be suspicious of any automation that can cause death and injury if it doesn't work right? Yes. Absolutely. These systems need to be very carefully vetted. When you don't do that, you end up with cars driving into cyclists [1] and barriers [2], and airplanes driving themselves into the ground [3]. Automation that puts civilian lives in danger is not something where you can "move fast and break things" because the things you end up breaking are innocent human beings.
