> The idea that some inexperienced people are shipping software using this tool is insane to me.
I really think you're trying to compare apples and oranges, in multiple ways. For one, we can test the software by running it, which is a pretty different problem from asking language questions, with a much slower ability to verify correctness (based on what you describe and what I imagine).
I'm not saying your experience is invalid. In my own adventures, the equivalent of what you did was my writing some incomplete bash in an existing script, wandered off to another part of the code. I then came back to that incomplete snippet, and though it was some unfamiliar syntax someone else had written (vim even highlighted it like it was special!). Naturally I went and asked ChatGPT what that snippet did, and wasted 15 minutes trying to corroborate it before checking the git history or something and realizing my own error.
As long as the tests are not also written by ChatGPT...
Many critical security issues require a deep understanding or the code or some intense fuzzing to discover, it's not enough to ask ChatGPT "write me X" then superficially glance at the output to validate that it looks correct. That's the part that worries me. Completely broken code will be caught immediately, but subtly broken code may linger for a long time and make it to production.
And from my limited experience with ChatGPT, it seems very good at making up broken things that look superficially correct.
I really think you're trying to compare apples and oranges, in multiple ways. For one, we can test the software by running it, which is a pretty different problem from asking language questions, with a much slower ability to verify correctness (based on what you describe and what I imagine).
I'm not saying your experience is invalid. In my own adventures, the equivalent of what you did was my writing some incomplete bash in an existing script, wandered off to another part of the code. I then came back to that incomplete snippet, and though it was some unfamiliar syntax someone else had written (vim even highlighted it like it was special!). Naturally I went and asked ChatGPT what that snippet did, and wasted 15 minutes trying to corroborate it before checking the git history or something and realizing my own error.