As a pretty senior engineer, I can tell you that Advent of Code or not, Google-Fu is a big part of your effectiveness in picking up new stuff and delivering code. Adding "Prompt-Fu" that equation seems to be the next logical thing to do.
Completely agree with this sentiment, and the general idea that this is a pretty monumental moment in time. In a couple of hours of work, I've been able to prompt for something like "Get an nginx server and node.js backend up and running", have it generate a rough outline of todo elements, add color/detail to each outlined todo, and write tickets. The tickets even included a title, what done looks like, how to QA, and additional background info as to how this ticket fits into the greater whole. The only thing I didn't get it to do was the actual code writing because I figured this article, or another like it, would prove out it's capabilities for me. Even if the code comes out looking rough like a junior wrote it, the ability to prompt for edits in the PR review should allow us to resolve most issues.
When you feed one level of prompt output into the next using some functional tricks, the ability to layer high-level ideas into the engine become extremely powerful. Even if this isn't 'true' AGI, with a few prompt scaffolding libraries, it won't exactly matter. I'm convinced this is going to radically shake up the entire software field at a process level. Both exciting and scary times ahead.
So writing code in the near future would not be unlike having manage 10s or potentially thousands of folks (ml models/agents) who have taken two semesters of CS. #EternalHacktoberfest
What your thoughts on doing TDD with this, write the unit tests to write the code. But go one step further and have it generate tests from the spec. Repeat until 1=1.
Near future, that seems right, further out I am guessing GPT-4 could enable the level to a couple years of real-world skill instead of two semesters.
With the pipeline as such:
business need -> tasks/cards outlined -> tests written -> code written -> refactor as changes needed
There is still a need for someone to manage the business to task relationship. Let the business person prompt, and the engineer to confirm the prompt output tasks makes sense technically, edit for any errors, and do any modifications needed for specific architectural choices or desired abstractions. With well-enough defined tasks, you can start to write the tests that conform to those tasks. Again, engineer checks the prompt-output, ensuring the tests line up with the card, making any edits as required. With tests written, the code can be written such that it conforms to the tests for correctness, cards for general I/O, and business use-case for domain-specific variables and such.
It's the same domain splits that occur in our current day-to-day practice that are cause for pain. Business person and product person have a miscommunication, the wrong tasks/cards get outlined. The task writer and the test writer have a miscommunication, the tests get written poorly and problem the business is trying to solve gets murky. The tests are written poorly so the code is written poorly. The code is written poorly so the product must be refactored.
The understanding of each others intent must be had and communicated effectively or the downstream problems will mount quickly. It's in this area where GPT-3 still seems a bit lacking, and maybe GPT-4 will resolve the issue a bit. Another worry I have with this sort of model is the spaghetti and debuggability a misunderstanding may have. When code is written more slowly, as is the case today, one is able to mentor the junior and provide feedback over the process. This not only prevents a huge mess, but allows the team to resolve any initially unsaid misunderstandings.
The speed and scale of messes one can now create with this tool is completely massive!
