Being efficient at writing code can barely make someone a 5x engineer. A team of people who write code effectively is easy to do.
Hyperproductivity in software is all about deciding what problems to tackle. Richard Hipp isn't worried about chipotle restaurant orders in golang, he's worried about how to store data reliably. That isn't a coding puzzle that ChatGTP is likely to help with. Either ChatGTP can do the whole thing itself (not yet the case) or it is a minor productivity boost because the hard part is articulating the problem.
Writing code quickly really isn't a challenge that high performing software engineers need to tackle. ChatGTP is a cool tool, we're all going to be using things like it in a few years, it'll change everything. But it won't make any old engineer a rival to the big names in software engineering.
I briefly worked with a software engineer who was making about a million dollars a year.
I mentioned that an approach would be a lot of work. His response:
“It’s just typing. Typey typey type (makes fingers on keyboard gesture).” Absolute lightbulb moment for me.
To become a senior engineer is to realize that the code writing part is inconsequential as long as there are no technical unknowns in the work.
It’s freed me to try out huge sweeping changes just for fun and generally detach from my code. Sometimes I delete large, intricate changes so quickly and ruthlessly that my colleagues have to ask me to reopen them.
We aren’t paid to write working code. We are paid to grow harmonious systems. Working code is table stakes.
Yeah, I'm really wondering if I'm dumb or something with this whole AI train, but my job as a developer that maintains existing (sometimes quite old) systems is much more about determining what actually needs to be done, gathering lost / not lost knowledge about the existing state of things in systems and figuring out where to change the code for maximum impact and minimal side effects. Lots of talking to customers/non-technical people as well to figure out if existing features can cover something they need and they just don't know about it and I happen to know about it etc.
I've been watching the AI hype (again) and it's completely going over my head, I really really don't get it - I doubt {x}GPT would be able to analyze hundreds of thousands of project code across a big system (multiple languages, multiple services, interconnectedness of everything) and tell me what to do, even in the case I would want to paste in everything proprietary to a 3rd party service which I really don't.
Maybe it just works for a subset of programming, but then again I don't really see how it differs to reading docs for generating boilerplate or whatever it's useful for in greenfield projects. I was also not really impressed by it generating improvements to code when you ask it the right questions since that's the hard part of the job, not typing out the code.
Don't downplay yourself like that. A 25k token GPT model is a step in that direction but we're 3 - 15 years from your vision of valuable.
Currently, the Language Models (GPT or GithubCopilot) are mostly good as ... copilots. You bounce off ideas off them or use them to kickstart something you want to do. It's great for junior devs (me!) but it still cannot do big context or cognition (what you describe), that part will be done by humans for quite a while.
I often suffer from impersonator syndrome, my freelance rates should be North if 70 per hour but I keep getting stuck at 40...
I grew up poor though and teaching myself coding moved be to earn more then I thought I'd ever earn without a college degree.
I knew about for loops almost a decade before I decided web dev would be my career, not just my hobby.
it's insane that you could get hired and not know how they work.
linked lists, sure. I still don't fully understand those lol, I was going for an associate's and that was in my c++ class but I've mostly only touched PHP, python, ruby, JavaScript... playing around with go and rust some as well, but I've never had to really think about linked lists, loops on the other hand is a daily thing.
hell, not knowing loops is like not knowing how to increment, like seriously, his to add 1 to an int.
Even not-so-great-sounding contracts pay 50-65/hr, might be a confidence boost. Level up your skills on udemy, if necessary (wait for courses to be 80% off or whatever, happens regularly).
Could also go corporate at a non-tech for low stress and a decent salary.
I'm actually thinking of pivoting from web dev (10 years a laravel dev) to prompt engineering, and entry-level ai stuff, I'm not going to perfect the next big algorithm, but I can use langchain to create custom KB enhanced chatGPT instances, or automation agents for small-to-mid size companies.
I'm super obsessed lately w/ AI, on youtube, github, HN, reddit, etc... and working on actually building a clone of myself to do the mundane parts I don't like to do... hehe... this is kind of a green field, and the opportunities are huge, might be the last dev jobs when AGI hits -- training custom AGI's...
I make over $1m as a SWE. “Senior Staff” or “Level 7” role at a big tech company everyone knows.
I mostly write code! I am way faster than most others here, and I have encouraged and mentored the people on my team to be similarly efficient.
I also help decide technical direction but it’s far from the “architecture astronaut” stereotype that writes no code.
It’s an infra team so no significant influence on product direction, other than considering our frameworks and tools products. I do consider them in that way, but it’s still a big difference from being involved in the direction of the external product.
Probably -5 actually, I got promoted to this level right around when tech stocks started to dip. However, I’ve since had some retention bonus grants that will make it even more absurd if the stocks do go back up.
Timing gets a 10/10 for my $750k year at staff level though.
Congrats on your career success! I'm surprised infra can pay that well; I suppose at some companies they understand infra basically ensures the ship keeps running as it should.
A lot of it is pretty related to company internals and is tooling specific. At a large company there is a lot of infra and tooling that can break in all kinds of fun ways, quality is very mixed, so helping the team become experts in this is very useful for productivity and unblocking oneself.
I see people on other teams that seem to be stuck for days sometimes, on a really basic error that they haven’t really tried to debug at all. Seriously, just read the error message or dig into the code!
My team also has a strong culture of just pushing code changes. This is including tons of deletions and simplifications, it’s not just writing a million lines of code.
It’s also a very senior team, so I can’t take credit for everyone’s growth from junior engineers. Another part of it is creating an environment that productive senior coders want to join, so that they can avoid process-heavy teams and get stuff done.
There is an extreme level of trust on the team because everyone is senior and frankly extremely good at their job, which allows us to be even more productive.
There's a couple where I work. They are highly specialized in a very specific area of CS, with decades of experience, and with a vast working knowledge of the specific internal framework that they've helped build over the course of ~15-20 years.
>> We aren’t paid to write working code. We are paid to grow harmonious systems. Working code is table stakes.
With all due respect, you may have learned the wrong lesson — or missed some important context — and the above is a false dichotomy. Working code isn't just table stakes; it is virtually impossible to "grow harmonious systems" if you don't have solid working code, because without solid working code you cannot grow the system reliably.
The other thing to note is that just because someone makes a lot of money doesn't necessarily mean that they are good at their job, or that they are a good team player, or that the way they operate and the context they operate in applies to you. I know consultants who pull in a million or more every year, and you know how they do it? They write absolute shit software that, while fulfilling the contract requirements, ends up hamstringing the client down the line. Sometimes prioritizing short-term gains might be acceptable or even necessary (e.g. the auditor asks you to compile data from a bunch of sources and you need to do it fast), but for projects with longer life spans one should definitely not blindly "typey typey type".
Table stakes is a poker term meaning you can’t even play the game without it. There’s no dichotomy here.
I’d go further and say that in big tech, well factored well tested code is table stakes and the ability to produce it quickly is assumed. Junior engineers have to prove that to get their first promotion.
If that’s all they show, it will be their last promotion, and that’s perfectly fine too.
I think this is very accurate to my experience in big tech. As a person who's only gotten that "first promotion" so far, I'd be interested to hear your summaries of what attributes / skills are looked for at successive levels: e.g. what engineers need to prove to get promoted to Senior, Staff, Senior Staff, etc.
I'm aware that I'm in the minority here, but I find that investing heavily in being able to type fast and handle the tools well actually changes what code I write and not just how quickly.
The faster I've gotten at moving code around, the more subconsciously willing I've been to try something with a modest probability of working out, and more importantly the easier it's gotten to throw away a draft that isn't working out and try again.
LLMs haven't made it into my coding workflow yet, but I can see how they could be useful and the trend seems to be that they'll be in most high-octane workflows at some point.
I think this is true but I also found limiting returns on going faster and at some point your just confident enough to try out if it is a good idea. It also becomes easier to justify, you become more trusted and you evaluate better.
Maybe that makes everyone hit that point easier, maybe it brings that point in for you. Not sure.
I used to attend cross-team meetings with a guy (works for AWS now) who was a tech lead and meeting facilitator. He gathered the agenda, presented during the meeting, assembled notes during the meeting, and distributed the notes via email at meeting conclusion. He keyboarded the entire time, looking up at the projected display. Fastest touch typist I recall seeing.
Yes and no. Most of my stuff is automated, auto completion and micro code gen. My thinking is the bottleneck, never typing speed. Measured it last summer a few times trying to beat my daughter and I think it was about 60wpm. For coding if I know what I'm doing its probably 30wpm in my editor.
Yeah, these days, IDEs like JetBrains are powerful enough that you can pull off significant chunks of refactors with a few right clicks (any sort of renaming, moving of methods, or deletion of code)
Typing will get you places faster, but some of the most productive tools we have don’t rely on it at all.
Define “efficient at writing code”. I find that for many people this means something primitive and absurd like typing faster or remembering things better.
Being efficient in the perspective of writing new automation so you don’t have to do the same manual things again can easily make someone a 10x (or greater) engineer. This typically means writing your own tools and not hoping some third party package or framework will do it for you.
Conversely the same is true. If a 10x engineer is in an environment where they are prohibited from writing original tools they will just be average or much worse when their motivation is destroyed.
I agree with the compounding impact of self-automation.
Also, some people just keep growing their creative outlook. They are not afraid to try solving bigger & bigger problems.
I worked with an engineer (individual contributor) at a company with a lot of related software products.
Time and again, he would hibernate then come out with a new well developed tool or product that either impacted the entire product line or started a new product line.
Once the value of what he had done was obvious, and his clever design had stabilized in a form understandable to others, he would hand it off to a team of great developers to polish, release, update etc.
10x impact would be an understatement.
If we measure impact, it is easier to see where 1x, 5x, 10x, or 100x can be achievable and verifiable.
Also x/2, x/10 or (and I have seen it) -x. One team leader, -5x at least, until they were removed.
Yeah a traditional 10x engineer would know how to create/use leverage to enable much more work to be done faster in this case creating a great tool, reminds me about a car mechanic how he showed me this weird tool and said it was one of the most useful tool he’s fashioned
We should not forget that different people have different talents. That creative superstar guy may have been great at coming up with new stuff but terrible at maintaining stuff. Or at debugging some urgent prod issue under time stress on a saturday evening. You need those people too that enjoy the day-to-day maintenance to keep the house clean, or the occasional firefighting. A good team contains all those types and they complement each other.
In this case, they were great maintaining stuff too. They remained available to dip into that work whenever it helped.
But as a rule, it would not have been challenging enough to keep them engaged for long. And it would have been a waste of their talents for themselves and the organization.
It wouldn't just have wasted their time directly, but wasted the opportunity for less (but still A-grade) developers from diving into a new area and maximizing its value. There was still lots of creative work to be done. These were not trivial new areas.
That's true, and I'd lean on the "much worse" side. When you join a project that has very minimal tooling and almost no automation, you're facing a dilemma: should I focus on improving the tooling while delaying features or should I deliver features while making future improvements ever harder to make?
The problem with this is that neither is good for your mental health. Unless you're joining that project with a specific goal of improving its development process, don't bother. Run away. Fast.
Especially if there are already other devs that are militantly against your take on dev process improvements. The level of emotional distress is hard to fully appreciate without experiencing a situation like this. Suggesting the simplest and most obvious things becomes a brutal fight that you're guaranteed to lose. Living without them makes you miserable in its own right. You're trapped between a rock and a hard place with no way out.
If out of various kinds of psychotherapy you pick the one based on an experience of living in a Nazi concentration camp as the most applicable - it's time to run. Really. Don't look back.
The other thing is making better choices of what to make, how to make it, and most importantly things not to do. That can often save 9x from most anything.
Yeah, being able to negotiate with the PM to still solve the problem in a easier way than initially proposed is probably where I've had the most impact over the years.
I discussed my experience using ChatGPT in HN before and I was accused of being skeptical. But it isn't true at all, I think LLMs are a wonderful tool that will change software development. However I'm not worried about losing my job.
As you say, I find the act of writing code is hardly the bottleneck to my productivity. One of the best case scenarios, LLMs become the ultimate high level language. In that case, Software Engineer roles would switch from coding to more of a mix of old school Software Architect, PM and entrepreneur.
By the way, I find interesting to run the argument of "coding is not the bottleneck" in reverse. What if you design an organization so that coding is the bottleneck? How transformative would LLMs be?
> As you say, I find the act of writing code is hardly the bottleneck to my productivity.
When people claim ChatGPT can replace software engineers because it can sort of code is always puzzling to me. 20 years into this career, and writing code is only a minor part of my day-to-day activities.
A part I do enjoy doing, but still minor part nonetheless.
> What if you design an organization so that coding is the bottleneck? How transformative would LLMs be?
The funny thing is that although ChatGPT has been an amazing coding assistant tool, it can't meaningfully write code beyond boilerplate to me. I use it more as search engine on steroids, rather than actually relying on the code it writes.
A few times it even got into a hallucination loop going through different flavors of uncompilable code when I asked "how to do X using library Y", and refining the question was getting nowhere.
ChatGPT is an amazing tool that provides me value beyond the 20 bucks per month I pay. But still has far too many limitations. It's sort of funny to see people LARPing that it will kill knowledge jobs in the near term.
What amazes me when I talk to people that work in "non-tech" companies is how common boilerplate work is actually out there. Man-millennia of effort to move data from one system to another, or generate a report, or trigger something. The first AI MuleSoft or similar platform is going to kill it.
Design Patterns are said to compensate for features missing in a language (e.g. Visitor for multiple dispatch). ChatGPT is sounding similar - solving accidental complexity that "shouldn't" be there in the first place.
No doubt chatGPT can't replace coder's yet. If current pace of improvement continues (a big if!), it defnly will be able to in a couple of years. Honestly I expect it to happen.
Though I don't expect software dev job to go away. It'll just change in nature. Less typey typey, more managing the LLM.
To me it looks like the rate of change has already slowed having played around with gpt2 and used copilot a fair amount, chatgpt, and now chatgpt with gpt4.
There's also not orders of magnitude more existing data we can feed it anymore.
I mean saying that using ChatGPT will make you one of the most elite coders in the world is a pretty high bar no?
I have been skeptical about all this, but decided to play with using ChatGPT for coding a task I had. It wasn't perfect, either in workflow or in final results but what was clear was that not adopting it will very quickly make you lag behind your peers.
I think the additive bonus is especially high for senior devs who are fast, in that those will be faster in grokking the generated code and better at guiding it to the right solution.
I guarantee you that in five years anybody who cares at all about their productivity will be using it constantly in their workflow, probably much sooner.
The tooling will be pretty interesting.. basically prompts per file and a overlaying diff per prompt..
Focus mostly on the almost functional design of software, with well defined input and output as the most important criteria. Also lots of "meta" tags, that filter the trained in network layers for "fast" code, "secure" code, "minimal working smaple" code by injecting addtional prompts. The trick is to use the prompts as a selection menue for the data variations stored within the Network.
I imagine that a web service could be specified in one file. At the top, a description of the service, including the language it is to be written in and how it will be built and deployed. Then an outline of the endpoints and their functionality.
A builder bot could then ask a series of questions and get permissions such as “can we use X package?”, and then update the specification before performing the build.
Chatgpt is an amazing learning tool, currently it helped me understand scripting very quickly and expanded what I was never would thought to do.
I know I could have used google but the way google presented me is not easy to find and digest, you have ads, you have article that is not exactly what you are looking for and the code examples are not exact. Gpt eliminates all that time and effort. It almost eliminates laziness
You need to use it now because it works, not perfect but it reached the 80/20 for enough things, and you still need to do the last 20. Even if not, it will get you some gains if you use it when it fits
I agree with your sentiment. I've generally advanced in my career by being good at deciding what problems to solve/not solve.
The problem is many, many people are not in a position to advance that way. I've worked in startups where you're almost always given an ambiguous problem and rewarded for meaningful outcomes.
Many people are working as part of team/projects where the general problem and approach are already determined for them.
Oddly, I am most productive when I have time to be slow. I make 10x better decisions about what to do, how to do it, and in many cases why NOT to do it.
This isn’t very conducive to corporate work though. It’s about timelines and features promised to customers.
I find I’m 10x more productive in the long term on projects I’m a sole developer on or when working with a small knit team. And I don’t measure that in lines of code or number of features.
> This isn’t very conducive to corporate work though. It’s about timelines and features promised to customers. [...] I wonder if ChatGPT can help there.
Obviously not, it will just make expected windows shorter and shorter.
"This would take a C developer a month, but you use Java so 2 weeks should suffice"
"This would take a Java developer two weeks, but you use Python so a week should suffice"
"This would take a Python developer a week, but you use ChatGPT so I'll expect it sometime tomorrow."
> Either ChatGTP can do the whole thing itself (not yet the case) or it is a minor productivity boost because the hard part is articulating the problem.
What a ridiculous dichotomy, or course there's plenty of room in between. The hard part might be articulating the problem, but the other part takes time. Make that part not take time and you can do more of the hard part.
Unfortunately, LLMs are still prone to make facts up, and very persuasively. In fact, most of non-trivial topics I tried have required double-/triple- checking, so it's sometimes not really productive to use chatGPT.
You are correct that I made an error in my previous response.
I apologize for the confusion I may have caused in my previous response
I appreciate you bringing this to my attention.
I apologize, thank you for your attention to detail!
I asked it to explain how to use a certain Vue feature the other day which wasn't working as I hoped. It explained incorrectly, and when I drilled down, it started using React syntax disguised with Vue keywords. I definitely could have tried harder to get it to figure out what was going on, but it kept repeating its mistakes even when I pointed them out explicitly.
Part of why I don't use ChatGPT very much for work is that I don't want to feed significant amounts of proprietary code into it. Could be the one thing that actually gets me in trouble at work, seems risky regardless. How is it you're comfortable with doing so? (Not asking in a judgmental way, just curious. I would like to have a LLM assistant that understood my whole codebase, because I'm stumped on a bug today.)
I'm not doing it right now, I'm more imagining a near-term product designed for this (maybe even with the option to self-host). Current LLMs probably couldn't hold enough context to analyze a whole codebase anyway, just one file at a time (which could still be useful, but)
The GP is undermining their own point by even mentioning “5x, opening the door to think about this in linear terms. But the concept of a “10x engineer” was never about how fast they produce code, and the multiplier was never a fixed number. The point was that some engineers can choose the right problems and solve them in a way which the majority would never achieve even with unlimited time.
As an example, if you took away the top half of the engineers on my current team and gave the rest state of the art Copilot (or equivalent best in breed) from 2030, you would not end up with equivalent productivity. What would happen is they would paint themselves into a corner of technical debt faster until they couldn’t write any more code (assisted or unassisted) without creating more bugs than they were solving with each change.
That doesn’t mean the improved tooling isn’t important, but it’s more akin to being a touch typist, or using a modern debugger. It improves your workflow but it doesn’t make you a better software engineer.
Maybe in 2030 the AI will be able to respond to that situation appropriately, like it won't just layer more sticky plasters on top with each additional requirement and make a mess but will re-evaluate the entire history of instructions and rearchitect/refactor the code completely if necessary?
And all this with documentation explaining what it did, and optimising the code for human readability, so that even with huge reworks you can still get the gist in a time measured in hours rather than it taking, what, weeks to do manually?
I mean, looking at the math that powers these models I don't see how they can replace reasoning. The tokens mean absolutely nothing to the algorithm. It doesn't know how algebra works and if you prompt ChatGPT to propose a new theorem based on some axioms it will produce something that sounds like a theorem...
... but believing it is a theorem would be similar to believing that horoscopes can predict the future as well.
Maybe some day we'll have a model that can be trained to reason as humans do and can do mathematics on its own... we've been talking about that possibility for decades in the automated theorem proving space. However it seems that this is a tough nut to crack.
Training LLMs already takes quite a lot of compute resources and energy. Maybe we will have to wait until we have fusion energy and can afford to cool entire data centers dedicated to training these reasoning models as new theorems are postulated and proofs added.
... or we could simply do it ourselves. The energy inputs for humans compared to output is pretty good and affordable.
However having an LLM that also has facilities to interact with an automated theorem proving system would be a handy tool indeed. There are plenty of times in formal proofs where we want to elide the proof of a theorem we want to use because it's obvious and proving it would be tedious and not make the proof you're writing any more elegant; a future reasoning model that could understand the proof goals and use tactics to solve would be a nice tool indeed.
However I think we're still a long way from that. No reason to get hyped about it.
Maybe the prompt engineer will learn from the conversation and remove their dead end questions. Every time you ask a question gpt is just re running it all includinG the previous context. If you start over with a trimmed conversation it's like the player was never there.
Churning out boilerplate is a large part of the job. The quality of that boilerplate is part of what makes great engineers great engineers.
Knowing when and how to use dependency injection, making code testable, what to instrument with logs and stats, how and when to build extensible interfaces, avoiding common pitfalls like race conditions, bad time handling, when to add support for retries, speculative execution, etc. are part of parcel of what we do.
If ChatGPT can help raise the quality of work while also increasing its quantity, that'll be a huge leap in productivity.
It's not all there yet, but I've been using it to write some simple programs that I can hand out to ops / business people to help automate or validate some of their tasks to ensure the smooth rollout of a feature I've developed.
The resulting ChatGPT code--I chose Go because I can hand over a big fat self contained binary--avoids certain subtle pitfalls.
For example, the Go HTTP client requires that you call Close() on a response body, but only if there's no error.
The code it spat out does indeed do that correctly. And it's well documented.
It's far from perfect; I've seen a subtle mistake or two in my playing around with it.
I'm now no longer so much an author but an editor for basic stuff. I now have my own dedicated engineer with a quality level that oscillates wildly between fresh intern and grizzled veteran.
If churning out boilerplate is what your employers sees as product, does it really matter? Coding udeals are valuable in a vacuum but most companies want those PRs merged, technical debt be damned.
> A team of people who write code effectively is easy to do.
Based on my experience, you’d be surprised how hard it is sometimes. I have a team of 10 engineers that failed to ship a basic CRUD UI after several months (and it wasn’t even close to done). Eventually it got taken over by one individual eng who finished it in two weeks.
I've seen this happen several times before, and it's never been because people weren't typing fast enough: it's always bad or nonexistent management, unclear accountabilities, or lack of technical direction or oversight (one failed CRUD project was written purely functionally by an engineer in Scala + Scalaz, absolute insanity). None of these are solvable by LLMs.
The problem isn't generating code. It's articulating a functional specification with enough precision and clarity that you can solve for the unknown, which is the program that satisfies the specification.
I agree with the conclusion that tools like ChatGPT won't be replacing programmers any time soon. And it won't be helping us to write programs until it can properly reason about the information it is trained on and prompted with. The problem with natural language is that it is often too imprecise. When you get to concurrency or optimizing data layout an LLM isn't going to understand anything about the problem space and the training data probably won't contain enough examples to produce something remotely plausible let alone good or elegant.
Writing, thinking, and asking the right questions will still be important for quite some time. We're not in the age of having a magical genie in our computers that can do all the work for us... yet.
Although using it as a procedural generation tool for fake data is nice... still not as nice as writing a property test with generators and shrink... but as a little time saver for quick prototyping, neat.
Uh no, ChatGPT is good as being a better (contextual) search engine, a rubber duck that explain code to you (which stil bad but keep improving), and a mid-level code generator junkies that can generate crud operation, unit tests or transpile code between language.
All of it will reduce mental load for senior devs and enable them to tackle more problems or more difficult problems.
> a mid-level code generator junkies that can generate crud operation, unit tests or transpile code between language.
Not really though, at least not yet. I've seen some impressive results for trivial cases: the type of thing you'd see blogged on new web devs blog 10x over, but I've yet to see anything that wasn't trivial and worked come out of GPT.
I think reaching peak optimization as an engineer is a mix of:
- Knowing or having control of the tools you use
- Knowing what problems are most important to revenue and which are the greatest risk
- Knowing the language well enough to plan ahead without much labor
When a company I worked at mandated an IDE it took me a while to be productive beyond typing speed. When I discovered my company had support for generators, I was no longer spending forever on boilerplate REST code. When I know my test harnesses well it's easier for me to prove functionality in a smaller feedback loop.
> But it won't make any old engineer a rival to the big names in software engineering.
This stood out, so I looked it up, and it seems Richard Hipp is 61 years old. I guess it is more about the "any" than the "old". Or maybe old engineers are engineers doing it the "old way" (pre-chatgpt)?
"any old x" is an idiom that references "any random" or "any", indicating x being interchangeable or not special in any way. It has nothing to do with age.
>>Either ChatGTP can do the whole thing itself (not yet the case) or it is a minor productivity boost because the hard part is articulating the problem.
Telling any AI system what to do will be the biggest problem. A lot of toy like algorithm problems can be easily stated. Stuff like inverting binary trees, sorting, and your myriad different Leetcode stuff is the easiest thing to do here.
Real world software has if/else statements peppered all over large code bases/domain logic. Lots of IO exceptions, lots of domain specific logic, all of this is nearly impossible to dictate in words. And even more you will eventually need some concrete way of describing things, which in all honesty will look a lot similar to code itself if you have to get it right.
Modifying code will likely be even more harder to describe in talk than in text.
Over my career, the higher up the pay grade I went, the less actual code I was writing and the more it was figuring out what to do, what's priority, how the existing code all fits together, evaluating risk, and how to make the right surgical incisions to accomplish the goals.
Glorious and quite rare is the moment when a problem is clearly written up by a PM and the codebase is greenfield or clear enough that just pumping out code is happening and taking up my day.
Most senior software engineers are less programmer and more analyst.
I think the headline was being tongue in cheek. It probably generated more clicks than “5 clever examples of using ChatGPT via CLI for programming and data science”.
The most direct example of this to me is in video games, there are 10s of thousands of failed games with amazing art and coding behind them, but then weekend projects like Flappy bird, Vampire Survivors and Wordle make it big. They just were lucky enough to find the right angle, a team of 100 PHDs working 10X as long would find less success.
Maybe I'm misunderstanding your last sentence - but if churning out boilerplate is something any old engineer can now have an AI do, won't it up the chances that someone previously overlooked makes a brilliant call about what problem to tackle and actually has the resources to execute on that call?
I'm doubtful. Being effective as dev usually involves a conscious effort to focus on the context of where we are, where we're going, and what's needed to get there. Also, the ability to recognize and avoid rabbit holes.
I'll add that programmers don't produce code. They produce business guarantees. And their main tool to do so are:
* Understanding the value proposition as a human, often as through a conversation; often with hidden criteria that you'll have to uncover
* Making sure the code respects the value proposition. As reading other people's code takes more time than writing it, ChatGPT won't help that much here.
----
One development to this story I'm anticipating for though is connecting ChatGPT to formal proof systems, and making sure it has sufficient coverage with 'internal thoughts' that are logic-verified (by opposition to killing the problem big-data-style). Sort of like the internal monologue that we have sometimes.
I say short and medium term we're sort of ok; but I don't know about 10 years from now.
Chain of thought reasoning is being explored with some success. It’s better for it to be explicit output instead of internal, remember that the next token is being predicted based on the previous ones.
Your two points may not be as sound as you think. The second one can be defeated now since ChatGPT is quite good at generating descriptions of what code is doing. The first point is more subtle, you can iterate on code with ChatGPT but right now you need to know how to run the code to get the results before providing feedback. Once tools are developed for ChatGPT to do that itself then it could just show the results and a non technical person could ask for follow up changes.
Usually the person with the ability to create the best products and services is the one with the deepest reservoir of background knowledge and aesthetic taste that they can apply to a given problem.
So, for example, someone who can productively spend 100 hours working on a blog post is probably going to get win out against someone who can only productively spend 5 hours working a blog post on the same topic.
And given a society where 99% of the rewards go to 1st place, 1% of the rewards go to 2nd place, and 0% of the rewards go to everyone else, having this ability is usually more valuable than the ability to pump out a little more work in some fixed amount of time.
Unfortunately the person who only spends 1 hour working on a blog post, but can spend 100 hours cranking out 100 blog posts, wins out. See Simon Willison, for example.
You're not wrong, but it's ultimately the same background and skillset as what I'm talking about that enables that also. E.g. Simon Willison could easily sit down and write a bestselling book if he wanted to.
Iteration time is important but part of that is knowing and learning what to iterate. Therefore there will still be a human in the loop, and by Amdahl's law a 30% speed up will never produce more than a 30% performance improvement.
Wow. 210+ comments and maaybe 10 of those actually talk about the article!
(yes, I've read them all because I can't believe almost everyone missed the point)
The point of the article is actually neat: a small shell (Fish) script to send query to ChatGPT or DALL-E and get response back in the terminal. As a terminal user myself, this is really nice! (I rewrote it for myself in bash which I use, but used the curl command virtually as is)
The post itself is hilariously over the top (1000x c'mon), quoting 50 Cent and including statements like:
> Unfortunately, due to inflation — real and imagined, 10X just won’t cut it anymore. We need bigger gains, bigger wins, more code, more PRs, more lines, less linting, etc...
Either people really need to be pointed out the sarcasm these days, or everyone's just philosophising about the title (I'm willing to bet it's the latter).
I asked Bing chat to summarize the article and suggest a neutral headline based on the content -
Headline: Here’s a neutral headline for the article: “Improving Programmer Productivity with Command Line Wrapper Functions around OpenAI API” .
Summary: The article at https://kadekillary.work/posts/1000x-eng/ discusses how to improve programmer productivity using command line wrapper functions around the OpenAI API. The author provides examples of functions for generating text and images, and for making code edits. The article is written by Kade Killary and was published on their website on March 29th, 2023
> (yes, I've read them all because I can't believe almost everyone missed the point)
That says almost everything there is to say about the article in 1 comment; and even you've sent most of your time discussing the title and meta-issues.
People aren't missing the point, it just happens that programmer productivity is more interesting (hence the link bait title, I suspect). The article stands on its own without comment.
I couldn't figure out how to replicate the gpt_data one, how did you replicate the "string join" part? I'm not exactly sure what it was supposed to be putting in the json. I mean, obviously the full data in the second parameter, but I'm unsure how to encode that..
What I did instead is just join all the args with spaces (so I don't need to use quotes around the prompt), with no care about the newlines, and it seems to work okay for me with multiline input:
Since I can use this with multiline input I didn't create a separate data-gpt one.
(Note I also used gpt-3.5-turbo model and not gpt-4 because the former is faster and cheaper and for quick one-offs from cmdline it's usually enough for me).
Writing code is the basic 101 of our industry. Anyone can write a loop or an if statement. With todays computing power, for most applications even poorly written code will run and will execute quite fast. What makes someone a 10x (and i hate that term) is the ability to take requirements, or better yet, a vague idea given to them by the business and turn that into software that benefits the end user and makes the company money. The 10x engineer gets to the 'why' that you are building something, not the 'how'.
RE>> Anyone can write a loop or an if statement. <-- Respectfully disagree.
I had an intern a few years ago -- recent college grad in CompSci. I tried my best to lightly mentor him. One day I was talking about the diff between a compiled language and scripting, mentioned REPL. To demonstrate REPL, I opened up both the windows CMD prompt and the Chrome Developer tools. I mentioned that with a REPL like the Chrome Tools, it's trivial to do FizzBuzz in JavaScript. I explained the problem to him and asked him to take a stab at writing it. This wasn't an interview question, just a discussion and a mentoring opportunity.
He couldn't. What he said next blew me away, coming from a CompSci graduate "Oh, loops, yeah, I never quite understood those. Like, for loops and while loops - I never really got that". I asked if he meant recursion, cause that can be tricky. No - he really could not write a for loop in any language. I wasn't going to shame him and I walked him through it, but I was disappoint. ಠ_ಠ
[Edit: Spelling]
Edit 2...Before I start claiming that CompSci programs are letting students down, I have to consider that the claim that he had a BS in CompSci may have not been accurate. I did not check or verify his transcript. The more I reflect on it, I think he may have had a degree is Web Design and we got pressured to add him to our Software Eng. team because the hiring manager (and his actual mentor outside of work) passed him off as a "Web Developer". Now that I think about it... that seems more likely....Edit 3: It was driving me crazy so I dug up the resume in my inbox ... it was def CompSci, listing C++, C#, Java, and SQL as technologies and data structures and algorithms as courses taken... I'm not sure what to think...
They didn't. I was one of two technical interviewers. We both recommended on passing [not hiring]. It was only a seasonal internship, so we weren't super grilling, but we still recommended a pass. We were overruled by the hiring manager who mentored this person outside of of the office.
I mean, you do need to know the "how" too. It isn't unimportant, it's just a more common skill. You need to be a 1x engineer before you can be a 10x engineer (if you believe in them).
At what point did we stop making fun of that 10x engineer thread and started taking it seriously? Who forgot that 10x was a silly little thing that made zero sense in the real world?
As far as I know the 10x number was made popular by Fred Brooks, not a meme, and it has pretty real applications:
> Programming managers have long recognized wide productivity variations between good programmers and poor ones. But the actual measured magnitudes have astounded all of us. In one of their studies, Sackman, Erikson, and Grant were measuring perfor-
mances of a group of experienced programmers. Within just this group the ratios between best and worst performances averaged
about 10:1 on productivity measurements and an amazing 5:1 on program speed and space measurements! In short the $20,000/year programmer may well be 10 times as productive as the
$10,000/year one.
He goes on to explain that because the primary cost in software development is the overhead of coordinating minds, a company is far better off paying for just a few really good programmers than they are 10 times that number of mediocre ones. Whether or not there's such a thing as a 10x programmer in isolation, this seems pretty self-evident and valuable to keep in mind.
> He goes on to explain that because the primary cost in software development is the overhead of coordinating minds, a company is far better off paying for just a few really good programmers than they are 10 times that number of mediocre ones.
If that’s really the primary cost, surely one of the metrics they studied variation in is the coordination cost imposed by the programmer on the company, they didn’t just naively assume the cost was the same but output different, right?
Because it seems like if it does vary, you’d want to minimize it by hiring the easiest to coordinate, and that might offset output differences in sone cases.
There are obvious examples of 10x performers in every creative endeavor. Why would it be any different in software engineering? There are people who find bugs 10x faster than others, design stuff that's 100x easier to maintain, increase morale of a team vs being an energy drain, make changes faster without breaking stuff, etc. There are a lot of ways one can become 10x more effective than another at building software.
There are examples, yes, but I took it as the implication that a 10x engineer shouldn't be the bare minimum for the entire workforce's ability to reason about a problem or dedication to its craft (and thus lead to the meme that one isn't worth spent effort unless they are in the >=10x group).
That’s the nature of any long-tailed distribution.
The lie is that anyone can become a 10x [insert thing here] just by trying harder. Self-improvement is a real thing, but everyone can’t be 10x or 6-sigma or world class.
Strive, reach, sure… but ultimately, just make sure you live your life. You only get one, and the clock’s ticking.
I often think of the Gene Hackman line in Heist: "I try to imagine someone a little smarter than myself, and I try to imagine what he would do."
It's not a level playing field, there are people who are just way more efficient and effective than the norm. Whether it's drive, focus, creativity, or knowledge (usually a combination of them), there are people who solve more problems and solve them better than average.
I don't know the literal number, but some folks are just better at it than others.
Perhaps you are just in a bubble and haven't met one?
I was at a conference once talking to a recruiter, when he stopped mid sentence and literally ran after one of these 10x engineers that had walked past.
There is some truth to the idea and most skills have a hierarchy/normal distribution/metrics/quality ranking that reveals partial insight into one's output.
John Carmack is definitely 10x or 100x (or 1000x) compared to me on low-level graphics programming, just factually, not that it says much about either of us.
But the '10x programmer' idea is soaking up and refueling some fanaticism. Let's all double down on the concept of high performance, for no particular reason we can agree on.
I have had coworkers that were 10x more valuable than the other coworkers. If you haven't then your company probably only employs people that are incredibly talented or incredibly mundane
You may have high performers (and low performers), including people that perform very well in a particular context or even help the whole team performing better. There is no question on this.
Putting an actual individual factor on it and especially on engineer productivity, like we were at a line factory producing everything in a repeatable and measurable way is more questionable. 1x, 10x, 1000x? People here are discussing 10x becoming 2x, it sounds very mathematical. Now if there is an actual scale with an industry baseline and an individual measurement, I'm more than interested to learn about it and I think employers should fairly compensate engineer based on the scale factor you would put on your CV.
I worked with a guy who was reputed as a 10x engineer.
After about 6 months on his team it became apparent that he was very good at communicating to his management, very good at writing excess complexity into his code, and very good at preventing the developers he was leading from taking on large or impactful projects.
My takeaway was that he was more of a "turn-everyone-else-into-1/10X" developer. I jumped onto a different team as soon as the opportunity came up (as did most of the folks who worked with him).
Maybe the 10x engineer isn’t true, but I’ve worked at companies where I was on a team and my output was equal to the rest of the team combined. So if there isn’t such a thing it must mean that there are 1/10X engineers and they’re prevalent.
If you're on a team of 10 people, 3 of those people will be doing about half the work. And then 7 are doing the other half. If there are 10 work units, the 3 are doing 5/3 of a work unit each and the 7 are doing 5/7 of a work unit each for a ratio of 2.3x
---
Yes, 1/10th developers are common - especially in larger teams. And there are also -1/2 developers too where someone will spend half their time just cleaning up after the -1/2 developer.
And just with the nature of teams, larger teams are going to have a significant fraction of the team doing very little work (note: this applies to consultancy teams too and if you get a team of 20 people to work on the project, you really should just get a team of 4 and stay on top of them as it will be less work to manage).
The “Mythical Man Month” book mentions something about “10X” or something, not necessarily in those terms. And people think that it’s a classic book (for whatever reason).
> We've known since the early sixties, but have never come to grips with the implications that there are net negative producing programmers (NNPPs) on almost all projects, who insert enough spoilage to exceed the value of their production. So, it is important to make the bold statement: Taking a poor performer off the team can often be more productive than adding a good one. [6, p. 208] Although important, it is difficult to deal with the NNPP. Most development managers do not handle negative aspects of their programming staff well. This paper discusses how to recognize NNPPs, and remedial actions necessary for project success.
> Researchers have found between a low of 5 to 1 to a high of 100 to 1 ratios in programmer performance. This means that programmers at the same level, with similar backgrounds and comparable salaries, might take 1 to 100 weeks to complete the same tasks. [21, p. 8]
> The ratio of programmer performance that repeatedly appeared in the studies investigated by Bill Curtis in the July/August 1990 issue of American Programmer was 22 to 1. This was both for source lines of code produced and for debugging times - which includes both defect detection rate and defect removal efficiency. [5, pp. 4 - 6] The NNPP also produces a higher instance of defects in the work product. Figure 1 shows the consequences of the NNPPs.
[5] : Curtis, Bill, "Managing the Real Leverage in Software Productivity and Quality", American Programmer July/August 1990
[6] : DeMarco, Tom Controlling Software Projects: Management, Measurement & Estimation (New York: Yourdon Press, 1982)
[21] : Shneiderman, Ben Software Psychology: Human Factors in Computer and Information Systems (Cambridge, MA: Winthrop, 1980)
Can you link to that thread? I'm having difficulty understanding how programmer productivity is supposed to be measured and compared. Can't imagine what a "10X engineer" is supposed to be. The article says a "1000X engineer" is just a person with a GPT-4 command on their terminal.
It seemed ridiculous on the face of it, but with time people started to find kernels of insight in the idea.
The idea itself has a certain sultry allure. That probably helped keep it wedged in the back of the mind long enough to find those kernels, rather than being immediately forgotten.
> First off, you’ll need an OpenAI API key. You can get this by going here and signing up.
Note that you'll also need to be from a Western country, or at least have access to both a phone number from one and a VPN. And if you want a plus subscription, you'll also need a US credit card - not stated anywhere but I've been totally unable to sign up for plus with either Irish or Belgium cards, it just immediately flashes "card declined" every time with no further info. There's about a 0.1s delay between clicking the pay button and getting that message so it's not querying my bank.
Why are you trying to pay for it? Let procurement at your company figure it out. You wouldn’t pay for your IDE or any other tools, would you? The business should be paying for any tools they want me to use.
Have you ever tried getting a company to fund a developer tool? More often than not, it's like pulling teeth, even if the service in question is a paltry $20 a month.
I've learned to pay for things like OpenAI myself because it's not worth the hassle to get a company to pay for it (for the purpose of R&D).
I’ve been I’m working professionally in the industry for 23 years. I’ve never had a company refuse to purchase any tool I say I want or need. I wouldn’t work for any company that would. My hourly rate is way higher than the cost of any tool. It is ridiculous to not buy tools that might make me more efficient.
When that tool is a 3rd party which takes proprietary code as an input surely you can understand pushback from big software, financial services etc companies?
Yes, I would - these tools are massive time savers, and if I suspect it'll soon become a situation where if you don't use these tools, you're going to get left behind.
If my org wouldn't pay for them, I would start looking for other jobs (unless your in an industry that can't use them like national security, etc), but I'm not going to let $30/mo stop me from using a tool that significantly improves my work just because of a middle manager.
Also, many programmers I know bring their own keyboards.. Chefs often bring their own knives, or mechanics with their own tool chests. Ownership of your own tools is powerful.
Plenty of people work as contractors/consultants where payroll pays them as a separate company. In fact you are probably much more likely to have something closer to a FAANG income working as contractor rather than as an employee.
High level engineers solve the right problems quickly and put everything together in a resilient fashion. GPT4 may help them do that more quickly but they probably already will use copilot and templates and past project material and various generators for anything tedious.
10x engineering comes from someone who is trying to "be a better developer" actively for many years. Most people who just use the tools to create the product won't ever become that much better. They will just become more knowledgeable. It is the same as any natural skill. You can play piano for years and never really get better. Or you can try specifically to do hard work of improving your technique, theory, and performance ability directly. It will be way more emotionally taxing to do so, but the end result after years of doing this is that you're vastly better than most. Not everyone can even do this. You definitely have to be in a growth mindset and not particularly stressed emotionally at the outset and during the process.
My coworker at Amazon was just making macros nonstop for all these things he wanted to automate within his workflow. He would focus specifically on the hardest problems that we haven't solved first and everything else around it would just spring up together rapidly (I don't even know how he did this). He ended up being heavily involved with AWS and later went to Netflix.
These were things like keybinds and shortcuts within his editor (which 10 years ago this was Eclipse at Amazon). There were other scripts he had but I do not really recall at this point what they were doing. If you google macros for software engineers you can probably find some useful stuff.
I've found ChatGPT very useful as a "desk buddy" who I can turn to and ask about obscure things. It's quickly replacing my Google/StackOverflow in my workflow.
I was working on a .pkg MacOS installer and could not figure out how to get my background image @2x.png to render on a retina display. Turns out I needed to create a directory with the .imageset extension. My attempts at searching for a solution online never led me to this, but ChatGPT immediately knew what to do when I described my problem.
It's also tremendously useful as an interactive manpage. I've found it's much faster to ask ChatGPT how a complicated CLI tool works than trying to read the dense --help output to find what I need.
It can even have a conversation about A* pathfinding, and how best to implement it with different nuances. It's crazy.
I wonder how employers will handle this trend, which is happening so quickly. Is it OK for an employee to get advice on their work from ChatGPT? Is it OK to commit AI-generated code? Is it OK to show internal source code to an AI to help them find a bug or refactor it? There are security implications here, and ethical questions. I think most sophisticated companies will only adopt this kind of tech when they can self-host it, or at least have some kind of guarantee that their employees are using a dedicated, isolated model.
With that said, I'm sure many people are already making heavy use of this for their work (as are students). It's already an out-of-control phenomenon.
What the articles like this miss is that we understand pretty well how ChatGPT writes code. But what about maintenance, and particularly debugging? How would it make these tasks. And we know that these things take much much more time than writing code.
I find ChatGPT is pretty helpful for debugging. A big chunk of debugging is missing stupid things, and for debugging, it's like having a second set of eyes.
It's almost useless for high-level design and architecture. Those tasks do take more time than writing code, but they're super-charged by rapid prototyping, which ChatGPT excels at. For example, right now, if I'm picking between three libraries, I can have working versions of my code implemented in all three in a few minutes, and built out a bigger mockup in a few hours. I can make a more educated decision from there. In the olden days, that'd be a few hours and many days, respectively.
It also does really nicely for code reviews, which results in better code. That also speeds up maintenance and, later, debugging. That's especially true in domains I'm less familiar with.
Some maintenance, it excels at. I'm the author of a major piece of open source infrastructure, which is widely used, well-architected, but every technology used is obsolete (JQuery, older Python web frameworks, legacy database, etc.). I feel like I could rewrite the whole thing in React+Node+etc. in a few weeks. There is close to zero architecture work -- it's mostly having GPT write -- or sometimes just translate -- code piece-by-piece.
I don't feel like GPT-4 obsoletes software engineers as it's used yet, but it does really raise the bar on what's possible to build.
I’ve asked GPT to refactor blocks of code and it sometimes does a really good job, sometimes it makes it extremely verbose and wordy but you can’t always win.
I just hate how slow it is even with Plus. For analyzing code for debugging you’d ideally have something that moves a little faster.
I guess we’re in the 56k days of LLM watching it print line by line.
I review PRs primarily for correctness and to look for nonobvious problems. I rarely find buggy code. Sometimes I'll find performance issues. In order to do what I do when reviewing, ChatGPT would need clear specs (yeah right) and a deep understanding of our system, domain, and even our customers.
Of course I'll try it and see how it does. But it struggled and failed to produce what I would consider trivial code when I tried to create a single usable page to spec in react native, so I'm not terribly optimistic.
That's actually how I spent a lot of my time. GPT drafts code of mixed quality, and I spend a lot of time reviewing it.
What's interesting is that GPT isn't trained to produce high-quality code. It's trained as an autocompletion tool. I'd be curious how smart such a system could be if we knew how to engineer it for quality.
Because people will generate more crap and throw it at you as the quality control. They'll have less knowledge overall as they cede more to chat GPT. Also your impact will be in cr, so they'll send you more people. You get faster and more responsive using chat GPT to review, so you'll do more reviews, causing people to send more frequent ones to get your feedback faster. Eventually just hire some interns to toss plausible shit at gpt and then you. Fire the ftes. Done.
GPT can't do full code reviews. GPT is very good at finding improvements, such as making code more functional, bad variable names, etc. In some sense, it's like ESLint or pycodestyle on steroids.
It won't find subtle race conditions, abstraction violations, or deeper issues.
I find it makes code reviews shorter and more productive, so perhaps you'll spend 10% of your time instead of 60%. You won't spend 0% of your time, though.
Which is why I still believe proper developers will still have jobs :)
GPT is just Spreadsheets in a way. Lot of companies had business users and SMEs create contraptions based on Excel and then when it's a hairy mess or a rube goldberg machine, there are "digital transformation" projects that are done to make them proper apps.
That might be the exact right analogy. People who see Excel for the first time seem to think they have superpowers. In actual fact the superpower is knowing how to use it effectively and knowing when you are on the edge of its abilities.
Especially that code generators are pretty old, ChatGPT is very late to the game. I see all this people spending time to create a landing page with ChatGPT whereas in reality you can just install Wordpress and it even looks nicer.
GPT is a much better debugger than coder in my experience. It is a drop in replacement for googling random linux errors too.
"I'm getting this error <error> with the following code. Please add log statements that will test for all the things that could be going wrong <code>"
Saved me about 15 minutes yesterday. Not only does chatGPT generate the probably causes of the error, it knows how to test the code for those errors. The human doesn't even have to read the error message anymore.
Plain and simple: Feed the code it produced back into it and ask it to modify the sections needing maintenance or changes, or just tell it the issues you're having and ask it to explain/debug them.
Be careful! I have been experimenting with just this approach and ChatGPT 4 often forgets important parts of the code when generating the new files. If you ask ChatGPT to break down every change into step-by-step changes with code examples, it does a really great job and you can check every change yourself before implementing it.
A good prompt is key. If pasting code make sure to run it through the tokenizer first to make sure it’s not too long. Be very descriptive about what you want it to do. If it goes off the rails just hit the stop button and then edit your prompt with the little edit button at the top right and try again, this saves on context length and it saves the previous prompt and response so you can switch between them and compare.
IMO, the problem is that it doesn't work. It could introduce other bugs, make incorrect changes, or mess up the code. If a good developer does this, it converges, with GPT I am not so sure.
We need to find a way for the system to iterate, debug, write unit tests, and repeat the loop. In theory it's doable, but we are yet to see the system which does this.
A lot of people seem to be hung up on the title and totally missing how useful this tool is. People are right that the tasks that you're having this tool do aren't tasks that a 10x (or 1000x) developer would do. But what they're missing is that this is exactly why you can use this tool to multiply your productivity. If you don't have to spend your time writing silly scripts anymore or converting JSON to yaml or whatever other yak you're shaving, your time is free to spend on other stuff, making big decisions, solving truly hard problems. And that's how you become a 10x developer.
Thanks for this article, I found it very inspiring.
It possibly enables you to build the wrong thing, 10x faster.
Do you think the company will let you just have fun learning for the other 90% of the time that you can free up? They’ll just expect more output. Being able to write code 10x faster will likely not free your time enough to become a 10x problem solver.
No, it frees you up to spend your energy on thinking about what the right thing to build is. If your head is buried in code all the time, that's when you can't stop and look at the bigger picture and ask if you're building the wrong thing or not.
10x being real or not aside, if there's a 1x-type they certainly don't do many of the things in this list in my experience and certainly not:
>Writes code that others can read.
>Reads the Docs.
>Updates the Docs.
In my experience, half of the differentiation between above average and solidly "1x" or middling devs is reading documentation and obtaining a deeper understanding of the tools, language, platform they're developing on top of.
There seems to be a lot of dissenting opinion on how these tools will affect software engineering jobs.
Those of us that have deep architectural skills, are able to deal with ambiguous requirements, balance UX with technical limitations, etc, will be safe for a good while.
Those of us that were trained in bootcamps to crank out MEAN/MERN/etc apps might be in a bit of trouble.
I say this not to disparage the latter group--many are great engineers who do great, reliable work. But it's ultimately rote work, something LLMs excel at.
There is a huge tranche of SWEs who are going to become redundant soon. Lots of them gave up careers as teachers etc for a more lucrative career in tech. It's unfortunate.
I'm hoping the AI scare and layoffs stop people from flocking to software engineering. Less supply of us means more demand, higher wages, better working conditions. Especially with how things are going, the last thing we want is a new generation of swes to compete with the dwindling amount of jobs.
I would think whatever we consider today as mid-level would become the expectation for entry-level. It would be nice if we had a culture of apprenticeship in our industry though.
AI-assisted programming tools have already increased my personal coding productivity to the point that they've made me more ambitious: I'm building whole projects that previously I wouldn't have committed to, because they would have taken too long (just past my "worth-it" threshold) otherwise.
Will I get anything more out of this? As a 10x engineer I can expect to make at most double what a normal 2x engineer (1x engineers mostly get out of engineering in my experience) makes when I get to the tail end of my career.
I like to improve my engineering skills. However I also like to take nice vacations, do wood carving, play mandolin, go for long walks in the park, see my kids grow up, and all the billions of other things people do in life. There isn't time to do it all: time spend becoming better at one is time stolen from the others.
Don't get me wrong, this is neat. But the only reason why this works is because there is one million tutorial out there who use this data for teaching new students. The bigger issue is edge cases, stuff where there is no precedence. This output is a starting point for something, but it does not even get you 3% of the way there unless you work for a teenager who is into basketball anecdotes.
It’s also trained on code and books. The knowledge is in there, it just needs more RLHF to improve its abilities. There’s a good chance it can handle those edge cases by combining the information it has in different ways.
If the books have the specific code that is. If the books have some code and discuss edge cases, that is not something a language model could handle. It could give you the abstract concept that applies but it would fail when you prompt it to create new, valid code based on abstract concepts that are in the data.
Why do you think it can only copy existing code? I’ve used it for some really obscure cases and it’s still able to produce useful output both code and language. Try using it to create your own programming language with completely different keywords and oddball syntax and control structures and then ask it to write programs in it, it actually works!
My experience with code assistant is strikingly similar to my current experience with Vim. Sometimes, it makes me slightly faster. Other times the experience breaks down spectacularly on previously simple things. And last but not least, it is difficult to get rid of it.
(shameless self plug)
wrote a little shell script to efficiently use gpt in the terminal. maybe someone else can make use of it:
https://github.com/AlexW00/cli-gpt
Okay, let's talk about a theoretical figure, called the "1000x novelist", a very productive author who uses advanced AI to write novels really fast. Clearly super productive, because novels get written so very fast.
Now, here is a question:
George R.R. Martin has been writing on the 6th book of "Game of Thrones", titled "The Winds of Winter" (TWoW) for over 10 years. It still isn't finished.
What do you think, which will sell more copies: TWoW, or all novels of the 1000x novelist combined?
I tried chatpt and asked a few questions about doing something specific in a Rails project. It was so specific that I had already found the sources online that were avaiable. It summarized the sources into directly actionable steps. At first I thought that was odd and unhelpful, but perhaps I expected too much. So then I tried giving it a prompt about setting up the devise ruby gem in a new rails project. That's one of those tasks that it helps to be familiar woth once and you can promptly forget; you'll rarely return to authentication in the lifetime of an app after its done and secure. It again presented me with the exact actionable steps for setup. I reviewed the gem's GitHub readme and found the response to be correct.
As far as I can tell, chatgpt performs the action of finding and simplifying the information needed to take the next step forward, with caveats for a lack of understanding unless asked and the potential to be inaccurate or wrong. But that's all it does at the moment. No better than asking the "devise expert" on my team how to setup devise from memory. Kinda cool experience, but I doubt it's a titanic shift because it doesn't have the depth of sematic meaning you and I do.
> As far as I can tell, chatgpt performs the action of finding and simplifying the information needed to take the next step forward [...] But that's all it does at the moment.
ChatGPT's output is certainly limited by its input. If you ask questions that can be answered by finding and simplifying information, then that's the answer you'll get. However, if you treat ChatGPT like a functioning software engineer, describe the problem space, explain the requirements, and define the boundaries, then ChatGPT will do much more. From that point on, you can ask it to explain why it chose a certain data structure, or to adjust an implementation approach, among other things.
With 15 prompts in 2 hours, I was able to pair program a novel 200-line program in Go that used AST parsing, command execution, pattern matching, file parsing, and custom data structures for specific data processing. All to solve a problem one of my teams experienced in the industry. ChatGPT wrote 99% of the code; I tweaked 2 or 3 lines because it had problems correcting an error in a regex (most likely due to the fact that ChatGPT sees tokens not characters). The code compiled, ran, and produced the exact result I wanted.
Researching, prototyping, and writing this code by hand would have taken me a few days probably. That's why the project sat on my TODO list for months
I often find myself delving into vendor code that is public methods but not necessarily advertised in their documentation.
I'm working on trying to figure out lang chain to basically feed it all my vendor and node module code and their respective docs, so basically hallucinating goes way down and the info is as fresh as my cron job to rebuild the langchain embeddings.
I'm doing this for laravel, filamentphp, livewire, alpinejs, vuejs, nextjs, react, svelte, typescript, and PHP docs as well, mostly just the docs... I'll probably work on limiting the info to the most important things or just docs for some of the things...
I'm then planning on having gpt agents specifically tuned to write unit tests another to write code, another to review the code another to run and validate the tests, another to model data and create migrations, etc ...
naked chatGPT can do a lot, adding custom embeddings changes it into a master of x topic and levels it up 10 fold.
50c had a mil' in the bank and was still on the grind. Increased productivity doesn't mean freedom from more work.
This article is beautiful and effective. Thank you for the information!
The culture around GPT at the moment is verging on fanaticism.
When car companies close down and people lose jobs, it's sad for the die-hard followers, but most people don't care so much. This closure of trivium-level programmery is being met with a ramp up in fanaticism and repetitive doubling-down on preset views. So odd...
Ok, dumb question, but does the API skip over the whole, "Thanks for asking that question, here's what I found:" stuff I keep seeing in ChatGPT when I ask it to generate non-text responses?
Even when I ask it the "generate a 10 row csv containing sample chipotle restaurant order data with headers" prompt, I get "Sure, here's an example CSV file with 10 rows of sample Chipotle restaurant order data, including headers:" at the top...
Are you using the API endpoint or browser chatgpt? I tried it 5 times via API and it works and then 2 times via the browser and it doesn't work. They are different models.
If you are using the API try lowering the temperature mb?
This is a pretty cool idea, at least to experiment with. If you use ZSH, I made a ZSH plugin version based on the article: https://github.com/antonjs/zsh-gpt
I've been enjoying using aichat for command-line chatgpt access https://github.com/sigoden/aichat (it supports both an interactive repl mode and you can invoke it with your query directly from the command line like the helper used in this article).
Not mentioned in post this but occured to me by reading:
I think I will start changing my functional prompts to require a JSON format in responses so that various aspects of the response don't need to be manually parsed, and requests can be more reliably piped to subsequent requests.
> Not mentioned in post this but occured to me by reading:
> I think I will start changing my functional prompts to require a JSON format in responses so that various aspects of the response don't need to be manually parsed, and requests can be more reliably piped to subsequent requests.
that adds more characters though so maybe hits token limits faster, I was thinking about how to have better contexts then found out about langchain and creating a sort of long term memory using vector databases. still trying to figure this out, but once I do, I think it'll be amazing what I can do with it.
I don't know what kinda code-bases you work on, but I think there are many more potential factors which determine how useful GPT might be besides "tiny and well-defined". I won't try to enumerate them here, but as an example if you are working in a much more / entirely functional way, you can think of your entire code-base as tiny and well-defined, even if it is composed of 1000s of functions. Can think similarly if the arch is microservices.
In which case the main problem an engineer is solving is how all these interact together. And not the X minutes that it takes to write each individual well defined tiny function.
It's often that little edge piece of script that kills productivity. Something you need but off your beaten track that takes a few hours or days to figure out. That's ai coding will shine.
Using Gpt is more like trying to get to point a to point b, it allows you to move faster, but you still need to decide what paths and modes of transports to take. So overall it could 1.5x you
this + fish...if someone installs them via installer rather than building from source, is that a security risk? Meaning, is fish so powerful that a compromised version of fish could scrape / intercept GPT comms and either prime GPT networks or disturb them?
change gpt-4 to gpt-3.5-turbo worked, gpt-4 api is only for beta testers and you need be on the waiting list to get approved.
I tested the scripts and they work really well, esp like the image generation script. However unlike the web-browser the scripts does not keep the context, which is not great.
is there a way for shell script to maintain context session somehow?
"Human: you seem to remember the context of the chat, how many context you keep, the most recent previous output, or a few conversations
AI: I generally remember the last few conversations and I'm also capable of keeping context in longer conversations."
I don't think there is a way to feed context back to the script efficiently, maybe need a local database or a queue of certain length to sustain the subject. To mimic the web-browser experience at terminal.
the real magic for chatgpt vs old chatbots is that it maintains context, how to keep that via API calls is something I do not fully understand yet. Need read the API reference I guess
ChatGPT feeds the context of the conversation back into itself. When it hits max tokens, it then summarizes the conversation to feed back into itself. This is one of the things where ChatGPT can start losing track of what it was talking about if that summary doesn't have key things in it or missummrizes the conversation.
For
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
It will then add in some text in line:
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
AI: That's great to hear! Do you need any assistance with anything?
Human: How many peks are in a bushel?
This is 93 tokens.
And then I get:
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
AI: That's great to hear! Do you need any assistance with anything?
Human: How many peks are in a bushel?
AI: A bushel contains 8 US (or 32 UK) pecks. Is there anything else I can help you with?
Human:
And then...
The following is a conversation with an AI assistant. The assistant is helpful, creative, clever, and very friendly.
Human: Hello, who are you?
AI: I am an AI created by OpenAI. How can I help you today?
Human: I am fine. Working hard.
AI: That's great to hear! Do you need any assistance with anything?
Human: How many peks are in a bushel?
AI: A bushel contains 8 US (or 32 UK) pecks. Is there anything else I can help you with?
Human: What produces does that measure?
AI: A bushel is a dry measure used for measuring fruits, vegetables, and other grain products.
Human:
You will see each time it is feeding in the entire chat history as part of the prompt. That is how you maintain the context of the chat.
To do this as part of a command line interface, yep - it would need something like .gpthistory which gets fed in to each message... though this eats tokens.
You can see some of this in the playground if you look at the 'view code' button at the top.
if you want a vision of the future, winston, imagine a half-wit shipping generated code they are incapable of understanding, modifiying or debugging into production, forever
Spoilers: the future is actually kinda just ok-ish. GPT is used in almost all professions but akin to Google. Nothing really changes, we come back to these threads and articles every couple years to look back and realize how plain stupid and hopeful humanity really is sometimes.
Adding a major new tool that is “used in almost all professions” is a pretty big deal even if it’s only used as productivity software, which IMO is still far too short sighted of its potential (a simple chat interface is just the baby steps before we figure out all the API driven usecases).
What was the last time we could look back at old HN threads and say that? Bitcoin?
+1 to all the comments about deciding what problems to tackle.
My nuanced 2c on it is that every line of code you write is a liability. Part of what revolutionized pricing mechanisms was FOSS and SaaS, which are 2 strategies of amortizing the cost of development across many many people. Every line you write for your company is a line with amortization across exactly 1. So first consider the value/cost tradeoff of FOSS and SaaS to see if you can compete. 1000x might be simply glueing together a few free/cheap systems which would have cost your company quarters. Putting in nice interfaces/adapters means you can easily cut them out if they become a regrettable decision.
Hyperproductivity in software is all about deciding what problems to tackle. Richard Hipp isn't worried about chipotle restaurant orders in golang, he's worried about how to store data reliably. That isn't a coding puzzle that ChatGTP is likely to help with. Either ChatGTP can do the whole thing itself (not yet the case) or it is a minor productivity boost because the hard part is articulating the problem.
Writing code quickly really isn't a challenge that high performing software engineers need to tackle. ChatGTP is a cool tool, we're all going to be using things like it in a few years, it'll change everything. But it won't make any old engineer a rival to the big names in software engineering.