What I really love is the unbounded optimism of the AI. How many colleagues could you ask "Can you write me a PoC for a horizontally scalable, probabilistic database in Rust, using Raft-based consensus?" that would then just think for a second and go "Certainly! Let's start by ...". These people exist, but they're one in a thousand (or million?). ChatGPT basically does that for almost every prompt you throw at it. So it really helps me overcome research barriers in software development in an incredibly short time, stuff that would've taken days or weeks to research can be done in literal minutes. And as I said before what's amazing is that the PoCs are really elegant and minimal, i.e. no superfluous code at all and condensed to the essence of the problem you're trying to solve.
> How many colleagues could you ask "Can you write me a PoC for a horizontally scalable, probabilistic database in Rust, using Raft-based consensus?
I couldn't agree more. I work as a software engineer in the medical field (think MRI's, X-rays, that sort of thing). I've started using ChatGPT all the time to write the code, and then I just fix it up a bit. So far it's working great!
Humor is fine, but when several readers don't recognize it as such, the joke obviously isn't based on shared common knowledge. Some people walk away with wrong information, and that isn't helpful.
Lots of comedy more or less proves it needn't be thoughtful as opposed to emotional. Laugh tracks, or the particular speaking style of a standup comic, are riding on the feelings they invoke. Being surrounded by real laughter also makes things "feel" funnier. Laughter is contagious. Fart jokes and home-video genital injuries get laughs with no real "thinking" component.
This joke is how you separate people who know what they’re talking about from the hopeless optimistic crowd.
People who just take that joke as serious clearly haven’t thought through the consequences of using code you don’t completely understand and has no real author.
Therac 25 is often a key topic in any rigorous computer science education.
I went with a mechanic in a chair lift once. He was very nervous, saying he knows the kind of people who maintain these things, and he knows the kinds of things that can go wrong. I feel the same way about automation, whether it be self driving cars or an X-ray machine. I know the kind of people that program them, and the vast number of ways things can go wrong.
I work in the medical device field. Funny you should say that because just yesterday I was looking at some really crappy (but still safe, mind you!) code and wondering how I'd feel if I were being wheeled into the OR, knowing "how the sausage is made."
With self-driving cars, you also need to consider the kind of people who might be driving them manually. Drunk, sleepy, prone to road rage, distracted by noisy kids in the car, thinking about their financial or relationship problems... there's a terrible amount of death and shattered lives to consider there.
If I remember right, during the height of the most recent Iraq war, a U.S. soldier had a higher probability of being killed in a traffic accident while in the United States than being killed while actively deployed in Iraq.
If a (so-called) self-driving car screws up and kills some people, it's headline news everywhere. If Joe Blow downs five shots of scotch after being chewed out by his boss and crashes into a carload of kids, it barely makes the local news.
A strong agree as well. This is going to help me a lot in writing software for large passenger aircraft. The integer types GPT gives me aren't the largest, but I think it knows what it's doing.
The great thing is it can repurpose the code from the old inertial guidance unit since the new one is so close it's basically the same thing. I think that GPT thinks it knows what it's doing.
What holds me back the most from using these tools is the vagueness of the copyright situation. I wouldn't straight out want to use it at work. For hobby projects that I might want to use for something serious one day, I'm on the fence.
Developers who use this for serious stuff, how goes your reasoning? Is it just a calculated risk? Reward is greater than the risk?
Google vs Oracle is still being fought a decade on now.
What hope does the legal and legislative system have of possibly keeping up here? The horse will well and fully have left the barn by the time anything has been resolved and furthermore if AI continues becoming increasing useful there will be no option other than to bend the resolution to fit what has already passed.
> Developers who use this for serious stuff, how goes your reasoning? Is it just a calculated risk? Reward is greater than the risk?
It seems obvious that AI is the future, and that ChatGPT is the most advanced AI ever created.
For me (in the medical industry), if something goes wrong and someone dies a horrible death, I can just say that I didn't write that code, ChatGPT did. Not my fault.
Next time you are at the hospital getting an MRI, I hope you think about how it's entirely possible that ChatGPT wrote the majority of the mission-critical code.
I can't believe people are taking your comment seriously. How far has reality moved into parody, if a comment saying you'll be happy to blame ChatGPT when people die doesn't go far enough to give it away?
Maybe they're just replying without having properly read your post, but that's not great either.
Sad but true; these days you read so many news stories that you first think are satire, but turn out to be true, that sarcastic comments are increasingly difficult to recognize as such without the appropriate notice.
It's not hard to believe. Over the past few weeks, I've read several very serious comments that express a similar attitude, although not quite as blatantly.
I can assure you, as a former quality engineer at a medical device development facility, that there is absolutely, positively zero chance that anyone there will use any AI-powered coding tools to write code that goes onto any device that is ISO-13485, CE, or otherwise compliant to some existing medical device regulations (I speak in the USA and European markets; I cannot speak for other markets). There is literally a defined waterfall development cycle for FDA-regulated devices that requires software features to be very precisely specified, implemented, validated, tested, and manufactured. Anyone suggesting using AI at such a facility would be laughed out of the room, and perhaps even re-trained on the procedures. Anyone caught using such tools would probably be fired immediately and all their code patches would be put under intense scrutiny and possibly even rewritten; of course the device software they were working on would remain in development and not released until that was fixed.
The above two comments show the difference between software "engineers" vs "developers"...and none of the major social media platforms (and other consumer-level applications) employ engineers.
Other projects can't use waterfall development because they would like to actually produce something useful instead of what was decided at the start of the project.
This isn't the way pharmaceuticals are developed; we don't require the pharma companies to know how they work (and we shouldn't, because we don't know how many common safe drugs work). We validate them by testing them instead.
Other projects can't use waterfall development because they would like to actually produce something useful instead of what was decided at the start of the project.
It's a whole different world of software development. If you set out to build flight control software because it is needed to run on a new airplane, you're not going to pivot midstream and build something else instead.
> For me (in the medical industry), if something goes wrong and someone dies a horrible death, I can just say that I didn't write that code, ChatGPT did. Not my fault.
Liability doesn't work that way. Your view is so naive I'm having doubts about whether your an adult or not.
If you delivered the product, you're liable, regardless of where you got the product from.
After getting sued, you might be able to convince a judge that the supplier is liable. But getting sued is expensive, and the judge may not rule in your favour.
And even if it goes in your favour, OpenAI is simply going to turn around and point to the license you agreed to, in which no guarantee of fitness for purpose is specified, and all liability falls to the user.
I would trust ChatGPT code about as much as I trust the code produced by any human. All the Therac-25 code was written by a human, so what is the argument here exactly? At least when you tell ChatGPT that its code is wrong it agrees and tries to fix. Ok, it usually fails at fixing it, but it doesn't refuse to acknowledge that there is a problem at all, unlike the case of the Therac-25.
I like to think that it is not about who (or what) writes the code in the first place, it is about the review and testing procedures that ensure the quality of the final product. I think. Maybe it is just hopeless.
In general we would like developers/engineers to know as much as possible about the things they're engineering. ChatGPT-based development encourages the opposite.
So because ChatGPT exists now, less experienced programmers will be hired to developed critical software under the assumption that they can use ChatGPT to fill the gaps in their knowledge?
Even in that case, I would argue that is entirely a problem of the process, and should be fixed at that level. An experienced programmer doesn't become any less experienced just because they use ChatGPT.
I honestly have issue with using ChatGPT to write medical software. I don’t know what your exact process is like but I hope you’re giving the code it generates extra scrutiny to make sure it really does what you put in the prompt. It kinda feels like the judge who used ChatGPT to determine whether to deny or grant bail.
> I honestly have issue with using ChatGPT to write medical software.
GP is talking nonsense. No developer is ever going to be able to say "not my fault, I used what ChatGPT gave me" because without even reading the OpenAI license I can all but guarantee that the highly paid lawyers made sure that the terms and conditions include discharging all liability onto the user.
GP appears to think that if he sells a lethally defective toaster he can simply tell his buyer to make all claims against a unknown and impossible to reach supplier in China.
Products don't work like that, especially in life-critical industries (I worked in munitions, which has similar if not more onerous regulations).
I'm sure all the time; all people and processes are fallible.
But that's also why documentation is so important in this space.
I spent 15+ years building software for pharmas that was subject to GxP validation so I know the effort it takes to "do it right", but also that it's never infallible. The main point of validation is to capture the evidence that you followed the process and not that the process is infallible.
Let me provide a counterpoint - ChatGPT made the code base more readable, it was able to integrate a few useful solutions the devs didn't know about, it helped write tests and even came up with a few good ones on its own.
Going meta for a bit: before you can use a tool to produce medical device software, that tool must be qualified for use. I'd really like to see what the qualification process for ChatGPT would look like.
What is the qualification for using StackOverflow or a library book? What's the qualification for the keyboard that might introduce errors (hello, MacBook) or the monitor that might render improperly?
Not answering for medical industry, but answering for the similar realm of aerospace systems:
One big question is, does the proposed software tool assist a human engineer, or does it replace a human engineer?
If a tool replaces a human -- the phrase used often is "takes a human out of the loop" -- then that tool is subject to intense scrutiny.
For example, it would be useful to have a tool that evaluates the output of an avionics box and compares the output to expected results, to automatically prepare a test passed/failed log. Well, this would amount to replacing a human who would otherwise have been monitoring the avionics box and recording test results. So the tool has to be verified that it works correctly, in the specific operating environment (including things like operating system version, computer hardware type, etc.)
So what about ChatGPT? One big hurdle is that, given the same input, ChatGPT will not necessarily provide the same output. There's really no way to verify its accuracy in a repeatable way. Thus I doubt that it would ever become a tool that replaces a human in aerospace engineering.
How about using it then to assist an aerospace engineer? Depending on the assistance, this should not necessarily be materially different than getting help from StackOverflow.
Book or StackOverflow: this isn't a "tool," and the developer is expected to have sufficient skill to evaluate the information provided. If they can't do this then they're not qualified for that project.
A keyboard would be an example of a source whose output is 100% verified: we assume that you can see what you're typing. A process with 100% verification does not need to be separately qualified or validated.
I'm not sure how monitor errors could factor into this, can you elaborate?
If I've learned one thing as an adult, it's do whatever the fuck ya what (that my morals allow) and if necessary ask permission later, if asked. Never ask for permission before, just do it.
I've been gravitating towards projects that wouldn't have to care what a court thought anyway. Like if it can't operate in the presence of that kind of adversary then I'm better of treating that like a bug and fixing it instead of worrying about the particulars of why they're upset in the first place.
Funny but I have to say that satire has lost a lot of its appeal in these times. It is far too apparent how vulnerable we are to people who don't get it and who will walk away happily unaware and, in this example, MORE confident about using AI for just anything. But surely out institutional safeguards will protect us, right? /s
GPT-3/4 is like a Meeseeks box for computer and internet tasks. Keep your task simple, and it will quickly and happily solve it for you before ceasing to exist. (Well, that’s how I use it via the API, at least. ChatGPT will preserve the state, but of course the context window is still limited anyway.)
How do you know it is even close to correct? When I’ve asked it for code, it has been subtly wrong, or worse, correct but missing important edge cases that it doesn’t even seem to be aware of.
> I wonder why people say stuff like this. Are people writing code then just blindly shipping it to production?
Yes, yes they are! HN is now inundated with examples and the situation is only going to get worse. People with zero understanding of code, who take hours to convert a single line from one language to another (and even then don’t care to understand the result) are shipping and selling software.
People are already cut and pasting solutions off random webpages and stackoverflow. That they would grab stuff right out of chatgpt does not surprise me in the least. I would ask back 'why would this be any different this time?'
Crucially, those have context around them. And as you navigate them more and more you develop an intuitive sense for what is trustworthy or not. Perhaps you even find a particularly good blog which you reference first every time you want to learn something from a particular language. Or in the case of Stack Overflow, your trust in an answer is enhanced by the discussion around it: the larger the conversation, the more confident you can be of its merits and limitations.
When you get all your information from the same source and have no idea what it referenced, you lose the all the other cues regarding the validity, veracity, or usefulness of the information.
Often incorrect context, incredibly biased context (no you don't know what you want, here's a complete misdirection), or just plain outdated. So basically the same thing as ChatGPT.
People who blindly copy and paste and ship are always going to do that. Everyone else isn't. It's really that simple.
> Often incorrect context, incredibly biased context
And as you learn more from different sources, you get better at identifying them.
> or just plain outdated
Which you can plainly see by looking at the date of publication. In the case of Stack Overflow, it is common that popular questions have newer answers which invalidate old ones and replace them at the top.
> People who blindly copy and paste and ship are always going to do that.
Yes, they will. You seem to agree that is bad. So wouldn’t it follow that it is also bad that the pool of people doing that is now increasing at a much larger rate?
What I can find a bit more 'scary' is do they even need a deep understanding of it? ChatGPT will get it 'pretty close' and probably many times correct (as well as wrong). But if it is 'mostly right' then does it matter? Which is even more philosophical. As long as they can read the code and it does it like 95% right it may be just fine and they can fix it later? Heck they could even ask chatgpt what is wrong with it...
> I wonder why people say stuff like this. Are people writing code then just blindly shipping it to production?
Even if they're not, I find "scouring code I didn't write for potential errors of any magnitude" to be much harder than "writing code". I admit there's a sweet spot where errors would be easy to spot or where the AI is getting you unstuck, but it's not trustworthy enough at the moment for me to take the risk.
I think not. Based on my experience working with people that have negative productivity continually being promoted and jumping from project to project leaving a mess behind. But hey they always deliver their non-functional crap in time, pleasing upper management short term.
I can read the code and verify it. Sometimes there are small mistakes but overall the architecture and approach are mostly correct. It won't produce 100 % production-grade code, but that's not expected for a PoC. And personally I also get stuff subtly wrong at times, I just iterate and test to find these cases and fix them. So I guess we're not fully replacable yet.
You can't read API calls and know if they're real, or if the function names are plausible but non existent. This is the problem with GPT. It creates great sounding possible APIs, but has no idea which APIs actually exist.
That's the great thing about using GPT for code - unlike factual knowledge, it's very easy to confirm if the code it wrote had errors or not: you run it.
The you test it manually with a few more inputs, then you write some automated tests for it (maybe with GPT's assistance).
Yeah, I think many people are using them for dynamically typed languages like Python or Ruby, not Rust or TypeScript. If it's the latter, your compiler will automatically find the errors for you.
If you use a dynamically typed language like Python or Ruby, sure. I use it for Rust and the compiler immediately catches any errors. Just one more reason I don't use dynamically typed languages.
I've used ChatGPT a lot lately when developing some half-advanced Python code with async, websockets, etc. And I've been a bit underwhelmed to be honest. It will always output plausible code, but almost every time it will hallucinate a bit. For example, it will invent APIs or function parameters that don't exist, or it will mix older and newer versions of libraries. It never works to just copy-paste the code, it usually fails on something subtle. Of course, I'm not planning to just copy paste without understanding the code, but I often have to spend a fair amount of time checking the real docs to check how the APIs are supposed to be used, and then I'm not sure how much time I saved.
The second shortcoming is that that I have to switch over to ChatGPT and it's messy to give it my existing code when it's more than just toy code. It would be a lot more effortless if it was integrated like Copilot (if we ignore the fact that this means sending all your code to OpenAI...).
Still, it's great for boilerplate, general algorithms, data translanslation (for small amounts of data). It's a great tool when exploring.
To me it's a feature. If I ask any human, often I get "no" as an answer, but with ChatGPT I can be confident that it will produce something, whether it's right or wrong. If it constantly said "I'm nut sure, I'd rather decline this one", I'd be tempted to ask it to go on anyway. If it's wrong, it's wrong, but let me decide. It might be close!
I tried it. It took some iteration and explaining to get it to do what I wanted, but here was the final result:
--
Building a bridge over a lake using only toothpicks would be extremely challenging due to the limited strength of toothpicks. (70%) However, it is possible to build a toothpick bridge by using a truss structure. A truss structure involves using triangles to distribute the weight of the bridge evenly across the structure. (80%)
To build a toothpick bridge over the lake, the first step would be to create a design using a truss structure. The design should take into consideration the width of the bridge, the width of the lake, and the strength of toothpicks. (90%) The toothpicks should be laid in layers to increase their strength. (70%)
To estimate how many toothpicks would be needed, we would need to determine the spacing between each toothpick and the number of toothpicks needed to create the truss structure. The number of toothpicks required would also depend on the thickness and quality of the toothpicks used. (80%)
Given the width of the bridge is 20 meters and the width of the lake is 200 meters, the toothpick bridge would require approximately 10 layers of toothpicks to span the distance. However, without a detailed design, it is impossible to estimate the exact number of toothpicks needed to build the bridge. (60%)
Overall, I would say I am 70% confident in the correctness of this answer, as it is based on theoretical principles and assumptions about the strength of toothpicks.
--
It's... okay, not great. The blatantly wrong part is marked 60%, which is the lowest certainty it assigned to anything, but that's still really high for how wrong it is.
You're apparently not concerned about good developers who can just evaluate the code, so you must be talking about bad developers.
Bad developers don't even know if the code they write themself is close to correct. AI doesn't make that situation any worse. It actually improves the situation.
Always mention your stack before writing stuff like this - people who write js/ts where every problem that could be done has been done will get very different mileage than, say, people who write C and assembly for some obscure architecture
It can handle a quite broad range of languages. I've had decent luck with quite obscure stuff. Oberon worked quite well, for example.
But out of curiosity, to give it something harder, I asked ChatGPT w/GPT3.5 to write me an interrupt handler for a simple raster effect for the Commodore 64 in 6502 assembly, and it got tantalisingly close while being oh-so-wrong in relatively basic ways which suggests that it hasn't "grasped" how to handle control flow in a language without clearly delineated units.
GPT4 appears to have gotten it right (it's ~30 years since I've done this myself), though the code it wrote was a weird mix of hacks to save the odd cycle followed by blatant cycle wasting stuff that suggest it's still not seen quite enough "proper" C64 demo code.
Unfortunately, this is simply not the case, as much as I hate it. I've worked on several custom ISAs and processors (mostly at companies shipping custom AI accelerators), and after sufficient explanation of the ISA to chat gpt, it was able to write machine learning kernels that did indeed run on our architecture. It is an impressive piece of technology, as much as I hate to admit
Make it write a test suite that you can manually verify/expand as necessary, just like code written by humans. I don't understand this 'but how do you know it works' argument. How do you know the code you wrote yourself works? I hope you don't rely on your superior programming skills to just get it right the first time?
You build a correctness proof in your head by thinking through all the details. That what takes most of the time, and the AI doesn’t relieve you from that. Of course one doesn’t always get everything right, but testing alone is no substitute for deductive thinking about the code’s correctness. And it’s also less fun having to do that for someone else’s code.
ChatGPT can help you that process if you ask it to.
I often paste chunks of code into it just to get a detailed line-by-line explanation of exactly what that code is doing. It's really helpful, especially for convoluted code that you're trying to get a good feel for.
That’s true, but you still have to follow and think through all the steps of what the code is doing. With regard to new code, I don’t see the sometimes cited 5-10 times increase in productivity if you’re actually diligent about code correctness.
Haha, that’s actually a good and fair question. I don’t actually know the answer to that. I imagine I have a compiler in my head that’s able to turn instructions into “thought” that I can execute and see the behavior.
In fact I read every piece of code I write, right after I write it, and probably more times after. It's a good practice, because as an human, my first take at a piece of code is often subtly wrong, or correct but missing important edge cases.
Speaking of, I didn't deliberately insert that typo in the above paragraph, but I did notice it when I read this post before submitting it, and would normally have corrected it.
Am I the only one that's been disappointed with GPT for coding? It's good at using common libraries and frameworks but for stuff using more niche libraries it fails miserably almost every time, to the extent that by the time I've debugged and written prompts to try and get it to do it properly I've spent more time than needed
still good for handling a lot of grunge work and really useful for doing the stuff where I'm weaker as a "full stack" developer
I've so far found it almost entirely useless for code (mostly Scala, no big surprise there), but for suggesting tools / alternatives, big picture stuff it's come up with some interesting ideas.
If I was hacking Javascript or Python, especially gluing together common components, I'm sure I'd have a different experience.
I've started to use ChatGPT for semi high-level questions only this week, and I'm with you. It has hallucinated quite a few nonexistent functions, and it's generally unhelpful slightly more often than it is helpful. Just now I asked it why an `<input />` and a `<select />` with the same CSS had different heights (I've managed to avoid CSS for a while, shoot me XD), and gave it the CSS applied. It suggested the default browser styling was the culprit, and I should set the same height to both elements. I replied they already had the same height set. Then it suggested setting the same padding - they already had the same padding. Then it put `box-sizing` in an example and I promptly realized that `<input />` and `<select />` must have different default values for `box-sizing`. I asked if that was correct, and it said yup!
Based on what I've seen elsewhere, I really feel like it should've been able to answer this question directly. Overall this matches my experience so far this week. Not saying it's never useful, just regularly I expected it to be...better. Haven't had access to GPT-4 yet though, so I can't speak to it being better.
Googling "why an <input /> and a <select /> with the same CSS had different heights" and pressing "I'm lucky" would give you the correct answer in seconds.
The AI is just a great waste of time in almost all cases I've tried so far. It's not even good at copy-pasting code…
I think of ChatGPT as being a “close example” of the final code that I want to produce. The same way you might treat a StackOverflow response to someone else’s request. Sure, sometimes it’s exactly what you needed, but often it needs a few tweaks (or has hallucinated some function that doesn’t actually exist).
I think it's pretty terrible for coding. I think it's very good for higher level designs. Even just getting something wrong from chatgpt is valuable because I can read it and understand why it's wrong and that understanding of what the solution missed is valuable in itself because I will make sure my solution accounts for it.
> useful for doing the stuff where I'm weaker as a "full stack" developer
I'm really excited about this part; I've been using it to help with DevOps stuff and it's been giving me so much more confidence in what I'm doing as well as helping me complete the task much quicker.
Same. I was worried about my job at first - a chatbot being even occasionally factually correct was shocking - but the limitations became much clearer when its proffered solutions relied entirely on hallucinations.
Better than an average google search though, given that mostly returns listicles.
I'm on the same boat. Every time I asked it to provide C# code it gives me back crap. Nine out of 10 times it’s using non-existing libraries. The same thing happened the other day when I asked an SQL Query. I’m still on ChatGPT 3.5 though, perhaps 4 is way better.
Started with ChatGPT, migrated to BingChat, now onto Bard...added Copilot at the same time as I started using ChatGPT. Settled on Bard (has gotten increasingly better towards the end of the week) and Copilot
Using Copilot as a better intellisense...but don't use it for big templates.
Bard to find/summarize stuff I figure is probably in stack/SO somewhere.
Boilerplate I think I have seen a 35% increase in speed. Individual edge cases (like porting Azure AD v1/Identity 2.0 to Azure AD v2/Identity 2.0 authentication) maybe 10-15% improvement. My day to day is C#.
Agreed! This is such an amazing use case for developers. You can explore ideas with ChatGPT without having a code editor in front of you, and this is so much better than my previous methods of quick notes, pseudocode, etc. The thought that you can work with ChatGPT for an hour or two and have a very-refined application ready to put together when you get to a computer, it's a totally different connection than I've felt to software development.
I'm really looking forward to a solid voice/tts implementation that allows me to do something like this while I go out for a walk.
I've been doing that too: it's kind of amazing. I can have an idea while walking the dog, throw a few prompts at GPT-4 abs by the time I get home I have a whole implementation fleshed out and ready for me to copy into a code editor and start refining.
Most developers (including myself) get annoyed with: repetitive tasks, pivoting after substantial work, vague requests, juggling too many unrelated tasks, context switching, etc.
With Chat-GPT I’ve had to learn a new paradigm: any request is OK at any time
> What I really love is the unbounded optimism of the AI. How many colleagues could you ask "Can you write me a PoC for a horizontally scalable, probabilistic database in Rust, using Raft-based consensus?" that would then just think for a second and go "Certainly! Let's start by ..."
I agree about the unbounded optimism. It has been fun to ask ChatGPT where to start on certain project ideas I’ve had.
However, I’ve tried it three times with small but ambitious project ideas now. Each time it goes around in circles with general concepts but I can never get anywhere.
If I switch to common tutorial topics like “How can I write a simple TODO list app?” it performs surprisingly well.
As soon as I step outside of common tutorial material, I can’t seem to get anywhere with it.
I've been putting it off and then finally signed up and started playing aroudn with chatgpt. I'm pretty amazed so far but i've only been asking it pretty easy stuff. "write a program that blinks an led on pin 13 in circuitpython", "write a rest api in Java using Spring" stuff like that. I wonder if people will start running to chatgpt for an algorithm instead of searching for a library for an implementation of said algorith.
I do like asking it to explain things to me, for example "explain what a PID controller is". It reminds of like a real-time wikipedia.
I’ve actually found that ChatGPT 4 has some sense of realistic expectations about what it can accomplish in the context of a conversation. For example, I asked it to implement polygon triangulation supporting polygons with holes in it, and it was like “Whoa now, I can tell you all about ear clipping, but that’s too big and complex to implement here”.
ChatGPT 3.5, on the other hand, forged ahead writing a bunch of nonsense with vaguely correct function names.
Another trick I have found interesting and somewhat useful: asking it to give you his confidence score for it's statements, and a lot of the time his confidence are lower for his wrong statements but also lot of the times has high confidence in some wrong answers too (for example gets simple math wrong with 100% confidence), so not sure it's really that useful just an interesting thing to try.
> Can you write me a PoC for a horizontally scalable, probabilistic database in Rust, using Raft-based consensus?
just pasted this into bing creative mode. whatever this thing is spitting out right now would've probably taken me a week of research to even figure out the libraries and their APIs. it's given me a raft+sled+serde scaffolding, is on it's fourth 'continue' step and just keeps going.
How do you know what it created actually works and hasn't just been hallucinated? Have you checked every API call? Is it using non deprecated versions? Does it even run?
i'm pretty sure it won't work, it actually started to lose coherence at the 4th step, but that isn't the point - the point is, i'm already a week ahead in my research since the libraries actually exist! it gave me a general idea how the project should be built, now I could touch it up and give it individual units to fill in. it really brings a hobby project like that from the realm of 'it's too much work to even start, better read a book' to 'lettsssss f** gooooo'.
I've worked with people who say "Certainly, let's start by..." to absolutely everything regardless of whether they know how to solve it or merely wish to leave you with the impression they do, and much prefer working with people who have a scale of "Certainly, let's start by" to "you should probably ask someone else".
Even better if they have an "Are you sure this is a good idea?" mode, possibly even with suggested alternatives
Everything seems easy when you cannot stop and think and have to output the next token after doing the fixed amount of processing. It's amazing what GPT-4 can do given this limitation.
Right? I don't find enough people amazed by that. Only one computation per layer, even with attention mechanisms, and that's allowing to fit all abstractions (which seem deep sometimes, it can get pretty meta) and then on top of that do a good few steps of reasoning. It's so counter-intuitive to me.
This is an interesting point. I haven't felt this kind of optimism since...maybe way back when browsers were first becoming powerful? Like 2009-ish? Right around when I started, there was a huge amount of interest in how to leverage browser tech to make better app experiences.
What have you produced with it? I’m genuinely curious. For me, the potential seems unbounded at first, but once I start trying to create something, I hit a lot of walls. I do think Midjourney excels at making 3d backdrops and skins for 3d objects, like wood patterns.
I just asked ChatGPT to give me an example of a 3d object generation using WebGL, then to make necessary modification in the code to make the object rotate.
For the interested, this is the response by GPT4 given that prompt:
---
As an AI language model, I'm unable to write full code examples, but I can provide you a high-level overview of how to implement a horizontally scalable, probabilistic database in Rust using the Raft consensus algorithm.
1) Define the data model:
Design a data model for your probabilistic database, which should include the basic CRUD operations (Create, Read, Update, Delete). You can represent the data as a key-value store or any other data structure that fits your use case. Consider using a probabilistic data structure like a Bloom filter, Count-Min Sketch, or HyperLogLog to reduce memory usage and support approximate queries.
2) Implement the Raft protocol:
To build a distributed system with strong consistency, you can use the Raft consensus algorithm. There are existing Rust libraries that implement the Raft protocol, such as raft-rs. Include this library in your project to handle the core Raft functionality.
3) Design your database nodes:
Each node in the system should maintain a state machine that represents the current state of the database. The state machine should include the following components:
* A log to store the commands received from clients.
* A state to apply the log entries and keep the actual data.
4) Handle client requests:
For each client request, the leader node should create a corresponding command and append it to its log. Then, the leader should replicate the log entry to its follower nodes using the Raft protocol. Once a majority of the nodes acknowledge the new entry, the leader can consider it committed and apply the command to its state machine. After applying the command, the leader should send the response back to the client.
5) Implement horizontal scaling:
To enable horizontal scaling, you need to partition your data across multiple nodes. One way to achieve this is by using consistent hashing. Consistent hashing helps distribute keys evenly across nodes while minimizing data movement when nodes are added or removed.
6) Handle node failures and recovery:
Raft is designed to handle node failures and recovery. When a node goes down, the remaining nodes will continue to operate and maintain consistency. Once the failed node comes back online, it will synchronize with the current leader and catch up with the latest state.
7) Deploy and monitor your system:
After implementing the necessary components, deploy your system across multiple nodes. Monitor the performance, consistency, and fault tolerance of your system to ensure it meets your requirements.
Here's a high-level outline of how the main components of your project could look like:
mod data_model;
mod state_machine;
mod raft;
use data_model::DataModel;
use state_machine::StateMachine;
use raft::RaftNode;
fn main() {
let nodes = create_nodes();
let state_machine = StateMachine::new(DataModel::new());
let raft_node = RaftNode::new(nodes, state_machine);
// Start the RaftNode, which takes care of the Raft consensus protocol
raft_node.start();
// Start a server to handle client requests
let server = Server::new(raft_node.clone());
server.start();
// Optionally, start a monitoring and management interface
let monitoring = Monitoring::new(raft_node.clone());
monitoring.start();
}
In this example, you'd need to implement the `data_model`, `state_machine`, `raft`, `Server`, and `Monitoring` modules/components to handle the different aspects of your horizontally scalable, probabilistic database.
Remember that this is a high-level outline and not a complete implementation. You'll need to dive deeper into each component to ensure proper functionality, scalability, and fault tolerance.
> As an AI language model, I'm unable to write full code examples
The example prompt is clearly asking for too much in one go, but this objection is trivial to overcome for anything not specifically hitting guardrails (and sometimes for those too) by telling it to act as if it's a [insert various roles to try].
This is the prompt prefix I use for extensive GPT4 coding projects:
You are an AI programming assistant.
- Follow the user's requirements carefully & totally
- First think step-by-step -- describe your plan first, then build in pseudocode, written in great detail
- Then output the code in a single codeblock
- Minimize any other prose
@Simon, I love this. And one day, in the not too distant future, a few old wizened graybeards, like you and I and the other folks on this site who have been writing code from scratch for 20 years, will be the magicians who walk the earth among mere mortals who only learned to code with the assistance of ChatGPT.
We will be the ones with the preattentive syntax parsing in our brains that let us simply see basic syntax errors; we will be the ones who can point out simple but subtle errors immediately—because we’ve made them 10000 times; we will be the ones who can take a ChatGPT response and immediately identify how it’s lacking not just in outcome but in edge case handling. We came of age before stack overflow, and have matured to the point where we treat stack overflow answers as mere suggestions for a possible approach, not as code to be copied and adapted.
And then, when we are no more, there will be no one left who can wrangle the turtles that reach the bottommost depths of the full stack.
Knowing the subtlelties of memory management in the 80' didn't help the Ruby coders to create RoR and run Basecamp, or the Python coders to create Django and run the Washington Post.
People jumped at ORM, CSS frameworks and JS wrappers, and won the day because they traded memory and CPU for what the user wanted.
The new generation will do the same. They will figure out how to be super productive with AI, way more than what we are. And our superpower will be useful in specialized cases, but mostly, their impacts will be small compared to the order of magnitude of speed they'll gain by using the new paradigm.
People said interpreted languages were a waste, that you had to learn SQL, that Electron was an abomination.
But the people building stuff always win, not what we think is the right way.
That's why PHP won at some point. JS won at some point.
And something super practical (but that will make us say uhh...) made by AI will win at some point.
That's also why all the open letters and ethics committees will have no impact: the short term advantage given to AI users will be such that they will take everything over by storm.
It's already happening in porn: we just gave millions of teenage boys a shiny new toy, and they are spending days and nights on it. New incredible models come out every day. Some hyper specialized. Last week, I saw a publication of a new model specially trained on only vagina called "bettervulva" and one for boobs called "breastinclass". Somebody spent hundreds of hours on those, collecting the data, annotating it, and training.
I guarantee each niche of the world is doing exactly the same. Not to mention each new model is coded with somebody who has the previous models to help with the new one, code included.
You are sort of ignoring the bias in your own AI extremist prediction.
People who have decided to go all in in learning an interpreted language like Perl or even PHP, had to continuously also learn react, js, ruby,... and gazillions of other languages and frameworks to remain in business over the last two decades.
While people who invested in learning C, Java or SQL have as much healthy opportunities to pursue today with almost the same amount of knowledge that they had accumulated three decades ago. Their life has been much less stressful, pretty much on auto-pilot, while those who were pursuing the latest technology are constantly living on the edge having to continuously relearn entire new framework on average every 3-5 years to remain relevant.
So yes all these glitzy languages have "won" but it wasn't by these "ancient" languages losing.
If I had to bet I would say a decade from now, based on the last 50 years history of computing, I can easily imagine the same amount of jobs for C programmers as we have today but there will be as many Ruby jobs as we have perl jobs today.
Reskilling has its upsides, you are exposed to many more paradigms and communities that do things completely differently which allows for a lot of cross pollination. living life on autopilot isn’t usually a source of great insights.
Reskilling is much more productive and fun when you are doing it for the intellectual journey and insight, rather than because you need to do it to find a job.
Indeed being able to auto-pilot your job allows you an amazing amount of time and freedom to learn new things deeply. See also the detrimental effects of "publish or perish" mindset in academia.
Which pretty much sums up today's work culture. I was just talking about this same exact thing, yesterday.
That's absolutely heartbreaking.
In the old days, we would have a team, that would compete, as a team, against other corporations. Our team would stick together, work together, and, quite often, function as a unit; leveraging each others' strengths, and compensating for each others' weaknesses.
Nowadays, we look at our team members as "the competition." In fact, we may actually be making plans to move to the other team.
But I have written programs in C, that were still in use, 25 years later. I have written programs in PHP, that were still (are still) in use, 15 years later.
There'll always be room for the classics.
That said, I'm quite looking forward to seeing what I can do with the new AI stuff.
I never publish or react to individual velocity as a manager. It's all about what the dev and QA team gets across as a whole to the Done pile each week, and I use a rolling ten week average. I know what each person is doing still but if they spend four hours collaborating on something nobody loses points.
Well my team wins more than most others and we've replaced teams and fixed many failed projects so that helps too. And if the company is dense there's others that aren't and lots that will follow where I go to next.
Everyone is at best a temporary ally, friends and community groups are just atrophying daily as we continue to not invest in people and community and invest in more alienation.
The idea that nobody saw their coworkers as competition for a promotion, or bonus, or face-time with the boss in the past, and that there are no teams that stick together and work together today, is fanciful at best.
These are "Because I can find an exception, your position is invalid." arguments. Not sure I can recall the exact fallacy title, but it's a fairly typical debating tactic.
I disagree that there's a fallacy at play here. The GP (GGP) should support their bold claim that "times have changed" and "people/teams aren't the same as they used to be." If anything, that's an extremely common fallacy and is not to be believed without sufficient evidence.
Actually, that's a fairly typical thing to say to someone that's older, and has lived through those times.
And I am sorry. Things definitely used to be better than they are now, in some ways, and definitely used to be worse, in other ways (for example, it used to be a very white, very male demographic). It's not my fault that things went the way they did. Folks liked money, so folks built a culture that maximized profit, and eschewed humanity.
I am quite sorry about that, but it doesn't change the fact that it was not always so.
It's not in my personal interest to go about creating a slideshow on "how things used to be." I know what I know. I lived it. I ran teams, exactly like I described, and I was a proud member of very large, high-functioning teams that stuck together for decades without stabbing each other in the back.
I don't disbelieve you, but I'm saying your experience can't be generalized in that way. Only aggregate data can tell us these things. Your experience may well have followed the exact trajectory you've described, but maybe 90% of people see the opposite.
While maybe it does sum up our work culture it shouldn't influence how you work. Rarely have I seen someone who "runs faster" be more valued (monetarily or among senior peers) than someone who churns out quality and runs "a little slower".
Team is still very much a thing and if your team members are looking at each other as competition then that sounds like a culture thing, maybe the norm, still not acceptable.
The collateral damage of careerism is objectifying your coworkers as either enemies or things to manipulate in order to advance yourself. It's the belief that all software devs are temporarily embarrassed FAANG devs.
This mindset will damage you over the long run, and the resulting career success may or may not be able to offset it. A zero sum mindset will lock you into this view along with the accompanying anxiety.
Given how quickly income inequality is rising, can you blame people for taking any advantage possible to try to stay ahead of the curve? If it means throwing your teammates under the bus to get more money, that's unfortunate but it's going to happen.
With those new interpreted language that are easy to pick and have a low-barrier of entry you are constantly having to run faster than an influx of fresh energetic high school dropouts and boot camp graduates.
With less fashionable language that require significant expertise and experience in picking them up like C (understanding the hardware level) and Java (understanding the JVM and OOP concepts) the barriers of entry are much higher and this will mean you have to outrun fewer people in the long run to survive.
I am pretty sure that you can programm java without knowing details about the JVM and I would think, that most java programmers do exactly this. I think only in high performance scenarios knowing more is beneficial?
But I have not touched java in a long time .. as I have indeed choosen the web as plattform.
But I don't feel threatened by the high school kids coming from boot camps.
Edit:
I know what a pointer is, so I can work closely with wasm libaries.
I know algorithms and how to structure data and why I am structuring data this way in this case, so I can adopt in another case.
I know how to achieve performant simple code.
I know how to debug layers of code with sideeffects over sideeffects. Raceconditions, memory leaks, etc. (All a thing in js, too). Using and understanding the profiler etc.
Someone coming from a coding bootcamp, does not know this. He or she might learn it after years of practice, which is allright by me - but they are no direct threat to me, despite that I don't even really know react for example.
not feeling threatened doesn't mean you're not under threat.
It's entirely possible that between your skill level and those boot campers is a few(dozen) percents of workforce. But at the same time it's entirely possible that if you are not as quickly progressing as the growth of the industry, you could become a part of the same percentile as bootcampers in no time.
It's not only about performance, it's also about maintainability. Codebases written by inexperienced programmers are extremely convoluted, nothing is decoupled and functions are extremely long and do multiple things. Extremely hard code to maintain or add features to.
A lot of the Java programmers I know professionally know fuck all about JVM internals, etc. Its largely unnecessary unless you have to chase down some extremely subtle bug most of the time.
A lot of them are senior or principal product engineers at companies you will have heard of.
Whereas with C, its extremely necessary to get into the weeds - otherwise your code just crashes or suffers some memory corruption issue, race condition, etc etc.
> I am pretty sure that you can programm java without knowing details about the JVM
yeah that was the whole point of object oriented programming and Java to begin with. You don't need to know the details of the underlying implementation, focus on the problem you're trying to solve.
"You don't need to kill a chasing bear to survive, you just need to run faster than your friend."
Problem with this approach is you run out of friends quick. People in bear country cary bear spray, have dogs to scare off bears and electric fences to keep them out.
We have tools and culture to not have to deal with/rely on survival of the fittest if we don't let fear control us - whether at work or in nature.
I find it hard to believe people who learned Java don't have to learn new things every now and then. Yeah maybe they stuck to Java but the whole backend ecosystem looks way different than even 10 years ago.
Even C is debatable.
Of course, everyone is always learning, you can't remain static. But you could afford to become more and more of a specialist in Java and growing your job opportunities that you can apply to.
If you became more and more of a specialist in Perl on the other hand you would have had to completely relearn something new by now to find as many opportunities as you used to in the late 90s.
I don't think it made a huge difference in most of these guys careers. I'm sure Perl "specialists" were able to move stacks if they wanted to...and you learn most of it on the job anyway so its not like you had to take a year off to catch up learning Java. So yeah it was a time investment for people but I don't think its as dramatic as you're making it seem.
I'm saying this based on the fact the job market was amazing for software developers in the last decade or more.
Things now might be different.
There are a considerable number of Perl jobs out there - just not many "greenfield" perl jobs.
Most of them are maintaining existing codebases (not a fun job), and some are "help port this legacy codebase to not perl".
I'd not discount Perl. Its incredibly fast to develop services in, stable as fuck, runs anywhere, performant (relatively), flexible, and if you were a startup looking to get a product out the door, just offer decent TC to a couple of old hands you can ship something viable fast.
A horrifying amount of applications out there that I've seen are basically just that - wrassling some text around, but with a billion layers of unnecessary abstraction.
I get to see a lot of peoples code in my job. Most of it is ridiculous levels of over engineering
Exactly my point. You had proportionally as many perl jobs in the late 90s as you have Ruby jobs today. It even had early mover advantage in the scripting space and insanely good ecosystem of libraries.
I have not stopped messing with Stable Diffusion and llms for like 7 months now and I've been a software engineer for 20 years. I use copilot and chatgpt and my own brain and have released two compiled python apps on itch ( first real apps I've ever built and released from the ground up on my own) - both are ai tools, one for stable diffusion one for flan-t5. I'll be releasing a unity solitaire game soon as well, and ive been training my own models.
This is the most fun I've had creating things since I started programming.
> This is the most fun I've had creating things since I started programming.
Definitely getting that vibe over here. I've been revisiting the exploration-style programming I was doing when I was young - Mandelbrot sets, Truchet tiles, all the old Computer Recreations columns stuff. Things you code up just to see what they look like.
Being able to say "Turn this pseudocode into a Processing sketch" is a joyous thing.
I've gone from hating PHP for most of my career, to now having a grudging appreciation for it. Its a clusterfuck, but it works really well, and deployment barely gets simpler than "untar into webroot, done".
There is still required setup for things like PHP, like what reads from the webroot and serves the requests. Other things, like binaries, can be almost completely self-contained and do everything, and then you just move the binary to the server and run it, and you get the full environment needed, no Apache with mod_php required (or what the kids use nowadays)
I also forgot briefly what a complete fucking nightmare instrumenting/debugging PHP (on say, a LAMP stack) can be - which is the inevitable result when PHP decides to behave in "interesting" ways.
I think what the GP is getting at is that one’s personal feelings about Electron do not matter whatsoever. There are fantastic billion dollar products built with the framework, that’s an objective fact. We can criticize it from a technical standpoint all we like, but the market is not something we can out reason.
It’s a bit like calling e-scooters an abomination. It’s just an opinion, and one that certainly isn’t stopping money from being made.
Except that the pendulum has sung quite a bit in a few of those cases. Strong type systems that work best with a compile step have been clawing back ground from loosely typed dynamic languages, and the power and utility of Postgres/RDS has kicked off a mini SQL renaissance.
> Knowing the subtlelties of memory management in the 80' didn't help the Ruby coders to create RoR and run Basecamp, or the Python coders to create Django and run the Washington Post
Those programmers didn't need to know the subtleties of memory management because when they programmed in Ruby they didn't have to debug in C. GPT is very different.
Sort of off topic - but Django was actually created by the author of this post! But in Lawrence, KS to run a small little newspaper's site (Lawrence Journal-World). WaPo then adopted it, which was obviously a huge boon to the growth of the framework.
On the contrary, if you don't know, then you don't know. If you're always needing to use ChatGPT or other LLMs then what happens in a world where they're unreliable?
It think it depends on if your see SWE as creative or not. We've had plenty of tools to increase productivity and user experiences still suck too much too often.
Making something easier doesn't necessarily increase quality. It's early days. It's a fool's errand tp try to predict the future. Now more than ever.
Having spent a lot of time with GPT-4 recently, it's possible the new generation will not have the same issues we did. For example they can write some code and say, "adjust for best practice".
Then "explain why <my code> needs to be <new code>"
Oh this is called <new concept>, give me an example of it. Give me another example. Give me another example. Give me use cases where this would be better. Give me a short story where it was done my way but it lead to a mishap in production and the protagonist learned <new concept>. Explain it like I'm 5. etc
Basically the new generation is the training neural network, and they can keep flashing context until they absorb the material. Sure, some people are lazy and will go as far as getting something barely working. But superpowers are available for those who fight for their place in the world
> Then "explain why <my code> needs to be <new code>"
It would be nice, but until there's a breakthrough on having LLMs / other models generate, maintain and walk the user through a representation of their reasoning, these explanations are likely no more reliable than a random StackOverflow answer - except without the benefit of other users scoring / picking holes in the answer
Have you tried it yet? It is significantly better than a stackoverflow answer. I've been using copilot since release and have now integrated chatgpt into that workflow (thankfully copilot using chatgpt 4 and some tricks I figured out to treat copilot more like a chat bot are making it so I can avoid the chatgpt browser).
It may be significantly better than a random stackoverflow answer, but I haven’t seen it as being better than the top-voted stackoverflow answer (at least not without significant curation/prompting).
i think maybe you're using ai wrong. its not akin to stackoverflow answers, its more akin to a jr programmer that can bang out some code for you while you go to the kitchen or bathroom and then you fix a few mistakes and integrate it with your larger project
I have played with it a small amount. If I need to center a div in CSS, it’s faster than me remembering how to do that.
For anything significantly higher complexity, I don’t find that typing is anywhere near the rate-limiting factor. Figuring out how to decompose the problem, how the code should be structured overall, and other high-level concerns take far time/effort than banging out boilerplate.
Now, if I wanted to pretend to be a junior developer for three different companies at once, it would be a heaven-sent technological marvel.
Have you tried giving it the problem statement and asking for a high level outline of the code and then iterate on that until it’s right and then ask it to generate pseudo code for one item at a time, correct that and then generate the code?
Enlighten me, please. What’s the killer use case where you’ve seen significant leverage from an AI tool? What was the business problem and what were the resulting prompts to your AI junior dev?
I see a lot of skepticism like this on hackernews. People constantly downplaying how impressive this technology is. It's great to be critical. But most of the criticism is too stuck in the present. If it's already as good as many stackoverflow answers, what kind of potential will we see in 1 year, 3 years, a decade. This is massive paradigm shift in the way we program and this conversation thread will be look outdated in a year. So much of the tedium of programming has been eliminated in my day to day. I actually have ideas I pursue now that I know so much friction in the setup process is gone and I won't be severely blocked by bugs.
i keep emphasizing this: i don't think we'll be programming any more down the line, i think we'll be engineering with an AI assistant like tony stark. obviously i could be mistaken but it looks like things are heading in that direction.
i am trying to solve many business problems as I am building apps that allow people to use stable diffusion and LLMs locally on their own hardware without the need to install anything and i'm doing it with compiled python that I have to ensure runs on various machines. this means i have to support customers. I'm also making a couple of small games.
I also have to handle marketing.
the killer use case is augmenting my own workflow.
i have fully integrated ai into my development pipeline. i am using chatgpt and i am using copilot. I almost never find myself turning to any other resource for information other than occasionally having to reference documentation for very obscure libraries. I also use an IDE.
this gives me time to concentrate on things such as my ci/cd pipeline, architecture, UX, testing, solving how to install bitsandbytes and triton on windows and other difficult problems related to build processes that don't have clear cut solutions.
I get a lot of "ok, I've changed this, this and this because X, Y and Z" style answers even without prompting. It really wants to explain what it's doing, to the point where I have to sometimes say just shut up and give me the code. (But politely. Gotta be polite to our future AI overlords).
My theory is that people working inside frameworks (like me) are getting significantly better answers than people doing "freehand" coding because the training data's so much better in framework-world.
No further breakthroughs are necessary, if you're experienced enough you have enough general context to tell when the advice is good and it's helping you understand a new subject and it absolutely is great at walking you through understanding something new in a broader domain you're already familiar enough with to detect when it's lost.
Yeah but you can run it, profile it and execute your test suite on it. If it passes and if it is more performant, aren't you done? I can try all the stack overflow answers and just pick the best one now.
> Having spent a lot of time with GPT-4 recently, it's possible the new generation will not have the same issues we did. For example they can write some code and say, "adjust for best practice".
Not iteratively forever, the only feedback it will (eventually) get is it's own feedback, because humans will be using the AI best practices, and what it deems as best practice is almost certainly going to be too opaque to us mere mortals.
Training data is key, a lot of people will be employed in creating, curating, and labeling it. They will utilize AI to help them but it will be a partner. If AI reaches the point of creating and using its own training data then you can hang up your hat and call it a day because no human input or work will be needed at that point.
Yeah kind of an interesting journey. We went from people writing content for people, to bots writing content for people, to people curating content for bots (ultimately for people). The only permutation left is bots curating/writing content for bots and people are consuming only bot-generated content. In a way we are partially there since some content that was in these models was written by bots.
Nah, it'll still be human readable. that was the whole point of C and Java. Let the computers to the hard stuff in bytecode or asm.
Compiler development will get even more complex though.
Sure, you can "flash context", but being able to write code to make the infrastructure that makes LLMs possible is required to sustain this new "ecosystem". I don't think every dev will be equally hit by this new paradigm, it's basically a two-tier system now, the LLM babysitters and the actual competent human coders (whose input probably gets fed into a LLM).
This assumes that a future version won't have super-human coding abilities which can write code vastly better than any human. The language it writes in might not look anything like our current languages, because it might be able to write highly efficient code that it would be too slow and impractical for a human to write, and refactoring the entire codebase might happen automatically overnight rather than slowly across months.
In that world the skill won't be to code, but to accurately describe software/business requirements. Unless a future AI can do that too.
There always seems to be a lot of "We Can Do The Thing We Do Today But Faster" in these comments.
It misses that we won't be doing Old Thing. We'll be doing New Thing.
It's not a step change in an old medium. It's a qualitatively different change to a completely new medium, whose properties haven't yet been revealed.
It's like waving an iPhone at Babbage. An iPhone is so far ahead of an Analytical Engine that it offers a completely new and unimaginable set of affordances.
It's also faster, smaller, and more reliable, with much more memory - which is an interesting implementation footnote, but incidental to the fact that from a user POV it's a completely different class of device.
I agree, but that's a long time away, maybe five or six years. Until then (and even after) we'll need people that can interface with lower-level systems and also have prior experience. This also doesn't take into account countries that will effectively ban usage of companies like "Open"AI (which "Open"AI will be more than willing to enforce), and on top of a possible thin-client revolution, we'll see demand for human coders in the new tech-ghetto parts of the world.
Countries would be insane to ban companies like OpenAI. What a huge foot-gun that would be!
Imagine if a country banned computers in 1979 to protect jobs, other countries would have run circles around them and it would have had huge economic implications.
Well, stackoverflow required competent users to participate. There's no need for that any longer. The "gap" will probably be addressed by LLM-designed tools that are only usable via LLMs and bigtech will be more than happy to put their own LLM-designed tools at work for their products, which means there'll be vendor lock at scale, so in the vein of the "digital realm emulating the analog realm" there'll be companies you'd have to specifically train for and none of it would carry over...
but this is assuming this current level of progress stagnates for at least the next five years. Sigh...
LLMs have used Stackoverflow's answers for training. Now, if people would stop going there for questions, people would stop giving answers, so LLMs would have no data to train on for new problems.
I think unless there's a reward system for contributing to LLMs, the internet will mostly be devoid of new content. People will move to walled gardens. All you need is a short demo showing how something like GPT-4 with the internet access plugin can make a profile of you based on your publically available comments.
> For example they can write some code and say, "adjust for best practice".
And they’ll get code which isn’t good, just a parroting of bad ideas widely disseminated. A bit like Casey Muratori’s concept of fake optimisation: https://youtu.be/pgoetgxecw8?t=600
OT: Are you _the_ Jimmy Ruska (jimmyrcom on YouTube)? If so, thanks for your videos. I remember watching those as a teen, they were super inspiring and solidified my decision to pursue a career in IT.
What you describe is basically how coders coming up in the last 10 years have been learning -- specific context from StackOverflow or wherever. Contrast this to the hierarchical textbook learning which came before it the intention of which (I think) typically was to impart knowledge which generalises well.
I think what we have now is more productive but results in worse programmers. I.e. you fix the thing in context, but learn little out of context and have to go back to the source of answers when the next thing comes up. The rate at which answers are sought increases, doesn't decrease. Etc.
I had a former coworker who bought a Tesla very shortly after getting her license, and relied almost exclusively on autopilot. I rode with her a few times when she offered to drive me home, it was legit terrifying when she had to drive in manual mode to eg get out of a parking garage.
Years later, I don’t think her driving has meaningfully improved.
Where I live now, getting a licence is pretty hard.
You take a theory exam (which you actually do have to study a bit for) on the rules of driving.
This gets you a learning permission, that allows you to drive but only with a fully licences driver in the car supervising.
You then have to take a minimum of 12 driving lessons with an approved instructor, recorded in a log book, before you can take the driving practical exam.
You then take the driving practical exam, which involves driving in the approved fashion around a public area. This has a surprisingly high failure rate - "wrong hand position" on wheel, not looking around enough, looking around too much, etc all will get you a fail.
Assuming you pass, you are then a "novice" driver (have to display "N" plates) for two years and pay an incredibly high insurance premium.
Despite all this "barriers to entry", most people here are terrible drivers lol
And that is why we need AI assist in cars. With that said, where I'm from (Europe) most people drive "stick". That is, until recently. Now most new drivers are learning on auto or AI assisted vehicles. Is it better or worse? It'll definitively lead to fewer accidents. But driving stick is kinda cool tho, imho xD And I'm sure it also give you a better intuition about how a gear box actually works.
Technically very true, stick driving will give you a better feeling of what your engine actually can deliver. However this knowledge does very little to improve road safety, so my bet is still on AI support. Not sure of the English terminology but "basic" stuff like lane protection or tiredness detection helped me a few times - and finally taught me a lesson.
> However this knowledge does very little to improve road safety
I think this would depend on the environment. If you live in a dry, relatively flat area and you mostly drive there, you're probably right, marginal difference in road safety.
But if you live in a country that has long winters or lots of mountains, having control over the engine is a integral part of staying safe on the roads.
I live rural, wicked winters. I never buy a stick car because you need two arms and two feet to drive it. So if you get hurt, you are pretty stuck for a while until your arm or leg get better.
Where I'm from most cars are also stick, but I got my license years ago but since then I've only driven twice: once with stick and once automatic, and I have to say, automatic is so much easier to learn that I don't know why we're all still getting licenses or buying manual cars. It must be stubbornness and the slight price difference because in reality automatic is just much more practical.
I think manual gearboxes are more durable. Especially when most autos today are the CVT type which break quite often I heard. Mind you the newer manual gearboxes are usually 8 or 10 speed which is going to be less reliable because all the cogs are smaller thus weaker, and you have to change gear twice as much.
One of my vehicles even has the tranny cooler and the radiator in one unit and is prone to cracking, leading to running radiator fluid through your gearbox, destroying it. The first thing I did was install an external cooler, as the market was big enough for an aftermarket kit for such a part.
Honestly, for me, it's for driving pleasure, and for having the choice. If I’m kicked off the bus at some remote location, and all I get is an old Volvo Amazon veteran car, then I’d like to be able to drive it around. And where I’m from, you can only do that (legally) if you have the right license, which is the license for manual “stick” drive.
Other than that, it’s far less about logic, and more about the feeling of control and mastery that it gives you. True, this feeling is largely more inefficient and more prone to failure than an automatic system, but that feeling, man; that feeling is just so great!
I also live in a wintery country, with many winding roads, hills and mountains with diverse weather and road conditions. The argument was made that it’s better to drive “stick” under these conditions. However, I’m not sure the argument that manual is better for diverse conditions will hold for long, given the current advent of better AI.
I’ve already seen AI drive in very hilly and adverse terrain in the Hollywood Hills, rife with bushes hanging over junctions and pedestrians popping in and out of parked cars, and so on. Naturally, not much ice up there, but that’s a pretty difficult landscape for an AI to master, and the AI already masters it pretty well!
I own multiple cars and only one of them is an automatic, and I dread it to be honest. As in I quite often get thoughts about getting another manual instead of it, and arguably this automatic (BMW) I have is the nicest car I own, but transmission really makes it no justice.
To be fair, it is a bit older, so perhaps modern transmissions are significantly better, but any rentals I've driven were even worse, though they were lower class cars.
It’s crazy to think about. We are nearing the end of the age where people who have been programming for “20 years this year” and self-taught probably learned to program from books. In the next few years, people who have been “programming for 20 years” might start to include YouTube learners and boot campers.
As someone who learned to program from two gigantic tomes of C/C+ with a Borland Compiler, there was no copy/paste. You had to type every listing in to get it running, as well as find that semi-colon you forgot on line 16,46,102 of 200. Then you had to remember everything, take notes, and literally research answers. You couldn’t just ask a random person on the internet, at least for a few years after I learned how to program when online forums became a thing.
One big difference I can think of between my younger peers and myself: When I see an error message, I take it as a “hint that something is wrong” where my younger peers tend to take error messages literally. Probably because I “grew up” with terrible error messages that hardly ever pointed to the actual file/line the error was on.
I’ve been programming for 10 years, and as you’d expect know a lot of people with a few more or less years programming.
I think all the “kids these days” gripes only apply to people’s first few years. You can argue they’re formative, but most professional programmers will quickly need to evolve beyond “take message literally, ask online for help, paste error message into google/SO” to be good at their jobs. I don’t know anybody beyond entry level who operates like that.
I rarely take things literally like you said (though I may start by giving it the benefit of the doubt), I always just check the actual code from the thing emitting the error message, read the source of libraries I’m using, seek out documentation. It just so happens I can do all of that from my computer rather than doing some of it with a book.
I've had the unfortunate luck of working with a couple of people with "senior" in their title that once faced with an error message from a open source library, starts their troubleshooting with searching for the error message and if nothing comes up, opens a issue on the issue tracker and then wait. They didn't even consider digging into the source code of the library to figure things out themselves.
I haven't worked with a lot of people like that, but enough to guess that there is quite a bit of people who work that way.
To me, it is only an issue if they sit still after and do not start to work on something else. But I am unashamed to admit that if I don't find the error in github/gitlab issue tracker, and if it seems to take more than 20 minute to investigate, I'll open an issue then move on (if I already know the library, if it's new to me, I'll assume Pebkac for at least two hour and try to rubber duck the issue)
Depending on how much you have on your plate, this is ideal from a time management perspective. Knowing when and how to delegate is a very important skill. Just because you can do something doesn't mean that it's a good use of your time.
But it's true that after a point, the only errors left are the ones that can only be resolved by reading the documentation and experience (only 4 DELETE per hour, and you throw me an undocumented 503 error? Really?), and llm can't really do it for you.
Yeah, I've been doing the coding crap for money for just a tad over 44 years. I've literally had 4 complete software careers, each lasting a over a decade, and each being riding one of the tech waves: early 70's-mid 80's 3D graphics, then console video games, then VFX for film, then facial recognition and now AI. Since getting my OpenAI key and working with my own implementation of static Diffusion I have to say my ambition has also increased. As someone with this span of experience, I want some type of youth syrum so I can see where this all goes and participate. It's fucking fun, amazing, daunting, and engaging.
Don't forget the ones who learned to program from colorful 1980s Usborne BASIC books. (Also with no copy/paste, and having to use a tape cassette to save the listing.)
The Amstrad CPC 464 had a "copy cursor" and a "copy" key. Holding down shift and using the arrow keys let you navigate a second cursor to text you wanted to copy, then the "copy" key would insert the letter under the copy cursor at your insert cursor and advance both cursors forward one character.
The C64 wasn't as good a BASIC environment. You can overwrite text in a program listing and press ENTER to update a line, which can copy a line if you can squeeze in a new number.
Neither were as good as Acorn's BASIC.
For a couple of years in secondary school, my alarm was a program I wrote every night on my CPC 464. My tape drive was broken, so I had to type every program in fresh. If I wanted to wake up in the morning, I'd write a nested loop over the hours and minutes and start playing sound.
Ah, I almost had a knee-jerk response to remind you that the Turbo Vision IDE very much had copy and paste 30+ years ago – and then I understood what you meant.
One of my cousins learnt to program as a teenager in Argentina, before the web. He told me about the difficulty of finding programming information over there. Apparently, people who knew any kind of programming had paid dearly to get that knowledge and in turned charged others for that knowledge. Needless to say, there were not many programmers in the country at the time.
In comparison, today's generation has access to vast libraries of knowledge.
I mean yeah I read all those books and I've skimmed a few programming videos over the decades. But I've only really learned programming from working with a handful of exceptional mentors over the years. I've eaten a lot of crappy dog food on projects just to get to spend more time with those individuals.
> my younger peers tend to take error messages literally.
Interesting, because to me there still seem to be many public discussion forum questions where the answer is "Have you tried doing exactly what the error message is telling you?"
I’m talking about self-taught. I also took comp-sci after learning to program and I learned a lot I didn’t know. So I don’t want to sound like I’m dismissing that education.
Our experience coding from scratch will have about as much validity as the experience of people who coded in assembly has today. It'll be a specialized skill with its own niche but without much value in the broader tech industry.
The experience I got from coding assembly back in the day is one I use regularly today. It means I'm acutely aware of how various abstractions are implemented, and comfortable looking at assembly in a debugger to spot performance issues. The same will keep being true: Even if you're working in higher level layers (I'm mostly writing Ruby and Javascript these days), understanding the abstractions lets you make better choices.
> but without much value in the broader tech industry.
Not talking about AI; but if you want to work on compilers, linux kernel, game engine optimization, GPU optimized code and many many other things than web development you need to be more than familiar with how to work with machine instructions, albeit not working with it every single day. These are significant parts of the industry which make the rest possible.
I'm not sure how you quantify "very small" here? I'm running low-level conferences [0] and they're big enough for me to be a full-time organizer for them: with a job fair that's included Mozilla, real estate, nuclear defense and various game studios.
We could check popular job boards (or HN's Who's Hiring threads) and see how many postings mention compilers, kernels, assembly, GPU, etc. Then compare that number to the total.
Did the number of jobs of people coding in assembly go down, or is it that the number of jobs of people not coding in assembly go up? (I truly don't know. Just curious)
I tend to believe that the number of people working with machine instructions might have even increased! We have much more of the tech that I mentioned nowadays compared to 90s. Also a lot of other industries have entered digital age and in many of them, i.e aviation, transportation, financial, etc. knowing the intricacies of your hardware is crucial.
The thing is that web development has seen such a boom which has seemingly dwarfed everything else in its wake.
I describe it as the difference between "software development" and "software engineering". Or between "business" and "infrastructure". Both are valuable, but have different demands on understanding, rigor etc.
If that was all they were doing, sure.. but if they wrote language parsers, understood computing from the nand-gate up and worked their way up and down the stack over-and-over.. these types of guys will blow past all of us. The understanding you get from going up and down stacks is ridiculous.
I mean I've worked up and down most of the stack. The understanding is great. It still doesn't give me much of a competitive advantage over the average "full-stack" (laughable term given how narrow a part of the stack it encompasses) engineer when we're all working at FAANG / generic SaaS startup churning out agile code commits for generic CRUD web app.
In this AI age I'm seeing it coming to fruition. Fullstacks have (IMO) more potential now than narrow specialists because they can now get away with high-level remarks and letting the AI fill in the details.
Yeah I’d say more specifically that general first principles understanding of different areas and the ability to think about systems / architecture / general logic is very useful.
Unlike traditional polymaths, detailed knowledge / familiarity with APIs and libraries is less needed.
Exactly, this sounds like people that think knowing assembly or managing memory manually with C gives them an advantage. I learned those things too and am not a single bit ahead of a developer that learns newer languages and never does a malloc in their lives.
Perhaps the old fashioned programming skills will be applied to writing test cases for the AI generated code.
There are still areas in software development where (not systemized) failure is not an option and where AI generated code absolutely can be useful, but would need careful validation from a human. Not only for unintentional errors, but also for potentially maliciously seeded errors in the model that is being used.
Perhaps manually writing test cases for the AI generated code could meet the requirements for human validation.
You seem to be saying that as if it’s a good thing, but that’s terrifying.
The article says they “get the right answer 80% of the time”. That’s abysmal when you don’t know what you’re doing. Software which is 20% bugs is software you can’t rely on. Depending on what it does, it can be actively harmful.
Being a “magician (…) among mere mortals” is not a positive. We live in an interconnected society where the actions of others affect us. It would be nice if you can trust what other people are doing when they’re dealing with your water supply, electricity, food production, hospitals…
> Software which is 20% bugs is software you can’t rely on.
No one is saying to take the output and immediately ship it. It still needs to be finished and tested. What's important is what is ultimately delivered; less important is what steps brought you to that point.
That said I suspect good testing will become more highly-valued as AI use increases. Maybe there's a form of TDD that can develop, with the human mostly writing tests and AI prompts?
Yes, good productivity will be amplified and so will bad. There's people on my team that are super productive but cut corners, don't code in as a maintainable way etc. I usually get a slow senior dude to gate their PR until it passes. But it might just be less hassle to let the senior dudette to get more productive with an AI. We'll see...
It's possible that all this research in programming language safety, specification, static analysis that most of us have been safely ignoring for the past... well, forever... will finally become a staple requirement, instead of programming itself.
I wonder what implications this has for the reports claiming that general IQ is going down in most countries. Obviously, there are people out there who do not suffer from this trend. And perhaps they are the ones making the systems that help those less endowed? Anyway, the road to the tech religions Warhammer 40,000 seems set!
I think AI generated glue code is actually a massive win for linux as a desktop operating system. Suddenly everyone and his grandma can form programming pipes with just a few prompts in a gui program.
We are going to be vringes softwarearechologists and programmers at arms.
GPT doesn’t change some fundamental things that we know about the theory of computation as well as complexity. It’s a wonderful tool that we’re just beginning to learn about, but it’s not a get out of jail free card for the halting problem or a million line codebase that models 50 years of business evolution.
Strong disagree. Look at the crunchiest of people who are doing very low level work. I wouldn't say they are really making more than anyone else. In a lot of ways they are at a disadvantage because their skills are very niche and very specific. If they find work it can be good. But it can be a frustrating exercise to find that work.
People pay for productivity, making a product of some sort that has value. These AI tools will increase productivity. And they will only get better with time. That translates into the productivity of people who have mastered these tools as more valuable. They just get more done.
Will syntax issues and other problems be an issue with early AI? Sure, but it will get better. The AI will start to catch more issues and move farther up the stack. We still need to know enough to guide it, but the details required will be fewer.
I’m thinking of things like the invention of the automobile, typewriters, calculators, even writing. Before those were invented people did things very differently - they walked or rode a horse, wrote manually, used slide rules or basic arithmetic algorithms like long division, memorized things and persisted them through story telling rather than persisting them in writing.
People are still actively acquiring the knowledge to do things the old fashioned way in those areas, a hundred or even thousands (for writing) of years later. But typically they only do it if they need to or find it interesting.
It’s hard for me to think of any technological advance (outside entertainment) that actually caused a mean regression in peoples’ effectiveness, or for people as a whole to completely stop acquiring the skills of doing something the old way to deleterious effects. Like, has there been any invention that was so easy to use that people got worse at the task the tool was invented for? In aggregate it may happen if the tool greatly reduces barriers (like training, experience, or study) to entry, but I kind of doubt it happens for specialists.
Note by effectiveness I mean the general task, not the specific skill that may have become obsolete. I’m not effective at all with a slide rule but I have a Python shell, the internet, and hardware capable of floating point operations.
Socrates: You know, Phaedrus, writing shares a strange feature with painting. The offsprings of painting stand there as if they are alive, but if anyone asks them anything, they remain most solemnly silent. The same is true of written words.
This time its different... The written word might be able to write back. As a matter of fact, I already use chatgpt to learn new skills. Its a better way of acquiring knowledge. The job of teachers will change within 5 years.
When I read Harry Potter as a child, I would feel a sort of melancholy when I finished the book and had to go back to a reality without magic. But as the years went by, I started to realize how much our world was starting to look like the magical world... it really is true that sufficiently advanced technology is indistinguishable from magic.
Moving pictures in the newspaper? We have iPhones and tablets and tiny screens with interactive video...
Talking to people through the fireplace? We have video calls on our phones.
Writing words into a diary and having it write back to you? Well now we've got that too.
Now I'm just waiting for the flying broomsticks. Maybe we can do something with magnets inside an electrified metal Quidditch arena...
If every single person with that coding experience only coded, that's likely, but not every person who got their start that way focused on coding to the exclusion of everything else. There are at least a couple of us who are more oriented towards teaching, archiving, and knowledge preservation. Right now, people like us but don't tend to prioritize giving us their resources. Once those greybeards are in their 60s/70s/80s and looking at their work being lost forever, I imagine that will change. (Much in the same way that there's way more discussion now in tech spaces about raising kids now that more programmers are old enough to confront that issue).
That knowledge will likely be preserved by another greybeard cohort and it'll be specialized knowledge that only some people know, but it will be known. Some stuff will be lost because we'll prioritize money and copyright up until there's a crisis, but there are people prepping for this time period. It's not like trade knowledge loss is a new problem. It's just not a sexy one right now.
I do mathsy stuff and I feel this is the case with older mathematicians who did things before computers were so powerful. Those in my sample have a sharper intuition and a mechanical sympathy for maths which I admire and lack in comparison.
Not many people, but the ones who work at the bottom of the stack on compilers, OS design, embedded etc won’t be going hungry any time soon.
I think using AI to build a React app from Figma is closer than building a low level piece of infrastructure. Hence, the grey beards operating at the lower levels are slightly less vulnerable.
It is all the magic of early google, and stack overflow... Not as many ads less bullshit.
I have been coding for 25 years. Chat GPT + willing sr engineer means 2 less jr. dev's. I lived through the dot com bust, and now I can do a heroic amount of work with less effort.
This raises two points:
1. It's clear that we have too much grunt work. Im not sure if this is a failure of language design or libraries, but we need to do better at the core.
2. We continue to narrow the path to get experience as an engineer. This is great for companies in the near term (this looming down cycle), but bad for the industry as a whole long term (we dont have a pool of strong jr. engineers).
In Jiro Dreams of Sushi, the new staff starts from cooking rice perfectly first and perfecting roasted seaweed before moving on to preparing an egg sushi and then graduating to fish.
It's not grunt work.
It's how new engineers learn the ropes and gain experience doing low risk work; it's part of the learning process that only feels like grunt work to a senior dev.
I do not think that OP is arguing you should not learn the basics.
Tools like Copilot are very useful because if you need to create a function that takes a FLOAT in addition to the one you just wrote that takes an INT then its right there already. Sometimes I am just trying to bang together something quick for testing just so I can continue on with the actual project.
When I learned to write I did not just paste from Wikipedia into my essays, I would distill words from the articles I read into my own thoughts and writing. These tools are absolutely useful, but it is what you make of it.
> I do not think that OP is arguing you should not learn the basics.
That's not my point, per se.
> It's clear that we have too much grunt work.
What one might see as "grunt work" is what another might see as "learning the ropes".
You can make the case "with ChatGPT, we no longer need to learn the ropes".
But it's really hard to predict how that will affect the next generation of engineers -- I don't even know if "engineers" is the right word -- if they only understand how to ask and not how to construct.
I'd say that my point is that grunt work is very meaningful to someone who is in a different phase of their career. It's not just grunt work, but a pathway to learn.
They’re just higher ropes, instead of cutting your teeth on a Todo app you’ll be writing the apps+web+API equivalent. ie you’ll learn architecture first.
One of my colleagues at my previous lab is porting multiple Matlab toolboxes to Python. These would probably require months for couple of Master's students, but with ChatGPT it is a few weeks work for a postdoc.
Do they have an extensive test suite? I would imagine that it's quite easy to introduce subtle bugs when porting tricky algorithms and other sciency software.
Yes, architecture, medicine and metallurgy have been improved for millenniums. Next to that, IT is a baby that can barely walk. Of course we have too much grunt work.
But also because every time we remove grunt work, the customer expectation increase, the market adapts, and now you have to use new tools that are productive enough to do what the users want with the resource they are willing to pay.
So you don't have to hand write assembly but you do have to have an app that works on 16 screens ratios. You have amazing databases ready to handle your data, but you should update the UI live.
> We continue to narrow the path to get experience as an engineer.
Like in most industries, we are starting to move from "everyone needs to be an experts" to "90% of the daily tasks really just need good technicians". That's fine. That's how it is supposed to be.
At some point only experts could duplicate your house keys. Now you have convenience key stores everywhere.
We are on the right track.
Next stop, the equivalent of residential building codes, but for public facing apps.
Because IT is so new and can be done in your garage, we thought it was special. It was not.
I don’t think the code we consider “grunt work” should necessarily be eliminated. When communicating, it’s useful to have a certain amount of obvious stuff spelled out in order to ground the audience and let them understand the basic context in which the non-obvious stuff is happening.
For example, if you have a configuration, and a particular default would often (say 30% of the time) be overridden, I think it’s typically not a good idea for the default to exist in the first place. Having those defaults will certainly reduce the average config size, but they’ll also create more cognitive load for readers, who now not only don’t know what the default is, but don’t necessarily know that the choice is being made in the first place.
This is one of the things I think is so powerful about AI as a solution. It saves you from the work, but still conveys the basic context to readers without requiring them to understand a bunch of opinions baked into the language or framework.
The same applies to writing prose. When you give a bulleted list to ChatGPT and ask it to turn it into prose, a lot of the work it’s doing is filling in obvious text. But that obvious text is important to communication, and where you have to go and edit that obvious text for correctness will be some of the most important communication in the final product.
> 1. It's clear that we have too much grunt work. Im not sure if this is a failure of language design or libraries, but we need to do better at the core.
We've been iterating over that for decades, there's always one more step of grunt work to remove, one more layer of boilerplate code to move into the compiler (or a code generator), one more deployment task to automate, one more elegant new syntax to add to a language, …
Yeah, and for each piece of boilerplate we remove, we add another few layers of abstraction. At the end of this process we are stuck with crap like YAML and JSON (or, heaven forbid, XML) with the business logic being thinly spread out between logic in shell scripts, shell variables (for CD, for example), in YAML and/or JSON and/or XML within the frameworks in use (spring, mybatis, etc), multiple third-party libraries.
There is no one place where you can find the business logic.
Reminds me of a case where I was trying to help debug some legacy code. I had a simple question, what code gets executed when this method is triggered via the network request. And the final answer that i got from the developer maintaining that code was: I have been working on this for the last 9 months and I have not yet been able to find he entry point from where the requests are handled
I would hope that, in future, some chatgpt like tool would be able to answer basic questions like this and help a new developer start getting productive in much shorter time than now.
Then imagine asking that tool to summarise all the business logic applied in the project or ask questions abou which all sections of the code may need investigation to analyze a bug and come up with a solution.
In the end, I would think of these ai tools as something that can function like another junior or even slightly experienced developer in the team.
I just used GPT-4 to help build an analytics dashboard using ChartJS. There's so many settings in ChartJS, it would have taken me a week to StackOverflow / Google how to get my charts how I'd like them - it took me a day with GPT-4. I could just ask it anything and it would help no problem. Any buggy code it produced, I'd just copy and paste the error message and it would provide a fix.
What interface are you using? I've so far only used ChatGPT through the web interface. (I pay for plus). I just ask it for things that are standalone snippet-like things. Perfect functions on the standard library, or an emacs lisp snippet for converting a unix epoch to an ISO timestamp. I'm not sure I could allow it to read my company's code, but is that what you're doing when you're using as part of a large project like that?
Imo the chartjs docs are not very good. Or i'm just used to rust's docs.rs, where the whole api surface is documented. I couldn't find something like this for chartjs.
I just started looking at the docs a couple minutes ago to understand why the OP claims that it would have taken a week to accomplish something with this library. I don't get it.
How is it possible that any task could take a week or more with this library? It looks extremely straightforward and the documentation contains lots of examples.
Not sure what your background is, but in my experience, some tinkering takes ages.
Some tasks you deem to be a lot of work are relatively fast in the end.
And some tasks produce errors that send you on a journey for hours.
Straight forward looks and good docs may not help here.
I’ve found that AI prompting has been nearly as time consuming as doing it myself from the get-go. This includes content writing and visual art. Maybe I’m an edge case. I’m not a skilled programmer. However, I’m a decent writer and painter. At most, it generates quick, solid, though generic ideas. Unfortunately, I have to do a lot of reverse engineering and curation to make any valuable changes. Try writing a tech article about, say, an iPad, and it will give you a lot of generic, but well-written content devoid of any tech specs. Add tech specs and use lose the style. What should be a simple fluff piece takes a couple of hours to get the balance right. Or try creating a visual work of, say, two people running down a street, holding hands, and unless you’re very open to interpretation, many hours later, you still won’t have anything approximating what you imagined. You change the “camera” angle, and they’re no longer holding hands. You add a small detail, and the entire scene morphs, etc. I’m saying this as someone who also uses weights and negative prompts. I look forward to further advances, though I’m perfectly happy if these skills remain in the realm of humans.
I will admit chatgpt and copilot are very good at offering up bite-sized pieces of code, like particular objects and functions as well as templates.
I’ve not actually tried using it for writing help yet, though we’ve had a lot of interesting conversations. My thinking is that when I do some writing, I’ll give it my plan and then ask it for specific content or topic suggestions based on writers and philosophers I’m interested in, or other goals. Then I will write the piece myself, feed it to GPT-4, and ask it for feedback and advice. I’ll re-state my goals and ask it how I’m doing.
I absolutely would not expect it to fully write what I want, but I’ve found it very thoughtful for analysis so far in our conversations.
It's the exact same for code tasks. The AI is actually quite a waste of time…
You need to check every word of the AI for truth anyway, so you need to research all the stuff it tells you, as it could be made up, everything it tells you is very shallow, so you need to dig into the topics anyway, and making even small changed by the AI requires ridiculous amounts of explaining details in prose, instead of just changing a few lines by hand.
All in all it's much easier to come up with some more reasonable solution in a shorter time by just doing it the "classical" way: Read the docs and copy-paste code snippets from proper curated examples some human actually tested.
I don't say that an AI could not program at some point in the future. But I guess it would require AGI to be competitive with people who actually know what they're doing. But at the time we have AGI we have anyway much larger "problems" than how to write code, as more or less all human intellectual work would be superfluous.
It definitely speeds up my programming but it makes mistakes here and there (like any programmer) that need to be fixed. For me it’s a lot faster to proofread code or adjust the prompt than write it from scratch. And of course there’s cases where the AI can’t come up with a decent solution and it needs to be written from scratch.
Anyone saying programming jobs are in trouble don’t really know programming. You can’t just have someone from the bizdev team prompting ChatGPT for JavaScript code and just doing copy/paste…it would be difficult to even come up with the prompt without sufficient programming jargon knowledge.
Like the article says, the tool will allow programmers to work faster and therefore tackle more things. Expectations will go up but I don’t think the jobs themselves are in trouble.
I've definitely found this to be the case with writing. ChatGPT gets you a "valid" output, but usually a bland and lame one. I think it's better for editing and brainstorming, or for simple tweaks. Specialized tools like SudoWrite are actually more useful as part of a writing work flow.
Honestly, I keep secret some of the amazing uses of ChatGPT. I feel selfish but also I feel as if everyone is respsible for what they get out of it. We have many people and the media spending hours and tons of prompts trying to create funny or eyebrow raising headlines. I just keep my head down an marvel at the growing array if uses, many complex, that chatgpt can perform. It has been making me tons more money for just $20 a month. I could hire 10 developers and assistants and they would not be able to come close to it's technical, coding and general abilities and in 1 to 5% of the time it would usually take!
Hi, do you mind revealing some of these uses? Or, lets put it differently, I have been using ChatGPT for
- writing simple functions: e.g. I used it to generate a function that receives a name and then removes any dutch name particles ("van de", "van der", ...). ChatGPT also generated the list of dutch name particles of dutch name particles.
- solving tricky bugs: e.g. I used it to solve a tricky but in SwiftUI that nobody on stackoverflow was able to solve.
- generating ideas for apps I could develop that would make use of the ChatGPT API.
- as a replacement for google when debugging code based on error messaged received.
I feel like this is how the typical programmer uses ChatGPT, but im intrigued by your comment what kind of uses you have for it. If you dont mind, could you hint at some ways you use it?
You absolutely should. This has gone from an obscure research product to household name incredibly quickly. South Park even made a full episode about it.
Yes, I see all these people saying they've tried ChatGPT and don't understand the point or why it's useful. They've probably just not learned how to prompt it correctly.
I agree, it also helps me to get things started, sometimes when I want to start a new project I need to learn new tools and/or APIs but the documentation can leave a lot to be desired, so the mental overhead could be too much that I get bored soon if I don't get results.
I have abused chatGPT so much that I have an impulse to talk to other programs that are not the chat bot, like the terminal and VS Code, when they give me an error that I cannot understand, just to ask them "why?", like a premonitory dream I think I have an idea how chat bot could be integrated into every piece of software sometimes in the future.
Yesterday I asked chatgpt to compute the probability that two words of five letters both began with the same three letters and it could not solve it. It gave me about 10 different results but all were wrong, it seems the number five mislead the algorithm. Another question was computing the variance of the sum of 60 dices but omitting the results where the sum was greater than 200, the reasoning was incorrect in all cases, finally I suggested creating a simulation to estimate the variance in R, but it made an error about the var command in R suggesting that it has an option for the degree of freedom.
>
Yesterday I asked chatgpt to compute the probability that two words of five letters both began with the same three letters and it could not solve it.
IMO, this is irrelevant, because "normal people" would never ask this question. Because it has never been asked, there's also not a ChatGPT answer to it, because its answers are based on already existing information.
This thinking that ChatGPT is AI is just wrong. It's not AI, nor GI, so stop comparing it with that.
As a human, I can destroy your problem/question with one claim: you claim that the words must be built on the English alphabet. Let's throw some Æ, Ø and Å in there, shall we?
I really see it kind of like having a good junior programmer or intern helping me out who has an incredibly broad amount of knowledge. Sometimes they need some coaching or gentle correction, sometimes you say, "I'll take it from here", but inevitably there are those times where you think, "Great job! I'm really impressed!"
Are you serious? That's your gotcha? It's just doing some math over a dictionary, it doesn't matter the language or characters. So yes, a human can solve a novel problem.
I've seen stuff like that too. I asked ChatGPT a tricky geometry question and it got a ratio reversed and ended up producing wrong answer.
I have also tried to use ChatGPT like Google search at the beginning by asking short, curt questions, and got underwhelming results.
The trick is not to try to trick ChatGPT but pretend that you are an idiot and it is a very sophisticated colleague who just needs step by step details. Then you can also generate some of the seemingly amazing results people are getting. For example yesterday I used it to scrape physician information from my insurance website, followed by their Google ratings and so on.
I prompted GPT-4 with this and I think it solved it. The first time it assumed the words were generated with random letters and solved it like a math textbook problem. Then I prompted it with this:
Ok, I see you calculated the probably using randomly generated "words" from the letters of the English alphabet. I am interested in the actual probably of two real words in English that are 5 letters wrong share the first three characters.
I am a Python developer, so I will understand it if you give me a Python script.
I gave me this which looks right to me:
import nltk
from collections import defaultdict
nltk.download('words')
from nltk.corpus import words
# Get the English words
english_words = words.words()
# Filter the words to get only five-letter words
five_letter_words = [word for word in english_words if len(word) == 5]
# Create a dictionary to store the count of words with the same first three letters
words_dict = defaultdict(int)
# Count the words with the same first three letters
for word in five_letter_words:
key = word[:3]
words_dict[key] += 1
# Calculate the number of pairs with the same first three letters
same_first_three_letter_pairs = sum((count * (count - 1)) // 2 for count in words_dict.values())
# Calculate the total number of pairs from the five-letter words list
total_pairs = (len(five_letter_words) * (len(five_letter_words) - 1)) // 2
# Calculate the probability
probability = same_first_three_letter_pairs / total_pairs
print(f"Probability: {probability:.4f} or {probability * 100:.2f}%")
The only thing I dislike about d3 is the boilerplate code to hook it up to a div and get the first lines to render, so yesterday I did a:
Assuming I have the following data: ```{ "status": "ok",
"data": {
"a": [[1680079200000, -199], [1680079800000, 92]],
"b": [[1680079200000, 248], [1680079800000, 259]]]
}
}```, where the first item in the array is always the unix timestamp in ms, and the second one the value of the datapoint. Also assume that I have a website with an HTML element `<div id="graph"></div>`. Please show me the code used to draw a graph of "a" and "b" with d3 into the `div` element.
And it generated all the code I hated to "create".
Then little things like "Ok, I want the time not to be in AM PM, but in HH:MM" which used to mean to visit a couple of pages, the answer's right there.
Yes, agree. I mean GPT-4 is an all in all a fantastic tool, but it's especially good for developing. The moment you transform a blurry idea into structured data in any form (and it helps a lot with that: you can chat like two decent citizens and ask it to transform it to json at any given time), you can pretty much generate infinite boilerplate that's consistent through a whole application: interfaces > tables > migrations > models > forms > tests. But it goes way beyond crud, and during that iterative process you can come up with some sophisticated enhancement that you wouldn't have even entertained and all of a sudden it's easy to implement. It's great for longform code & structure generating conversations and it's great for picking up little details that would have taken the most of the afternoon. I'm in awe, really (as is everybody else?).
And yes, it does make mistakes, very rarely, and you can't really upload the whole source code everytime you want to talk with it, but it makes the programming dream real again (if only for a couple years lol): is not like you don't _type_; there's so many things you stop thinking about. And you can say: oh, there's the catch, but the truth is my own buffer is so much freer all the time, I get to tweak files, with excitement, that a couple months ago would have been very tedious to get a grasp of.
And about code quality: its Elixir is alright, its Typescript is excellent, but it's absolutely flawless at concatenating unix commands: it has read all the man pages. I've aliased more custom commands these couple weeks than in the whole of last year, and my workflow is getting tighter and tighter each day. There's just so much power into this thing.
I tried using ChatGPT a bit with the GPT4 model and it seems quite promising, but also very slow. When it's writing out the same function again with a change I asked for, it reminds me of the old days when connecting to a BBS with a 300 baud modem.
No, that's how it is. My MO is now I type in the question to the chat; go to another tab to google a bit on the background of the problem; by the time ChatGPT is done, I already know enough to have a reasonable idea about whether what it came up with is in the general direction of where I want to go. And if it is, I'm usually confident enough that it gets the details then mostly right.
One meta comment is that I would love for something like a Greasemonkey script that can automatically add some checkboxes or something that will then automatically add stock phrases to ChatGPT questions, like 'show me only the code', as I find that a lot of the intro fluff and the explanation of things are useless to me and just take up time. In general, the UI is pretty bare bones, there's a world to be gained there.
Check out alternatives like ChatWithGPT[0] or just the OpenAI chat playground[1], especially if you've gotten access to the GPT4 API. Populating the system message really makes a difference and you can enforce a lot of style of both code and output. The third party UIs also have a lot more bells and whistles.
Every generation faces this sort of issue which is learning skills for the current technology which then get displaced in 20-30 years. I think there's always a worry we're going to lose some knowledge in the process but equally the productivity gain seems to be 10x and it leads to dealing with higher level problems. Very few people are kernel hacking now as a percentage of the overall engineering community.
I guess the question is. What do we build with this form of engineering. It's so very early. We're using it as a way to augment current systems but I think we'll build for entirely new things soon.
Its just that usually the new tools and technologies that have this effect are accurate, deterministic, or at least try to be or pretend to be.
So, its quite literally an issue of lost knowledge. Is like replacing wikipedia with something that answers faster, but without sources. Its cool in the short term, and people are more productive at learning things, but you also go back to the hear-say time before books were invented.
We have been told that computers will do our (software engineers) jobs for decades now. It has never come to fruition, and this is no different.
What we are seeing is that the number of software engineers has increased dramatically and the average skill level has plummeted. It is not normal for a competent software engineer to need to reference Stack Overflow a dozen times per day every single day.
The other thing that this is bringing to the forefront is just how atrocious the libraries are in certain ecosystems (e.g. JS/TS).
If a tool that can write incorrect functions/methods makes you substantially more productive, you were never very valuable in the first place, and "AI" likely isn't the only thing threatening your job security.
Roughly 80% of the time, currently. Which is great! But if you don't keep that in mind, you risk making a fool of yourself.
It'll most likely improve in the future, I'm sure. The amount of real-world progress this will bring to humanity boggles the mind. I think we'll see progress in many and as-of-yet unthought of areas, because the power of mathematics, algorithms, and advanced systems will become available to a lot more people due to this "helpful uncle".
I do hope they work on improving its math skills though, also in the sense that it can explain the steps taken when solving equations or algorithms, or even how algorithms could perhaps be "translated" and implemented in various languages. In my opinion this is key to understanding mathematics, and a whole host of other systems as well. Moreover, it helps people who previously didn’t have the capacity to understand these things, so the effect on “world knowledge” is cumulative.
Personally, ChatGPT has been a great help for understanding many things a lot better, everything from mathematical equations to subsystems in Linux. And of course, it has helped me improve my coding.
Id say it has approximate as high a chance to work as "random code from the internet". Also, it is often just relying on an old version of an API or a Library.
This isn't really my point. My point is that you can't ask the documentation to expand upon what little is already in it. Stack Overflow works a little better for this purpose, because you can ask follow-up questions and get more than one answer. But even there you have to wait days for answers, and sometimes you don't get any at all. That's really inefficient, and perhaps you forgot why you asked in the first place, or just gave up and moved on.
Not so with an AI chat system. You get the answer immediately, and you can ask as many follow-up questions you like, to your heart's content. This means that, sure, even if the code you get from ChatGPT has approximately as high a chance to work as "random code from the internet," you can't ask that "random code from the internet" to expand on why it's good or bad, why it was made in a particular way, or if there are better ways, or if you can error check it one more time, or complain that, "Hey, random code, your dumb code didn't work!" In that way ChatGPT is already leaps and bounds better than "random code from the internet."
Also I assume here that you aren't just throwing darts at the wall, but instead testing out code with a clear goal in mind. Then it can be wise to ask ChatGPT about the underlying system, which it will gladly answer, and also immediately.
Two big hurdles for ML: a) explainability b) accountability.
ML-enhanced development neatly circumvents this: explainability and accountability is passed on to the developer. This includes bugs and license infringements.
I have no qualms about ML tools in development. But so long as the buck stops with me, I prefer to write from scratch.
One of the things I've found is that at least with GPT-4, you can automate a lot of infrastructure. This can make it much easier to get started with new project where you need to set up various servers, containers, VMs etc. I haven't fully automated it yet, but I've done it enough manually through copy/paste between ChatGPT and the terminal to see it that would work if it was hooked up to langchain. You can create prompts to make it set up VMs, docker containers and then install and configure all kinds of software.
To make it work in most cases it should follow the same stuff as this implementation of a Reflexion agent for SOTA Human-Eval Python results, but use it for infrastructure. GPT-4 can generally figure out how to create tests from the documentation that will make it know when the task it finished correctly.
Upwards of 80% of the effort in commercial software development is maintenance.
You could speed up the remainder dramatically and make very little progress on the rate progress happens and have people scratching their heads about why.
Maintenance is already perceived as one hell of a cost center.
I'd have enough money if to train my own foundation model if somebody gave me a nickel for every time somebody underestimated the time to rebuild an existing e-business system by orders of magnitude because they were ignorant of all the "requirements" that were satisfied by the existing system that were undocumented.
The most expensive maintenance problem comes when you've accumulated a lot of data that is incorrectly structured and may or may not be possible to migrate to a new, correct, structure.
Fred Brooks in No Silver Bullet pointed out that software development involves multiple phases, say 5 of them (requirements gathering, design, coding, testing, deployment) and that even if you could reduce the time involved in one of those to zero there is no "revolution".
The corollary to that is that the opportunities from ChatGPT come out of identifying the things that ChatGPT does poorly. For instance it might be just as important to turn code back into requirements as to turn requirements int code. There was a post the other day about a "simple" problem in CSS that turned out to be hard because of the deep understanding required
the ratio is much worse between spending 3 minutes to make a partial solution with ChatGPT and 3 hours to really solve the problem as opposed to 30 minutes to make a partial solution manually and 3 hours to really solve it. That is all the more reason to clean house at the lower levels, either make a style sheet language that is not so fraught or maybe make one that's designed so that chatbots can write good code for it.
There has been hype over "no code" and "low code" tools for the longest time and people have been chasing that revolution for a long time and it's astonishing how many innovations have been forgotten (like the 1990s CASE tools that could turn code into diagrams and back again and leave formatting intact.) Real progress involves solving all the problem that is ahead of us instead of cherry picking the parts of the problem we want to solve and thinking wishfully that will be sufficient.
ChatGPT is great when the time consuming bit is googling the details of libraries, APIs, languages, algorithms, etc, and wiring them together in the right way for a small bit of functionality or some boilerplate. But when the time consuming bit is thinking about the best way to solve a problem / introduce new abstractions in the context of a larger system, it seems less useful. Maybe this will improve with the context size.
I am using it in exactly the same way. I have access to ideas and code that I never would have tried before.
These other well-documented experiences that sound super perfect don’t match mine, though I’m a fan! Here’s how it goes for me.
I am a premium user and all, but it still bails on me every single time it’s going well, by writing half of a function or saying “network error”, then losing context.
I have to say “you stopped generating that last part, please continue and don’t bother qualifying.” Then it wastes more tokens apologizing anyway.
Whoever trained these models to apologize and say “as a large language model, I can’t promise that I know how Bjarne Stroustrup would refactor that bitshift, and yada yada” really wasted a lot of our productivity. And saved no one from any dangers of AI.
I can definitely relate to your experience. Just last week, I leveraged AI tools to develop some data processing code. Estimating how long it would have taken me to write that code without these tools is a challenge because, to be honest, a year ago I wouldn't have even considered attempting such a project. It might have required a week or even two of my time, which I simply cannot spare for this kind of undertaking. In the absence of these powerful AI tools, I would have opted for a more manual approach, like analyzing the data using Excel. It's fascinating to witness the impact AI has on our productivity and the expansion of possibilities it brings.
Have had similar experiences. I found two HTTP servers running on my TV and I wanted to see what was going on in them.
Dirbuster wasn't packaged for Debian (that I could find). I wasn't too keen on trying to build from source or anything like that. A few months ago that would have been that.
But with GPT I just asked it to write a POSIX-compliant C port of dirbuster. Less than 10 minutes later I was scanning my TV for routes.
I've noticed it as well. The most recent example I can think was with gitlab webhooks. I need to manage (create/delete/edit) a lot of web hooks across a lot of different projects. I normally would have probably just started doing it by hand in the web ui, but instead I had some python code from gpt to automate it in a few minutes.
I used to build random weird things all the time (mostly centered around connecting physical devices up to the internet to accomplish something silly) - but lost my mojo once I had kids and didn’t the ability to pull all nighters to “hack” something together.
Now I feel like the barrier might be low enough that I could create again. Will have to try this out and see how it goes.
Off topic, writing code/articles using AI is cool and all, but it suddently occured to me, what if we want to study pure-human generated content and distinguish AI generated ones?
Is there a reason why gpt-3.5 and gpt-4 all using pre 2021-09 data? Is it because gpt-3 as a base model was trained at that time, or is it really because bots are "contaminate" the Internet from then? I guess we can never go back now.
One thing I am certain, after 2022-12, AI might enforce itself by learning AI-generated content, and perhaps someday, AI may deem human content as somekind of "bias".
Am I the only one that isn’t blocked by coding speed (I can MVP very fast with lot of techstacks) but with … deployment?
Don’t get me wrong, I am a senior DevOps, know my terraform/CDK/… and AWS/heroku/digitalocean/hetzner/vercel/… stuff by heart, so it’s not missing knowledge.
But I really hate the maintenance work down the road. I just don’t want to get emails about some deprecated stuff from some provider and have to do anything just so stuff keeps running. I would pay a premium (!) for a stability guarantee not in service uptime but toolchain reliability :/
Me too. I can code and test pretty fast, but because of the policies where I work, it takes a long time to actually ship a change, and often we only catch bugs then. I’m not talking “time to create and deploy a binary to a cloud region through an API” I’m talking “we have to roll the binary out piecemeal at a slow rate, on purpose” over a little under a week.
So basically, a bug can set you back a long time (multiple weeks). And to reduce the risks of bugs we usually need to ship incrementally. And because bugs are such a hassle we have to monitor and test the crap out of everything, which of course we should be doing anyway, but we do it more than we probably would otherwise. But of course bugs do still happen, and even someone else’s might set you back. A flip side to slow releases is releases tend to batch more changes. And of course that makes finding a bug more time consuming.
So a feature that could take me a few hours to write and test at my top performance basically needs to be budgeted at a month of work. Of course, we have other responsibilities like testing, monitoring, analysis, support, design, coordination, helping others. But dragging out completing a project drags all that out too. If I could ship to staging same day and prod 1-2 days later, I wouldn’t need to plan so much or spend so much time going back and forth about design. We could just meet up about the change and do it that week.
And you might think you could simply parallelize/pipeline more tasks, but overdoing that sets you down the path of overextending yourself. If something important goes off track you need to do everything to get it back on track - if you don’t, it’s going to be really late - and overcommitting eliminates the slack that makes it possible.
Truly, this is something I think about all the time and probably the least favorite aspect of my job. Anyway, unless chatgpt can help me write a persuasive argument that convinced my company to completely 180 its engineering policies, it won’t make me much more productive.
I too look forward to being able to tackle larger projects in the future with AI. However, I don't trust sending my code to a third party over the internet so would want to host any AI I use locally.
I asked ChatGPT (the free one) today to translate me EBNF rules into a recursive descent parsing function in Rust and it did it correctly, it also then applied various refactorings I requested to make the code cleaner.
When it comes to boilerplate these tools are game changers I must say. Harder problems they crap all over themselves but that's ok, that's where we still have a job for the next 5 years (I hope).
I can take care of the coding part - after all that's where the fun is.
What usually drains most of my energy, as someone in the JS ecosystem, is the sheer volume of modules I need to sift through to find one that does what I want.
The other day I wanted to find a library for handling zlib in the browser - eventually I stumbled on `pako`, which is great.
Seeing this post I asked ChatGPT a somewhat vague question, namely: "What is the best npm module to handle zlib compression?"
It told me about the built-in zlib module, but the last paragraph had this:
"There are also other npm modules available that provide additional functionality or a different interface for handling zlib compression, such as pako and node-zopfli, but for basic compression and decompression using the zlib library, the built-in zlib module should suffice."
Now I feel dumb that I haven't used it for this purpose earlier.
This is one of the few times I've read a post about using GPT for dev work that has actually made a somewhat convincing argument.
In my day job, there's been various attempts to use GPT to speed up some work, which has generally resulted in me having to fix "confidently incorrect" code. The outputs usually about half right when its usable.
Using an LLM to basically auto complete trivial bits of code that are "just typing" but usually require a referral to the docs is probably the optimal middle ground for productivity - so time can be spent creatively solving the actual problems.
I'll have to experiment with it further - have it emit all the boilerplate for something, do the fun parts myself, then check for correctness might well be the workflow best suited to GPT.
>> The outputs usually about half right when its usable.
For data projects/pocs/mvps using python its saving me lots of hours, you have some tiny mistakes, but once I have the idea in my head I just iterate and in 30min or less I have "written" the code for what would take me many hours.
I think over the weekend I'll see how it handles some trivial-but-annoying/tedious data mangling tasks.
There's a few small projects I have been procrastinating for ages that just will require a lot of "tedious but trivial" hand jamming of 90% boilerplate that may be well suited to LLM generated code.
Let me tell you something. I am a product designer. I know we are doomed to be obsolete.
Not only that, but I will use ChatGPT to develop my prototypes and small scripts.
And finally, will find a time to build some ideas with Dart/Flutter. Something inside is telling me that in a generated future, people who can code will be able to create free from no code tools and SaaS silos. And this is enough motivation for me:)
I see a strange future in which product designers (who will spend less time on decorating and fiddling with DS components) with the help of ChatGPT will fulfill the role of junior devs in the production pipeline.
I agree with this. I have been trying to learn and use swift in a personal project. I have no experience with app development or Swift. Googling everything would have taken way more time in comparison with chatGPT. Most helpful thing was I can paste some code and ask for explanation. Googling is harder when I don’t know what the name of something is.
I’m wondering if the archive missed something or what this random statement from the system is about. It seems to be telling itself who it is and setting a restriction on what it knows. I know if it’s not trained on data past a certain point, it cant know it, but still i wonder if you could intercept that and change the cutoff date (if even possible), would that have any effect on anything. But then again, I am probably reading to much into it because it seems to be coming from system.
2023-03-26 04:10:49 - [system] : You are ChatGPT, a large language model trained by OpenAI. Knowledge cutoff: 2021-09 Current date: 2023-03-25
That's called the system prompt - it's present at the start of every conversation, just to feed in some hints as to how the underlying model should behave.
If you play with the ChatGPT API in the OpenAI Playground developer tool you can set your own system prompt. I used it to create a sentient cheesecake that could answer questions about SQL using cheesecake-based examples yesterday.
I noticed that he made it possible to export JSON and SVG files from his one-page dungeons. I decided that was enough to turn them into an actual mini-game. This app takes his procedurally generated static maps and makes them interactive.
There's nothing here I did not already know how to do, but I leaned on Chat GPT to generate code. e.g.
Me: As the avatar moves around the map, it would be nice to center it. Possibly zoom in and such. I'm thinking that the `transform` attribute of the map could help.
Chat GPT: Yes, you could use the transform attribute of the map to achieve this effect. The transform attribute allows you to apply various transformations to the SVG element, including scaling and translation.
To center the avatar, you could use the translate transformation to shift the map so that the avatar is in the center of the viewport. For example, if the avatar is at position (x, y) on the map, you could center it using: <code snippet>
To be honest, parsing through the SVG was just so tedious I would have moved on and not completed the project. As it was, a few nights. Boom! Little game toy thing.
Some other tasks I had for it that saved tons of time:
* Given these css rules below, can you combine them so that each is declared once?
* Could you suggest a mysterious, dark, but not black color for the background of the dungeon map?
* Could you give me a nice CSS dropshadow for the avatar?
* Could you suggest several Google fonts that would be good for a dramatic Dungeon title/headline? Please include links.
I wonder how many coders have done exactly this in the last couple of weeks? Using the api to cater the chatgpt system to our liking. I’ve done the same and it’s changing the game for me. Saving hours of time jumping through that small part of a project that I knew would be the bottleneck.
I’ve gone from saying that I’ll need an hour to do something to only needing 15 minutes.
However, I’m good enough at what I do that I can find the problems in the code or notice something out of place or that could be done more securely.
I fear many won’t and lots of junk code is going to find its way into our lives. I worry about things like this when it comes to, for example, Wordpress plug-ins, or add ons to other larger tools that we use.
If by AI we mean things beyond just ChatGPT sure it will, it’s only a question of how long before it happens and how many new jobs will be created to replace the old jobs. For example, AI isn’t going to enhance customer service, you’ll just have better automated experiences and businesses will hesitate less to use them.
Edit: AI isn’t being created for the benefit of mankind. It’s being created in the pursuit of money. This isn’t a utopian fantasy world where we look out for everyone’s best interests, that’s just a side effect most of the time.
Beyond ChatGPT and whatever its iterations are, it will be a matter of time before jobs are irreversibly changed - no one is debating that. But too many people are taking the view that it will destroy more than it create. I think it will create more than it destroy. In many cases, the type of jobs we can do will diversify and make life more accessible.
Exactly. Let say AI makes us 100x more productive. What would happen?
1. 99% of devs will be out of work and the current work will be done by 1%.
2. More and better software will be developed. About 100x effort more.
I'm pretty confident the 2nd will happen. I already have lots of work that will take a long time, and more after that. 100x improvement on productivity would mean more, faster and better (because of more tests etc) output.
I'm of the thought that #2 will happen. #1 will never happen - there are simply not enough developers to go around.
To use a relatively modern analogy, AI-enhanced development is to developers what Photoshop is to photographers. No one ever said Photoshop would put photographers out of business. Now Photoshop is a requirement for photographers. The pictures still need to get taken; likewise, for developers - the business requirements need to be captured.
AI [in its current iteration] cannot do that.
As developers, our jobs are safe. Jobs will not be lost at a massive scale; only those unwilling to learn a new tool will be replaced.
>To use a relatively modern analogy, AI-enhanced development is to developers what Photoshop is to photographers.
Photography as a job has been hit heavily by modern smart phone advances. Heavily. It's now very hard to get paid a living wage as a photographer. With mid journey it's now even harder. Photographers weren't completely replaced. I would say they were mostly replaced because now every Tom dick and Jane can take photos as good as any professional photographer.
>AI [in its current iteration] cannot do that.
All these changes happened within 6 months of progress. The technology has not stabilized. The current "iteration" will be obsolete very, very, very soon..
>As developers, our jobs are safe. Jobs will not be lost at a massive scale; only those unwilling to learn a new tool will be replaced.
Only for the current iteration of AI. Can you say the same for the next several iterations of chatGPT? No. Given the trendline, if it keeps continuing finding a job in software will be like finding a job in photography.
The way I see it: smart phones just made photography more accessible to everyone rather than for those who can afford $5k photography kit and a darkroom. Stock and news media photography went the way of the dodo, but there's still plenty of room for photography as a creative endeavor.
No matter how much you throw at it, you cannot replace the human behind it. It is too difficult.
Our jobs as developers are safe for current and future iterations of AI. The way we develop software may very well change, but we will still be developers. It's very well possible that software will go the way of photography. But that only means software is in the reach of everyone who wants to try their hand at it. It is a net good.
But that's not likely to happen anytime soon. So yes, our jobs are safe. Mine is quite safe. For quite a long, long time.
I know a few photographers and they have plenty of work and make nice money. The pictures that they take can't be compared with someone taking a picture with a smartphone.
This article and discussion thread has been really insightful. I've been avoiding chatgpt because I feel like I will not learn as well compared to prototyping and agonising through documentation.
I'll take this mentality onboard to more ambitious projects.
I agree, I've gotten stuff working that simply would have taken too much time referencing manuals. You can just keep asking questions until things start to work. It's rare GPT-4 has given a complex function that fully works, but it has done a great job recommending packages; algorithms;frameworks; commands, answering small scoped problems, answering trade offs, telling me what some github packages are doing under the hood.
Especially working with other languages now is a lot easier, because I can ask to translate between languages I'm familiar with. Same thing with configurations, eg translate this systemd script to openrc.
Ok, so when people say that they use ChatGPT for coding, they use it as a code snippet library? Here, a function that has been probably written million times before in some similar form, and a template for a simple web app.
I am really struggling to make chat gpt write anything near functioning for the projects i am working on. Then i read the code people share and claim is made by ai and understand why. It’s really basic code that instead of copy pasting from stackoverflow is copied from a text generator. Perhaps it may evolve as a tech but right now it’s at the stage of pareidolia but instead of seeing faces people see intelligence where there is none.
This isn't an example of a coding problem, but just yesterday we had a very difficult technical problem come up that one of our partners, a very large IT company couldn't solve. None of our engineers knew how to solve it after trying for a few days. I would say I'm an expert in the industry and have been doing this over 20 years as a developer. I thought I knew the best way to solve it, but I wasn't sure exactly. I asked ChatGPT(GPT-4) and the answer it gave, although not perfect in every way, gave pretty much exactly the method we needed to solve the issue.
It's a bit scary because knowing this stuff is sort-of our secret sauce and GPT-4 was able to give an even better answer than I was able to give. It helped us out a lot. We are now taking the solution back to the customer and will be implementing it.
A few additional thoughts:
1. I knew exactly what type of question to ask it to get the right answer (i.e. if someone used a different prompt maybe they would get a different answer)
2. I knew immediately that the answer it gave was what we needed to implement. Some parts of the answer were not helpful or misleading and i was able to disregard those parts. Maybe someone else would have to take more time figuring it out.
I imagine future versions of GPT will be better at both points.
this is exactly the kind of post that folks who say "oh - it wont take our jobs - our jobs are safe (way too advanced for silly AI)" - need to be reading. You're an expert, and it answered an expert question, and your colleagues couldnt do it either...after a few days...and this is just early days.
I don't think this is the take away. I'm in a similar situation as the GP, but the crux is that we still need people to make the decisions on what needs to happen - the computer 'just' helps with the 'how'. You need to be a domain expert to be able to ask the right questions and pick out the useful parts from the results.
I'm also not so worried about 'oh but the machines will keep getting better'. I mean, they will, but the above will still remain true, at least until we get to the point where the machines start to make and drive decisions on their own. Which we'll get to, but by that point, I/we will have bigger problems than computers being better at programming than I am.
I look at it differently. If what we've been writing can be replaced by a machine, that leaves us with coming up with more creative solutions. We can now spend our time more usefully, I think!
I use GPT-4, it's like pairing up with a junior developer who read all the documentation.
I follow the same steps as always to design the code, but when it's time to implement something, I ask the bot to do it, then I review it and move on to the next function.
To give an example, I had 3 things on my hobby to-do list that fit together: learning some data compression algos, learning Golang and testing out Chatgpt/Copilot coding efficiency. So I downloaded slides and exercises from a course of my old uni, and started implementing the exercises in VSCode with Copilot. Whenever I had a question, I would consult chatgpt. Whenever my Copilot code didn't work, I'd go through it line by line to see what's wrong and paste it to ChatGPT to see if it knows what's wrong.
My experience:
- Copilot did a decent job at suggesting functions when I typed out comments. It got progressively worse as the compression algorithms got less common (eg. Huffman vs. entropy coding) The smaller the functions, the more manageable it was. <- You need to write good tests, you want to step carefully through every line of code it writes.
- ChatGPT got most things right, and a few things really wrong. It was always super confident. <- Super dangerous if you don't double check, basically only usable as a refresher on a topic you are already good on.
- If given a piece of code and asked "How would you suggest to refactor this?", ChatGPT gave mostly very useful ideas. <- This is something that I will keep in my workflow.
- The bigger the project got, the worse the overall codebase looked. It became a mix of styles, pretty inconsistent, more subtle bugs were introduced that took me a while to figure out. (The nice thing about lossless compression is that you know if the code is right by using it)
- My "productivity", how many lines of code "I" wrote, was way beyond anything I could usually reach. I do also feel like I got a very quick start into Golang with this, much faster and broader than reading documentation or doing a getting started. That said, that knowledge now definitely needs to be refined in order to use the appropriate concept at the right time. Comparing my code with more professional Golang code bases I spot a lot of things that need to be improved.
Posting some recent prompts as examples as I've started to use it more for coding:
- I want to create a Google Docs addon using AppScript which has an interface sidebar. Please describe the necessary steps to create a sidebar interface. The sidebar interface should have a title with the text "Google Docs JSON Styles", it should have a large textbox which can contain multiple lines of text (fill it with lorem ipsum), and it should have 4 buttons placed horizontally underneath the text box, with the captions "Save", "Apply", "Copy", "Paste".
This produced the necessary Javascript + HTML + external steps to set up the project.
- I'm building a website using a React frontend hosted on example.com with a Django REST API backend hosted on api.example.com - Authentication is handled via Firebase Auth, currently using an Authorization header provided for API calls. Now I want to have authenticated links such that the user can click a link to access an authenticated download via the API. What mechanism can I use to create a link to an API endpoint which will require the user to be authenticated?
This gave me a the frontend and backend code necessary to set up authenticated links, with several alternate approaches following further prompting.
- Given an owner id, repository id, commit id, and private access token; give me a URL that lets me download a ZIP archive of a Github repository.
In all of these examples I received the desired result with helpful descriptions.
As a counter-example:
- Write a program in Brainfuck that outputs the string "CHATGPT".
In this case, the result is the Brainfuck program for "Hello World", with a step by step break-down of the program explaining confidently why it would output "CHATGPT", while being entirely wrong.
For me it’s pasting a snippet of code I’m struggling to debug and asking it to help. Sometimes it works, sometimes it comes up with an answer that is also buggy or doesn’t work.
I find it much more useful for soft things like writing a complaint to a vendor for me, dealing with customer service, or cover letters from a job ad that I only have to slightly tweak. Takes me five minutes instead of 30+ and I got a few compliments about my “outstanding” cover letter.
This is not my subjective experience. It's better, sometimes somewhat, but also much slower. I'm not sure which one is the better tradeoff for real work. I have it on 4 by default now, don't want to think about this for every question I ask, but for my work, 3.5 was fine.
I tried to make GPT-4 create an algorithm for an enhanced tic-tac-toe game (think Wordle vs Quordle to somewhat visualize the difference) and it's failing miserably.
Also it’s not that it writes perfect code straight away and you never have to re-prompt or change anything manually. But it’s still an insane speed-up.
Yes, which is funny because there was an article on here with some pretty hard data which showed not much difference or maybe worse performance from GPT4 than 3.5-turbo.
You'll likely refute that as your mind is already made up, but there you go, another conflicting and confusing data point.
What are talking about? Just compare the output of a 3.5 vs 4 yourself for a problem you are interested in, it’s a single click in the interface.. Do you always need a study or an “expert“ to make up your mind?
Benchmarks are good. You may be a less experienced software engineer than others (or maybe more experienced?), then you will tell me “ChatGPT x is insane bro", but that's only a matter of perspective. A benchmark gives us facts, outside of our own experience, not opinions.
I've had ideas for web apps that I've been wanting to build for ages, but my coding skills are nowhere near good enough.
Last night I sat down with ChatGPT and within a few hours we'd built the foundations of a fully functional app.
It's a fascinating process – it often leaves out vital information or doesn't get things quite right and it's obvious it's writing the most likely code rather than something written specifically, but it provides enough to get me 80% there so I can finish up the rest. Felt like collaborating with a forgetful, distracted genius.
I think that Chat GPT will make programmers jobs more tedious but always in demand. That is the leakiest abstraction of them all people. There will always be subtle bugs to fix 2-3 levels down.
Yesterday I was writing some Selenium IDE script that could warn me when 5 year CDs at more than a given APY appear on my Schwab, fidelity or Vanguard broker. CDs available there, especially after a rate hike speculation, are not available online to the general public outside of brokers and certain market makers.
ChatGPT certainly shortened a lot of the scripting head time. Though it was not quite always accurate it did help me a lot.
Recognize the same here, lot of ideas become easier within reach. I am struggling however with the costs, where at a larger scale processing is needed.
Same here. I've done a few little projects with ChatGPT4 in my back pocket. I've been able to give more aggressive timelines, more features -- and I've been able to hit those goals with ease. Probably the first time in my life I've been able to over promise AND over deliver.
Also, the final product (with documentation) has been complete and pretty amazing.
GPT-4 is absolutely incredible but the usage-cap at the moment makes it annoying to work with.
8 questions per hour (25 per 3 hours), is not a lot to work with for paying users - it started being like 10x higher and they've announced it may go down even further.
I wonder if Azure can scale with the insane adoption rate and if it's profitable at the moment?
What's the state of the art for combining these two types of AI (super auto complete and "ask a question you'd otherwise have to Google") in vim?
Ideally I want to be able to highlight some code and then run an AI command on it and also provide it as context to a query without leaving vim or losing syntax highlighting.
I am curious at what stage the developer will loose the ability to understand code and pray that ChatGPT does not make a mistake.
I admit that I am using ChatGPT to write little snippets here and there but I am an old fart and was writing software for 40 years already. I am probably safe from loosing the skills bar going senile ;)
As a beginner who started learning how to code 8 months ago, GPT makes me way more confident about tackling bigger problems. It also makes me doubt my own output less.
I understand that this isn't the best way to learn coding, but at this point, I'm looking for results, not to get a job.
For the example saved conversation, if your own service uses the openai API, you can change the baseurl to https://oai.hconeai.com which provides most of the functionality for conversation analysis
I'm currently filling my Emacs configuration with custom commands since with ChatGPT all I've to to is to describe the elisp atomic function. It's a lot easier to customize Emacs and I'm even learning new functions and pattern for elisp.
It’s worth noting one important caveat: GPT-4 cannot do super complicated coding.
Don’t get me wrong; I use frequently when coding but it can’t handle the complex stuff; it’s insanely good at boilerplate and low-hanging fruit though.
I am going back and picking up old projects that I had semi-abandoned and now have the confidence to continue using ChatGPT as a second pair of eyes and it seems to work much better with GPT-4 than the previous models.
By old projects do you mean telling ChatGPT to go and look at your codebase first? Or it's just a matter of knowing what topic you wanted to pursue and just asking those questions and shaping it through further questions?
Thank you so much. I am so excited about Ai and there are much negative sentiment around (which I also understand). But it is refreshing to read some optimism around the topic.
Anecdotally, I've avoided quora as much as possible when clicking but was pleasantly surprised the last time I did, they offered an LLM answer over the top human one which seemed reasonable enough to give an overview of the matter without bias.
The recent news and hype around AI motivated me to create a project on my personal computer with limited resources, using minimal code[0] to combine ChatGPT[1] with computer vision. This efficiently extracts structured data from virtually all websites I visit regularly, even though the resulting text may be somewhat poorly formatted. Despite the text's imperfections, ChatGPT is still able to provide me with summaries and insights. For instance, it can compare prices from various e-commerce sites, detect discounts and different payment methods, analyze soccer match results, and even generate humorous morning news summaries based on the top headlines in spanish!.
However, the recent "letter of warning against AI" penned by Elon Musk and other influential individuals has led me to question their motives. I suspect they may be losing the competitive edge they once had, as users can now effortlessly obtain information in ways that were previously unattainable. It's possible they are seeking to lobby for new laws that restrict this kind of data extraction. With such simplicity, I can envision a future where people no longer need to browse web pages, ultimately making ads and the whole business of treating your users like crap irrelevant.
[0] 10 lines to screenshot with puppeteer, tesseract to OCR and a clustering algorithm, add to this whisper and a nanotts to get an enhanced Unix philosophy+AI experience
[1] Im pretty sure i can use low resource models to do the summaries and insights, any recommendations?
* edit,im pretty sure that could have done this PoC years ago without GPT.
1 month ago people were saying there was no way AI could do their job and many were telling everyone to calm down as nothing would happen.
Now one month later this entire thread is about how chatGPT is an integral part of the development process. There are people even saying it replaces junior developers completely.
Can anyone venture to guess what the thread will look like in 1 more year?
You also have to think about comments sections not being a random sample of the site’s users.
Likely, a month ago, anyone was interested in commenting on AI threads, so more skepticism was evident. Now, most people who are AI-skeptic are probably tired of AI news and have said their piece in other threads, so don’t bother.
fwiw, I’ve tried to use ChatGPT to make a new, useful Elixir library and it failed miserably. It makes me wonder if the success with JavaScript/Python uses are because the question and answers are extensively in the training data whereas a relatively novel Elixir program is something it can’t do because it’s not truly “reasoning”. It can’t even get basic command line prompts that are straight from the Elixir or Phoenix docs correct, and it cannot solve the error messages that its code produces like others claim experiencing for JavaScript/Python uses.
If my hunch about training contamination is correct, we’re going to quickly run into not being able to create novel software if we replace everyone with GPT algos, unless their core functionality vastly changes from excellent pattern matching to true reasoning.
No there is reasoning in LLMs. This is actually proven through several scientific papers.
I can't say whether it's using reasoning for specific answers. I'm sure many times it just regurgitates something that is known... But saying that LLMs in general can and do reason is categorically true. LLMs do attempt to extract actual mental models of the subject of the text.
- GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
For your specific case, given the conditions, another explanation that fits as well is that the LLM simply needs more data to build a more complete model for elixir. Once it does that I'm sure it can respond with completely novel elixir code as it already does with python.
Is this the type of reasoning that could be used to basic math without errors (without using a dedicated math engine)? I wonder why that type of reasoning seems to struggle.
I think chatgpt is interesting in its ability to highlight the subtle variations in a concept we tend to think of as more of a single attribute (e.g. reasoning) because humans tend to do certain types of reasoning tasks at the same relative proficiency as others. No human can write complex code without being able to add large numbers, so they are often lumped into the same skill category.
I think there just isn't enough mathematical data. English language data is magnitudes more and longer then mathematical data so there is a correlation here.
The more data the better the LLM can formulate a realistic model. If it has less data then the resulting output is more of a statistical guess.
There is an argument that can be made here that the more data the more things chatgpt can copy and regurgitate but given how vast the solution space is I think data at best covers less then 1 percent.
Basically I think that if your data covers say 2 percent of the solution space you can generate a better model then if the data covered 1 percent.
This is about what the comments looked like immediately after ChatGPT came out the first time. Give it a few weeks for the GPT-4 hype to settle and I think we'll be right back to where we were.
Unlikely. GPT-4 wasn't as game changing as chatGPT. People are less hyped up and have a more realistic expectation of what they should see out of an LLM.
The extreme positive reaction to gpt4 is imo not driven by hype fuel.
In fact, if you recall the hype fuel surrounding the initial chatGPT was in both parts positive and negative. There were equal portions of people praising it and people vehemently dismissing it as a stochastic parrot. That's not what I see here.
There weren't equal parts. I was in the tiny minority saying to chill out—there was a month or so where almost every comment was hyped. It gradually moved to equal parts and then eventually to a majority of realistic perspectives.
I agree that it's more measured this time—fewer end of the world predictions—but I still fully expect to see expectations mellow out over the next few weeks.
