Well before AI co-pilots something happened to the good old admins--they started to disappear only to be replaced by "AWS devops" (their job titles) who have never wired a network using routers, switches, and cables. I noticed that they started lacking basic networking knowledge and couldn't set up networking inside AWS. They just didn't know what a gateway, NAT, or subnet is.
Similar things are happening with AI co-pilots. We have an increasing number of people who "write code", but the number of people who can understand and review code is not increasing. There is also a problem of injecting ethics and politics into those tools, which can produce silly results. I asked Bard to write me a Python function to turn the US Constitution into a palindrome. Bard refused and gave ma a lecture on how the US Constitution is too important to be played with in such trivial fashion. I then asked it to produce code that turns the US national anthem into a palindrome, it refused again. So I asked it do the same but with the Russian national anthem and it spat out code without telling me off. I then asked it to generate code for simple tasks and it did an OK job, except the formatting and the fonts used were different every time, because it just lifted code from different webpages and recombined it like a massively hungover student waking up to realise he's supposed to hand in the assignment in one hour.
> Well before AI co-pilots something happened to the good old admins--they started to disappear only to be replaced by "AWS devops" (their job titles) who have never wired a network using routers, switches, and cables. I noticed that they started lacking basic networking knowledge and couldn't set up networking inside AWS. They just didn't know what a gateway, NAT, or subnet is.
I see the same arguments about higher level languages. All of the tech industry is about standing on the shoulders of giants. I do believe that lower level knowledge can help a great deal but using this argument I can say something like:
Kids these days don’t know basic assembly, they have no idea what an or/and/nor/xor gate is, they’ve never built a computer from components (no, not cpu/mb/ram/etc, I’m talking transistors and soldering).
Maybe LLMs are different but I don’t think they are, they are yet another tool that some people will abuse and some will use wisely. No different from an IDE or a higher-level language in my book.
I have next to zero idea how many things I use daily work (like my car for example), it doesn’t stop me from being able to drive where I need to go. I have configured lower level networking equipment in the past and I couldn’t be happier that I don’t have to do that drudgery anymore.
All I’m saying is this is a dangerous argument to make because someone can always one-up (one-down?) you. “Oh you had NAT? Luxury! In my day we didn’t even have a network, we had to….” (See also: Four Yorkshiremen [0])
I do not wholly agree with you. Just to pick on one thing in your reply...
> Kids these days don’t know basic assembly
It is a problem. Knowing assembly is crucial if we want to have more efficient, faster hardware. Someone has to write kernels and drivers. The efficiencies happening on the lower level of the stack make the upper layers of the stack faster, more stable. It is also important that we have a large pool of devs who know how to do it. Otherwise that knowledge will be taken in-house and shared under strict NDAs preventing devs working for competition and effectively killing all the efforts of the Open Source community to wrangle free access the computer out of the claws of corporations.
This is not a problem that AI has brought about first, but co-pilots are accelerating it.
Just to provide some context, I could not wait for 16-bit, 32-bit, and then 64-bit architectures to become popular. I was the happiest dev around when object oriented programming tools became available, same to Linux, web, all frameworks and even cloud computing. But AI as it is currently being sold is a shit tool. If it stays and doesn't disappear after the VC money runs out, we will have to adapt our ways of working and educational courses to teach devs to review work of others recompiled by AI (it can't write shit on its own). Not sure it is worth the effort, because every time a new framework is introduced the results generated by AI will be of low quality until the models get trained on exaples written by humans... What's the point of using AI in that case?
Agree here. As a 90's kid I dove into computers, Internet and programming head first. I went into the field out of passion, not money. I received an engineering degree.
If I was the same kid now, I'd get into a bootcamp or self teach and get a web dev job because the money is lucrative. Yeah sure, maybe i'll dabble in foundationa litems here and there, but most won't. It doesn't advance your career in web/software.
IMHO we will see a brain drain in computer/electronics engineering in 20 years when the pipeline of graduates dwindle and those with experience retire. Hope LLMs can innovate enough to make up, I guess.
> If I was the same kid now, I'd get into a bootcamp or self teach and get a web dev job because the money is lucrative. Yeah sure, maybe i'll dabble in foundationa litems here and there, but most won't. It doesn't advance your career in web/software.
That sure sounds like it has nothing to do with LLMs and everything to do with factors not all related to even computer science. “Capitalism” or maybe the more trendy “late-stage capitalism” is probably more to blame here than anything else.
Also I reject the idea that there are no kids today that get involved in tech/computers for the joy of learning/exploration.
I bristle whenever I see arguments about “kids these days didn’t learn exactly how I did so it must be wrong”. It reeks of “old man yells at cloud” (note, I was also a 90’s kid). It’s the same BS I heard as a kid about how computers would rot your brain or they were horrible for <insert stupid prediction that didn’t come true>.
I agree that it reeks of old men yelling at clouds but there is data driven precedent here: with the advent of smartphones, computer literacy in education has dropped like a rock. It isn’t just kids these days, it’s a genuine shift in computing across the entire populace that happens to really effect the funnel for programmers at the entrance to that funnel. Copilots are an extension of that development, not necessarily something wholly new.
I’ve heard this anecdotally from every educator I know at the high school and college level. CS students are increasingly entering without any computer proficiency whatsoever so it has to be built up from scratch.
Yep, despite all the talk of young people being "digital natives", installing apps from Android/iOS app store, taking pictures and chatting with friends doesn't teach you much that is valuable for office work.
PC gaming and specifically modding is what I can thank for much of my computer proficiency, but even that is easy these days with Steam and its workshops for many games.
Of course you’ve now effectively circled back to the early 1980s at some level when there was not a lot of expectation that college freshmen had touched a computer much less were proficient even if the wanted major in CS/EE.
I think the apt analogy here isn't "old men yelling at clouds" but the historical way that one certain previous generation become more adept than those before or after at working with a certain then-new technology -- thus becoming a uniquely skilled generation in that technology, moreso than their elders or their youngers, never to be repeated again.
I'm not talking about millenials/gen X with computers -- I'm talking about silent gen/(boomers/greatest gen?) with cars.
In both cases, a new technology dominated the world, but was new and brittle at first. The kids who saw it in their youth were fascinated, but because of its brittleness they had to become at least minor experts in at least minor troubleshooting and mainteanance (like tuning a carburetor or defragging a hard disk) just to access the coolness.
I mean, it really was a pain to be a car owner in 1960. A lot like being a computer owner in 1995. If you wanted to enjoy one, you were going to need to change a spark plug or registry value once in a while. And you had to be ready to recover from an engine overheating or a blue screen of death, because these were not rare events.
Then the future generations increasingly lost touch with those skills because the technology got smoothed out as it developed further.
So I think it's quite plausible that future generations will permanently have less interest in serious skills with computers since the same thing has happened with cars. There is a way smaller perecentage of people my age (millenial) who are "into" cars or have moderate familiarity with car repair than people in their 70s. So it seems plausible the same pattern could continue to play out with computers, rather than a "old man yells at cloud" illusion
> Knowing assembly is crucial if we want to have more efficient, faster hardware
this is a goal for maybe 2% of the industry. And those folks know their assembler or they don't get hired. I think we are producing skills in the proportion of its requirement in the industry.
I hate to dash your hopes of never seeing LLMs write code again but Llama3 70B is not terrible as a copilot and it’s small enough to run on comparatively cheap hardware or even consumer hardware when quantized plus it’s publicly available so from now on there will always be some form of copilot service or tool that will be in use, that ship has sailed. The good news is that it doesn’t look like we’ve maxed out the capabilities of GPTs yet so as more are trained and compute becomes cheaper and as we produce and accumulate more data they should keep getting better for a while at least.
> Not sure it is worth the effort, because every time a new framework is introduced the results generated by AI will be of low quality until the models get trained on exaples written by humans... What's the point of using AI in that case?
LLMs will not replace us, they just take some of the tedium out of writing code. Yes, they will have to be trained on new frameworks and languages but I don’t see that as a problem. People have to wait for SO questions and blog posts to be written about new frameworks and languages as it stands today.
I’m not sure I agree with your arguments about needing a lot of devs to understand assembly or the like. Or rather I disagree with your prediction of where that leads. We are effectively already there and it hasn’t led to your future. We have an amazing open source ecosystem and that billions of people rely on daily without knowing how every part of it works. Open source has only grown during that time.
I’ve never done the math myself for calculating the distance between 2 points on a globe. I just grab the Haversine formula and keep going. I don’t see that as a problem. Similarly I’ll reach for an open source library to solve a boring problem instead of doing it myself. I quite literally see no difference in that and using an LLM.
> Kids these days don’t know basic assembly, they have no idea what an or/and/nor/xor gate is, they’ve never built a computer from components (no, not cpu/mb/ram/etc, I’m talking transistors and soldering).
At each layer, though, we have a legible formal theory bridging one level to the next, and the abstractions tend to be so reliable the last thing most of us will consider is a hardware/assembler/compiler bug (though they do occasionally happen).
LLMs bridge layers illegibly and probabilistically. I can already see how they're helpful, but they do have some different characteristics that mean reliability and the connection to lower levels of the stack are an issue.
> LLMs bridge layers illegibly and probabilistically.
I’m not sure I agree but I want to make sure we aren’t talking past each other.
Using an LLM to write code does not have this problem IMHO.
Using an LLM as part of the code does have this problem.
I’m not writing off the usefulness of calling out to an LLM from code but it does bring “illegibly and probabilistically” into the mix whereas using an LLM to write code that’s human-reviewed/modified does not.
In fact, you should be unable to tell what I wrote vs what the LLM wrote if I’m doing right (IMHO). “Clever” code is always bad no matter who/what wrote it.
The only time I’ve been asked “did you use <insert LLM> to write this?” Was due to the speed at which I completed something or because it’s in a language that the person asking the question knew I didn’t know very well.
LLM's can be good at guessing the code you want, but I've seen them guess the comment or documentation correctly, then the implementation is the opposite of what you asked. It also generated tests which were for the incorrect case, with comments showing the opposite of what you wanted.
For somebody new at the language or new to programming in general, that difference can be imperceptible until it gets to a review.
I've also managed to spot people using an LLM to write stuff due to them doing things which won't pass linters or code tests.
The subtle differences between comment/code can absolutely be a problem, I’ve fallen prey to it as well but as long as you test/execute the code you’ll catch it if you didn’t catch it from reading over the code. I often reject anything overly clever or that I can’t reason out quickly when it comes to LLM generated code.
As for not passing linters or code tests that’s unacceptable behavior from an employee. If that happened more than a few times I’d be having a talk with the employee followed by showing them the door if they continued to do it. I’d behave the exact same way if no LLM was involved.
I do wish LLMs like Copilot could use the lint rules and/or tests to inform the code they generate but thankfully it’s normally a keystroke away for me to reform a block of code to match the style I prefer so it not the end of the world.
To use LLMs (or any tool) you have to be able to show you can use it correctly, safely, effectively, etc. If you can’t do that then you have no business being in a coding role IMHO.
Points for a great twisting of the metaphor I used, I quite like it.
I agree _someone_ has to have that knowledge, I reject they _everyone_ must have that knowledge.
I have zero desire to learn more about logic gates, they don’t interest me and while they enable me to do the work I do I don’t need to know about them to do the work. The same way that I couldn’t tell you how my car engines works (ok, at a high level I could, probably about the same level as I could talk about logic gates) but I’m able to drive from point A to point B without issue.
There are people who enjoy assembly, let them work on it. I’m not one of those people and I don’t have a problem with those people but I do have a problem with people who think I must know assembly to be an effective developer.
> I have zero desire to learn more about logic gates
Sure, but you know that they exist, which to me suggests that you learned about them somewhere and probably have at least some foundational understanding of how they work
The problem is when people can skip any of that broad foundation and go straight to specializing somewhere, a lot of general knowledge is lost
Part of learning the broad foundations is also learning that those options exist for a specialization path
Someone who may love working on hardware designs may wind up spending their life as a mediocre web developer because they took an AI assisted web bootcamp that skips learning important fundamentals
It's an engine, and can make all kinds of things move.
If you don't know it works, sure you can drive a car, but a lot of missed opportunities to leverage the same exact tech for other use cases disappear when you don't know fundamentals.
Interesting, the college I went to (2009) didn’t teach it at all. C++ was the lowest level they taught and they switched to Python after I left (as in they use it instead of c++ last I heard).
If I ever needed to write assembly (which probably means something has gone horribly wrong in my life) then I’m sure I could learn it on the fly, just like I’ve learned other things never taught in college on an as-needed basis.
This argument falls apart for me personally. I'm sure we could go back a decade or two and find someone saying something like:
It’s quite alarming to think that the most advanced language many new software engineers are familiar with is merely C or C++.
Imagine graduating without ever having programmed in assembly. It’s not that C or C++ aren’t capable languages, but we’re definitely in a strange era when the deeper understanding of machine interaction through lower-level programming is becoming a rarity.
I don't think that's quite the same, but in general I agree with that sentiment, even though you weren't serious about it.
Folks should know their basic C/C++ just like they should know enough x86 to make sense of what godbolt.org tells them. But oh, well, that's probably a lonely hill to die on. Having a rough idea of all these levels of abstraction work and interact is something I profit a lot in my line of work, but that's just a small niche and I get that (profiling and optimizing scientific, signal processing routines).
I consider assembly to be a basic skill like arithmetic. Most of us have a calculator on hand 24/7, but being able to look at a clock and say 'I've got 12 minutes' is useful.
I had to take an "operating systems" class in Java 15 years ago, luckily i already knew enough about C and assembly to know it's a waste of time. the professor thought it was more important to teach "fundamentals" than low-level programming.
i got a c in that class (the grade not the language) because i didn't know java and kept having difficulties with the way it forces you to shoehorn OOP into everything.
I don't remember seeing a class on it but it was a long time ago. I also don't remember it being required to get a degree. I just looked on my college's website and they offer 2 assembly classes it looks like, I'm not sure if they existed when I went there (I assume they did) and neither looks like it's required but I don't have the time or energy to dive back into that kafkaesque hellscape of figuring out what's required to graduate.
However, I didn't get a degree so maybe it was shoehorned into some high level course. I had a job writing software while I was in college and dropped out to do that full time once I felt I wasn't getting any value from continuing my "education". It's never been an issue in my career and I've done well by my own metrics at least.
Yeah my husbands college (I'm a SW eng, he wants to be one) teaches C++ as the base...and this is a "just" a state school in a Arkansas.
Also gotta correct myself, after asking him it looks like I was wrong. Assembly isn't a direct requirement but it's one of three electives you have to chose from.
You have to choose between either assembly, cybersecurity or data science and assembly is considered the easiest apparently.
I feel comparing knowing assembly to knowing about NAT/Subnets/Gateway is a false equivalency.
Outside of a small set of folks most devs need NOT know assembly due to compilers being very good at converting to assembly.
Not the same Subnet/Gateway/NAT. These are first class constructs in most cloud developments and has huge implications on security, performance and costs. Unless you have some kind of automated way that abstracts away these issues (akin to compilers) most devops folks would need to learn about these concepts while working in a cloud environment.
I think there may be a difference in kind here than with a lot of previous labor-saving abstractions.
To use an LLM, you still need to know the language you are using it to write. You don't need to know C or assembly as a Python/JS programmer today, but you do need to know JS to be an LLM-heavy JS programmer.
> You don't need to know C or assembly as a Python/JS programmer today,
I agree, not everyone does. Maybe we have progressed to the point we can all say those people are wrong and their arguments were always wrong, maybe not.
> but you do need to know JS to be an LLM-heavy JS programmer.
I also agree. That's my whole point, you do need to know the language to be able to effectively use an LLM.
My issue is with people who immediately reject LLMs as a net-bad because they do part of the job for you. I reeks of "well I had to do it the hard way so you do too" which I cannot abide and immediately flips the bozo bit on them for me.
Okay, but if you don't know what the JS is doing and why how are you ever going to architect stuff efficiently or think up more complex things in the abstract? I fail to see how you can just know HTML and CSS, that's like saying you have the alphabet down but somebody else can speak or write for you. HTML and CSS are just markup and style languages, it isn't really code.
CoPilot is just blundering along using what it has learned from elsewhere - which is not always correct.
I'm very impressed with the best models — but, because I remember how awful NLP used to be, "very impressed" still means I rate the best as being around the level of an intern/work placement student most of the time, and even at their best still only a junior.
It's great, if you're OK with that. I've used GPT-3.5 to make my own personal pay-as-you-go web interface for any LLM that's API-compatible with OpenAI despite not being a professional Web Developer.
It's fairly fragile because it's doing stupid things to get the "good enough" result.
Bu that's OK, because it's for me, I'm not selling it.
(As for the lesser models… I asked one for a single page web app version of Tetris; it started off lazy, then suddenly switched from writing a game in JavaScript into writing a machine learning script in python!)
> They just didn't know what a gateway, NAT, or subnet is.
If you ask me, they shouldn't have to, in the same way that you probably don't know how to diagnose a coax cable problem.
The major public clouds all have completely software-defined networks (SDNs), and concepts like subnets are basically emulated for the sake of legacy systems.
Why would we want subnets, like... at all? Why can't all systems just have an IP address and use point-to-point communications? That's literally what the underlying SDN does with the packets anyway! The subnets and routes you see don't map 1:1 to the underlying network. The packets are stuffed inside a VXLAN packet and routed between random hypervisors scattered randomly across multiple buildings. Just give up the pretence, give everything an system-assigned public-routable IPv6 address and be done with it. Better yet, auto-register everything in a private DNS zone and auto-generate internal use SSL certs too.
The amount of wiring we do manually is just insane, just so that Azure and Amazon can help greybeards pretend that they're operating a Cisco router and need to stand in line to file the paperwork required to get an SSL certificate.
> you probably don't know how to diagnose a coax cable problem
No, that was definitely part of the sysadmin job description ~25 years ago, both line discipline and physical medium problems (I had a TDR and a fiberglass fish at my desk for my first sysadmin job and was kind of bummed to find out that isn't remotely normal anymore)
Problem is that all those systems are proprietary to individual clouds.
Congratulations folks! We undid several decades of progress and went back to the 1960s and 1970s when software can only run on the specific mainframe for which it was written.
Which is why I don't write for the cloud. My personal stuff goes into a dedicated linux server setup with docker compose, and my work stuff is all hosted on windows servers.
I'd prefer switching those windows servers to Linux, but we're too integrated with Microsoft at this point.
It’d be easier to swap those Windows servers out for Linux than it is to migrate the typical cloud native application from one cloud to another.
The cloud is the most closed, most locked in, most rent extraction oriented computing paradigm since the pre-minicomputer mainframe era. It’s really a return to that era.
Computing tends to cyclically reinvent wheels. I suppose next we will reinvent minis, micros, LANs, etc.
I think you’ve stumbled on the solution there - *something* happened between the 70s and today to address that thing from the 1960s, and it was motivated by capital-seeking. There’s a high probability that it will happen again, and if it does not then it’s likely the market didn’t demand it. Or, the world is just one big corporation and we’re all screwed. One of those.
The problem with that idea is that it requires every router to know about every system on the network. Unless they abandon Ethernet and replace it with MPLS source routing, which maybe they should. You'd still have a maximum subnet size, because the hardware firewall that would still have to be in front of every server would have a limited capacity.
I've noticed that ChatGPT will speed up the monotonous part of writing code, but will often inject a few bugs into the code that I need to fix. In a way, ChatGPT has become my personal junior developer (which I only use for basic stuff) while I'm still dependent on my own expertise to review and fix the code, but at least it's still faster than using google and writing most of it myself. Juniors will always have bugs in their code, but when you bypass the step of writing the code and figuring it out for yourself, how do you effectively learn and become a better developer?
In a way, ChatGPT both replaces the need for junior developers and creates a situation where you'll end up with a developer shortage because not everyone can start right away as a mid-senior level.
This is definitely appealing to my curmudgeon heart. But I translate it into my corner of the world and the equivalent is everyone should know c. We don’t need everyone to know c.
Everything else being exactly equal would I like someone I work with to know c over not? Sure.
But would I take a world with many fewer programmers that all knew c over the one we have today—definitely not. It would be a significantly poorer world.
Your mentioning of AWS devops reminded me of a story my dad, a mechanical engineer, told me. He travels a lot and works with engineers all over the world. He was talking to me about how the new "wave" of engineers (mainly in Canada and America) that he works with there is a a complete lack of people that know how to do things like manually calibrating various parts by hand.
He's had to completely rethink how they approach their industrial designs and build machines and transition to much more electronic based things with fewer moving parts, and build things they can just adjust with computers and point-and-click GUIs, because people just don't know how these old things work from a physical reality perspective. My dad was very distraught by the caliber of engineers he's seeing in the "next generation".
But that's not the only thing.
In Canada, our aging population of tradespeople are retiring in droves and we have a dearth of younger people to replace them. It's so bad, that in my province they've completely redesigned the Grade 11-12 highschool curriculum to have "fast track" paths into trades where 80% of their time is spent in coop/apprenticeship (good thing!). We have a housing crisis and we have nobody that actually knows how to build houses in our country while the federal government imports 1 million people per year with nowhere for them to live. Most kids these days just want to be "influencers".
Spelling and grammar in young people is absolutely atrocious because they are totally dependent on autocorrect; and they can't type using physical keyboards because they only ever used a touchscreen keyboard on phones and tablets.
It's not just programming being destroyed by copilots. It's an overall dumbing down of our entire civilization.
I have the same issue with young software engineers. We used to install every Linux distro because it was fun. Junior are now scared of the command-line.
Another example that I see all the time: I recently had to review a pull request from a junior. 200 lines of very simple C++ doing almost nothing, you can't fail that. I had to write more than 50 suggestions and comments on very basic stuff that the guy should have learned when he started coding. Yes, they must learn the ropes, but it seems like the passion is not there. We used to be passionate about coding, and all I see is people who went to private college to make money but don't enjoy what they do.
I was a hobbyist programmer installing all sorts of Linux and BSD distros on my computers since the age of 10. My first real software job still had me go through a period where I had 50+ comments on some of my initial changes because life before work doesn't really prepare you for what you need to know on the job. If they aren't learning from your feedback and constantly making the same mistakes then I understand, but it's somewhat misguided to expect folks to simply know a lot early in their career.
Heck even folks coming in with experience, but from a company with a very different coding culture or simply a different primary language can result in the same situation.
Do you/they learn is the question. Not programming but a different role in tech but I’ve had a recent intern who did a very competent job and vacuumed up feedback and opportunities to learn. And I’ve had a longer tenure person who moved in from a different type of role at a different company who… did not. And just couldn’t appreciate how what they were creating was a net time sink for everyone else. They didn’t last.
Niche-ness isn’t a hard metric, but the argument is the same for “this is what happens when a profession hits a critical mass of people just in it for the money and not at all for the enjoyment of the profession.”
lol, i'm reinstalling a linux distro right now just because I want to try a new one
I work with seniors that are scared of the command line. I can somewhat understand why Microsoft might be trying to turn everything into AI Copilots because when I'm trying to explain to someone how to revert changes in a file with `git reset/checkout` they practically recoil in terror at the suggestion of using the terminal. They are married to their git guis and have no idea how it works. Best AI-ify as much as possible to keep people in their ecosystems.
But are they any less productive? My experience is no. I was a command line aficionado when I entered the workforce, but I saw several people get the same results as me in a purely GUI environment. I see the same today. They get shit done and that is what is required. I feel like older devs romanticize certain parts of their workflow too much.
They're not necessarily dumber (maybe in the sense that they would be less able to survive on a desert island), but rather the same neural circuitry is being repurposed for other things. If anything the environment is dumber and they're just better at efficiently navigating it than we are.
When I need to explain to other so-called seniors how to revert changes to a file with git checkout and they recoil in terror in using the command line, then yes - I believe they are actually getting dumber.
No. Most kids strive to follow whoever "mentors" and teaches them.
What is sad is that this job is left to influencers. Not the kid's fault if our education system, parenting and attention economy are completely f'd up.
> There is also a problem of injecting ethics and politics into those tools, which can produce silly results.
Well, they can, but the "no filter" option does not necessarily generate satisfactory results either. I find it hard to blame Google for not wanting to put their brand on an AI providing a ranking list of races by intelligence or whatever kind of offensive thing people are going to try to make it produce.
We are talking about a code generator. Dev tools ought to be neutral when it comes to politics and ethics otherwise you get a smart hammer that will refuse to nail Jesus to a cross. That's not how tools work.
Safety devices put into tools to prevent misuse or accidents are also not that unusual. Feel free to change my example to something like "a code snippet that will use the 23andMe API to prevent users with insufficient European ancestry from registering an account" if you want a more code-specific example of reputational risk from an unfiltered AI.
> and it did an OK job, except the formatting and the fonts used were different every time, because it just lifted code from different webpages and recombined it like a massively hungover student
Tangent, but it used different fonts? I can’t imagine how this is possible given how LLMs work under the hood (sampling likely tokens which represent Unicode character sequences - it doesn’t operate on rich text so afaik has no way of seeing or changing font information). Is there something weird happing with the frontend breaking the code/not-code formatting they do (as a cosmetic thing rendering the output)?
As constitution - changing one but not another could be a matter of cultural sensitivity as well (although I bet Bard isn’t that smart).
As a Polish person, I don’t mind you changing the constitution, but for example with our national anthemn we are way more attached to changing how it sounds than Americans are to theirs. Ditto where our flag and our symbol can be used.
As for Bard - did you try arguing with it that constitution guarantees you the freedom to play with it? I managed to convince gpt to do some weird stuff by arguing for cultural sensitivity.
What Bard couldn't understand was that what I actually expected it to do was to write (well, steal, these things are simply giving answers that look like plausible answers by recombining other people's work) a function that takes a long piece of text and turns it into a palindrome. It cannot deal with ambiguity and imprecision, but will refuse how to acknowledge that fact and will keep generating random answers like a public schoolboy who insists on winning an argument despite not knowing a thing about what it is he's arguing for or against. When I asked Bard a more precise question and asked for a function that takes a long piece of text and turns it into a palindrome it gave me a different implementation of it every time I asked the same question behaving like a kid who has no clue, but will give a random answer and watch the teacher to see if the teacher accepts it.
I will not argue with Bard or any other tool of that sort. I have better things to do. They are supposed to be good at translation, so I thought ok, let's try giving it a piece of my own writing in English and ask it to translate it into French (a gendered language). It could not infer from the text that the internal monologue that the protagonist was a female, so it translated it all using male voice. I asked it to switch gender to female voice but it stopped doing so after translating about two pages of the text and even on those two pages it kept switching to male voice. Then, after translating three pages it replied "I don't know how to do it" when I asked it translate another page, even though I used the same prompt I was using to ask it to translate previous pages. These are toys not tools. They are shit at their job and a waste of time and electricity.
Since you are Polish, I was told by my Polish colleagues that onet.pl, a large Polish news site uses AI to auto-translate articles from foreign media sources and that all of those translations are done in the male voice, even when the original text clearly describes what a woman did or said. My colleagues say it's been going on for a while and nobody bothers to fix these translations. If true, the future is here, it is shit, and AI is good at one thing that VCs love to fund, i.e. extraction of value through destruction of value.
I agree. The state of level of “basic” knowledge in developers are dropping.
The consequences are that developers can tackle basic tasks which are supported by the frameworks they use, but once something is not supported or straightforward they don’t know what to do and get completely stuck.
From society’s point of view, the usefulness and value of the task force decreases and important problems are not solved or aren’t efficiently solved.
They don't know what they don't know. (Devs not knowing where to start writing a piece of code that works on actual bytes and not strings or numbers is a classic example of this problem.) Sometimes they are good devs in their space and mean well, but they spend their working lives on a different level of abstraction and in a different problem domain. The danger with co-pilots is that devs using them will even not know what to ask for, because they will not know what it is they need to write. There is value in going through docs, writing your own code, making mistakes, seeing your code fail, fixing it. That's how you learn to spot mistakes made by AI, but I don't think that having a coding puppy and spending time fixing its mistakes is conducive to writing better code or improving developer productivity.
> We have an increasing number of people who "write code", but the number of people who can understand and review code is not increasing.
Something like this was happening with coding bootcamps as well. Lots of people who can code what they were directly taught but don’t have the basics to go outside of that.
Yes I agree. But good companies are aware of this fenomenon and these so called chatgpt programmers are considered juniors. There is always place for you as an experienced developer who understands memory management, io performance, stack executions, step debuggers, compile optimizers and much more.
> because it just lifted code from different webpages and recombined it like a massively hungover student waking up to realise he's supposed to hand in the assignment in one hour.
That’s CTO level coding right there, gotta tell bard about my great app idea.
> I asked Bard to write me a Python function to turn the US Constitution into a palindrome. Bard refused and gave ma a lecture on how the US Constitution is too important to be played with in such trivial fashion.
Jesus, they have coded pearl-clutching and finger-wagging lectures in my computers. You know, those machines that used to be brutally efficient at doing what you asked of them.
> those machines that used to be brutally efficient at doing what you asked of them.
In a way, that's a natural thing to happen. Computer systems are weird in that we have an extremely good model of how they work at a certain level: "processor will (almost) always process assembly exactly as it should, etc."
When computers first came around, this "computer does exactly as you say" thing was weird to people because there weren't really any such things before computers. People would make small mistakes in their code and wonder why the computer can't "just do the obvious correct thing like any sensible person".
However, as multiple complex software systems interact and build on top of each other, this ability to "understand everything" at one of these "perfect model" levels (e.g assembly, logic gates, state machines...) becomes useless. To work with them, fuzzy models (like those we use for other humans or animals) are more appropriate. Complex software systems will routinely not do the same thing when given the same command. They will change over time, break in incomprehensible und unpredictable ways, and, as the newest addition, argue with you instead of doing what you say. That development makes total sense. At least those systems can now "take a hint".
I'm all for moving to higher-level skills, when I was an adjunct professor I usually didn't put a huge emphasis on syntax unless I thought it was important to explain a concept, and I very rarely took off points for syntax.
That said, I don't know that I would allow AI for learning to code in the class. I do not consider "prompt engineering" to be a substitute for programming, and it's very easy to get AI to write some code and copypaste it, not understand anything it said, and be done.
Last year, when I was administering a test, I said that they were allowed to use any IDE they wanted, but they could not use any AI assistance. I am pretty convinced one student just copypasted my test questions into ChatGPT or BingAI and then copied the answers. I didn't have a way to prove it so I had to give him the benefit of the doubt and grade it assuming he did it honestly.
Before someone says "YOU SHOULD TEACH THEM TO USE THE AI TOOLS LOL THAT'S THE FUTURE!!!", stop. That's not true, not yet at least. Just because I can call an Uber doesn't mean I get to say I know how to drive. Just because I can go to McDonalds doesn't mean I get to say I know how to cook. I was teaching them how to write Java and Python, the goal of the class was for them to learn Java and Python, it wasn't learning how to copypaste into OpenAI.
You can't learn to play the guitar by reading books about it, or watching YouTube videos, or reading sheet music, or getting "personalized AI tips".
You can only learn to play the guitar by picking it up and spending a lot of time mucking about with it. There's a bit more to it, but this is really the core: you need to play and get that muscle memory and "feeling" for it. You need to "rewire your brain".
Coding, or any other skill for that matter, is no different. The only way to learn to code is to actually write code. Even if you fully understand anything ChatGPT gives you (which most students probably don't), that's no substitute for actually writing the code yourself.
I don't see how it's not hugely harmful for the development of students and junior programmers to use AI. Even if these tools were perfect (which they're not), you need to develop these basic skills of learning to read the code. You need to make mistakes and end up with crummy unmaintainable code, and realize what you did wrong. Etc. etc.
Even for senior programmers I'm not so convinced AI tools are actually all that beneficial in the long run.
Only when AI systems will be able to fully understand entire systems and full context and can completely replace human programmers will that change. I'd estimate that's at least 50 years off, if not (significantly) longer. Anything before that: you need someone who fully understands the code and context, in depth.
Having a vague, conceptual understanding of a topic is often confused with expertise. Also known as knowing enough to be dangerous.
What worries me is that AI code is getting complicated and harder to correct. I recently noticed that after my IDE generated some code I felt a palpable sense of dread.
I often have to go over it in detail because it makes subtle, hard to catch errors. It pulls me out of my flow. I am starting to dislike that effect and I am certain huge swaths of the (new) coding population will lack the skill and motivation to correct these things (as various earlier comments here show).
This same issue existed with Stack Overflow, but AI further lowers the barrier to be dangerous by throwing a brick on the gas pedal.
I’ve also found it makes otherwise knowledgeable and experienced people think they can do stuff without learning it. I’ve had several people on my team tell me they think Copilot will help them get up to speed and help out with some of the stuff I’m working on. So far none of them have done anything and I’m not sure how Copilot explaining a block of code is going to do anything for them. It’s already a very easy syntax to read, and they all already have coding experience in other languages. It’s a tool to let them think helping will be easy, so they volunteer, then do nothing because it isn’t the reality of the situation.
Coding, or any other skill for that matter, is no different.
Back in the early days of computing people wrote software by working it out on paper, encoding it on to punch cards, and then giving that program (deck of cards) to an operator who loaded them and ran the code. If it didn't work properly you would get back a print out of the 'debug' which amounted to a memory dump. You'd then patch your punch cards based on working out where you'd screwed up, and try again.
That was the computer software industry for about a decade before time-sharing, VDUs, etc.
People absolutely did learn to code by reading books and nothing much else. Access to computers was so restricted (because time to use a computer was shared between lots of people) there weren't any other options.
Heck, I learned a lot of early web stuff like TCP, HTML, etc reading books at my parents house when I was at home from uni and didn't have my computer with me, and that was the late 90s. Of course you can learn coding by studying the theory without practicing. It's just a lot less fun.
That is actively engaging your brain, just at a very slow pace with a very slow feedback loop.
Dijksta's "I had everything worked out before I wrote the code, because the computer didn't exist yet" is the same. It's like working mathematics out: that's not "just reading", it's actively engaging. Actively writing. Completely different from the passive consumption of AI output.
> Of course you can learn coding by studying the theory without practising. It's just a lot less fun.
No you can't. When you first started an editor or IDE you wrote some vague code-shaped junk that probably wasn't even syntactically correct, and didn't learn programming until you practised. Of course you need to learn some some theory from a book or instructor, but that's not the same as actually learning something.
Yep Knuth learned by reading an IBM manual with source code while sitting on a beach during summer vacation. Decades later systems hackers learned by reading illicit copies of Lions' Commentary on Unix w/Source
Exactly, it's the same with autonomous driving or any other activity (even art!) that has always required a human. While it can "look cool" and like it "understands the context" on the surface, when diving deeper it will _always_ misunderstand the nuance, because it doesn't have all of the lived human experience and reasoning.
Being even 80% (or 90 or even 95) there isn't enough - something will always be missed because it's only able to "reason" probabilistically within a narrow area not far away from the training data.
This is very well put. I think the next point of discussion is what fraction of "software developers" "need" to know how to code in the deep way you describe.
I don’t think the guitar analogy is a great one. You have to learn by picking it up because no expert plays the guitar using personalised AI. But there are expert programmers who use lots of AI tools, so AI is more of an important part of learning to code.
Doesn't matter what (some) experts use; it's about learning fundamentals. You can only become fluent by writing and reading code, at times struggling to do so. This is the only way to learn any serious skill in life, from programming to music to chess to woodworking to cooking to physics to ... anything.
I'd say that especially because assistants are not good enough yet, it makes sense to allow students to use them. LLMs will produce output that is messed up, students will spend time figuring out how to fix that and learn the language grammar this way.
I recently learned Swift & Metal this way - knowing Python and other languages, I kept pasting my pseudo-code, asking GPT to convert it into Metal/Swift, and then iterating on that. Took me a week or two to get the language without reaching for any sort of manual/tutorial. Speaking from experience - it would take me 2-4x as much time to learn the same thing otherwise.
If GPT was any better, I'd have no need to fix the code, and I wouldn't learn as much.
I want to emphasize that I'm not really a luddite here. I pay for ChatGPT (well, Kagi Professional), I use it daily, and I do think it's a very valuable learning tool. I even encouraged my students to use it outside of class if they needed help with any of the theory.
I use ChatGPT all the time for learning new stuff, but a) I already know how to program well enough to teach a class on it, and b) it's a supplement to doing stuff on my own, without AI help. I don't feel I learn that much from just copying and pasting.
Totally agree with your process though; correcting and arguing with ChatGPT is an insanely good way to learn stuff. It's really helped me get better at TLA+ and Isabelle proofs.
You can upgrade any family member to Kagi Ultimate plan for $15/mo and it will give you unlimited access to GPT-4, Claude 3 Opus, Gemini 1.5 Pro and other state of the models.
Yeah, they don't advertise it very well. It was actually a pretty good value add for me; I was already paying $10 for Kagi search, and $20 for OpenAI, so the $25 for both actually saved me $5/month.
I think you can upgrade to Ultimate at a per-account level for another $15/month.
EDIT: Accidentally wrote "pro" when I meant Ultimate.
> I recently learned Swift & Metal this way - knowing Python and other languages, I kept pasting my pseudo-code, asking GPT to convert it into Metal/Swift, and then iterating on that.
Significant here is that you knew other languages already. I've had the same experience with GitHub Copilot, but I'm cautious about recommending it to new learners who don't yet know the fundamentals.
All indications I've seen indicate that it's easy for people who already know the foundations of programming to use AI tools to learn new tech, and that can actually be the most effective way for them, but that it doesn't work nearly as well for people who don't have knowledge of other languages and frameworks to lean on.
> and that can actually be the most effective way for them
A lot of it is just a general lack of learning material for that demographic. For most languages, you get to choose between an introductory text that assumes you've never even heard of a "pointer" or an advanced text that is meant as a reference for an experienced developer of the language.
There's not really a lot of stuff out there to teach someone who already knows another language, who just wants to know the syntax and common idioms in that language. For example, a C programmer can easily pick up Python, learn some syntax, and go nuts -- but they just won't know about stuff like list comprehensions or dataclasses unless someone points it out. They'll write Python code that reads like C, not like Python.
It's similar with maths - calculators are invaluable, but being actually good at advanced maths just NEEDS you to grind through basic calculations so you adopt those basic "pattern recognition" skills on how numbers work, can be broken down and swappe out.
Yup. I gave a lab homework to an SE class of 42. After reading through at least 12 LLM generated, similar looking, and wrong code submissions, I've cancelled the whole lab assignment. And students complained in class evaluations about it.
An alternative is to keep the lab assignment but grade them on a final exam. Then it is in their interest to do the lab assignment by themselves rather than with an LLM.
>LLMs will produce output that is messed up, students will spend time figuring out how to fix that and learn the language grammar this way.
I’ve never met anyone who prefers debugging over writing fresh code. Did LLMs just automate away the fun part and leave us with the drudgery? That sounds like a horrible way to learn. All work and no play…
It also would mainly teach how to get good at spotting those little mistakes. But without context from writing code, it seems like it would be harder to pickup on what doesn’t look right. They’d also miss out on learning how to take a problem and break it down so it can be done with code. That’s a foundational skill that takes time and effort to build, which is being farmed out to the LLM.
With super difficult algorithmic code, for me, it works best to just start writing and then fix, than to get stuck not knowing how to write.
Algorithmic code aside, I’m learning cuda now, and I just told gpt to write me some code and I debugged it then (with gpt’s assistance as well). It took me an hour to produce something that I expected it would take 2-3 days otherwise.
As for picking up what doesn’t look right - if it doesn’t work, you can pinpoint place where sth is messed up using a debugger/prints, and then you learn how the code works on the way.
I remember that when I was learning to program 30 years ago was the same - I rewrote pong game from a magazine, then kept changing and messing up things until I learned how it should work.
It’s like having infinite stack overflow examples (and explanations) to sample from and iterate on. It’s unlikely any one of them will solve a complex problem you’re working on but by reading and experimenting with enough of them eventually you grok it.
I think it would be quite mentally difficult to be successful at writing device drivers (any hardware device, many different fields) if a person had the slightest reservation about debugging. I've got past a lot of blockers by creative trial and error and making it work was the satisfaction driver. I actually enjoy the making things work part. I have had employees who felt the same, one of my jobs was telling management to fuck off while the person worked on the bug, for days without result if need be[1]. The "just writing code" part is how one actually gets to the interesting part where I earn my salary.
I have no idea how an LLM could help with that process.
[1] I successfully found a bug in a CFD solver (not a device driver) in the early days of templated g++ only after 2 weeks of fairly grueling half time work. Missing ampersand in an argument list. Believe it or not I was very happy and doubled down on the code. However! I once fucked up a job situation where my MPI-IO driver failed because I had a very subtle linux RAID hardware bug that I could not find after weeks of effort. I found it later, too late. That truly sucked. I really don't know how LLMs could possibly help with any of this.
In 1993 or so g++ couldn't do it, but I suspect that all c++ compilers today would. Then why would I need anything else? However the point is that when dealing with proprietary hardware devices you occasionally get a situation where the incantations as "documented" should work, but they don't, and the usual software process diagnostics are silent about it. Some domain specific experienced creativity is required to coax a response to begin finding the illness. Yes you can pay for support and escalate but small shop management sometimes is a bit hesitant to pay for that.
I am very curious how an LLM is supposed to be trained on situations whose context does not exist on the open internet.
Perhaps only after an assignment where prompts and answers are shown, and most of the answers look plausible but have a bunch of subtle mistakes, and students need to determine which ones are wrong and why.
I recently learned javascript this way - knowing English and other languages (Thai, Mandarin), I kept pasting my pseudo-code, asking GPT to convert it into javascript, and then iterating on that. Took me a week or two to get the language without reaching for any sort of manual/tutorial. Speaking without experience - it would take me 2-4x as much time to learn the same thing otherwise.
:) My story is not entirely true, but close. My point being llms are learning language and logic (mostly English currently). Programming languages are just languages with logic (hopefully).
And if you think the ability to shorten meaning of a complex idea is exclusively the purview of code, think of a word like "tacky" or "verisimilitude"- complex ideas expressed in a shorter format, often with intended context with significant impact on the operations in the sentence around them.
Maybe I’m dumb but I’ve never found a single textbook, manual or tutorial that taught me everything I need to know about a subject. I always have to make a circuit of multiple sources and examples before I understand fully.
I guess what you really learned is that the Apple-specific ecosystem technologies are so crappy, you always use a middleware to defeat them. Either Unity, React Native, or in this case, a 170b parameter super advanced LLM.
There's just no getting through to those Apple guys to stop making their own thing and being a huge PITA.
I don't do Swift or JS/React professionally (well, tiny bit, but I'm not a "React professional"). However, I've made non-toy Swift/SwiftUI* (~7k loc) and React (~4k loc) project.
Imho Swift/SwiftUI is way more of a joy to work with. In particular, SwiftUI is much easier to visually parse whereas JSX is noisy and IMO hideous. I also ended up needing to bring in way more third-party dependencies for JS/React, something I hate doing if avoidable**.
* It is my understanding that many iOS professionals prefer UIKit bc SwiftUI is still missing stuff. I only ended up needing to fall back to it once or twice.
** ClojureScript/Reagent do solve most of my issues with JS/React (by essentially replacing them). Clojure is well-designed with a great standard library (so less need for dependencies or implementing stuff that should be in a standard library). Hiccup is way preferable to JSX.
The assistants might be good enough to produce homework level or test level answers to contrived questions. But then utterly fall over in a real world codebase.
I have yet to find an AI tool or combination of AI tools that can hold my hand and give me correct solutions for my coding problems.
I have worked exclusively with React/Next, Node, and Mongo for years until I took over a legacy code base in October.
I had to teach myself Python/Flask, Vue, and Laravel without any kind of documentation or guidance from the previous developers or anyone else in my company.
AI has been able to help me get a bit further down the road each time I run into an issue. But AI hasn't been able to hand me perfect solutions for any of my problems. It helps me better understand the existing code base, but thus far none of the answer provided have been 100% accurate.
Real world tasks don't fit in a chat window, and using AI is barely an improvement over not using it. Just from one of my latest LLM coding adventures: it gets versions wrong, imports wrong, messes up versioning when learning documentations, resulting in many wasted hours. It's ok to fill in boilerplate, not ok to help you do things you couldn't do without AI.
If anything, it's like cooking with kids, not like going to McDonalds. I'd give extra points if they can solve the task with LLM in time. The hardest parts in programming is finding subtle bugs and reviewing code written by others - two tasks that LLMs can't help us with.
If they can easily solve the tasks with LLMs then it is a legitimate question to ask if you should be teaching that skill. Only common ones can be solved that way though. Why don't you give them a bugged code to fix, that way LLM inspiration is not going to work, if you check first to make sure LLMs can't fix the bug?
I don't think I agree with your last paragraph. ChatGPT is getting better and better, I have reason to think it won't continue to keep incrementally improving. As such, I think allowing students to use it to answer coding questions is going to make it so they don't actually understand anything.
I said in sibling thread that I'd be fine enough with having a class like "Software Engineering Using AI" or something, but when the class is specifically about learning Object Oriented programming and Java and Python, I do not think having heavy use of ChatGPT is a good idea.
Also, not all the questions were pure coding, I had some more conceptual questions on there, and ChatGPT is really good at answering those.
If you throw the same question at it 15 different ways, it can eventually give you ideas for optimizations that you probably wouldn't have thought of otherwise. It knows parts of APIs that I've never used. ByteBuffer#getLong, ByteBuffer#duplicate, StringBuilder#deleteCharAt, RandomGenerator#nextLong(long)
> I do not consider "prompt engineering" to be a substitute for programming
It's also not nearly mature enough for learning it be a good ROI in a degree program. Community college or adult education class? Maybe. Bootcamp track? Sure, I guess. Novelty elective for a few easy credits and some fun? Totally.
But if I'm a student in the midst of a prolonged, expensive program spanning years, learning how to coax results out of today's new generative AI tooling is not preparing me very well at all for what I can expect when I try to enter the workforce 2 or 4 or 8 more years. The tools and the ways to interface them, "Prompt Engineering" or whatever else it's called will inevitably evolve dramatically between now and then. So why am I learning it while I'm still deep in my academic bubbles? And what are prospective employers getting from a degree that focused heavily on some now-defunct and dated techniques? My degree is supposed to mean that I've learned foundational material and am ready to be productive on something, but what that mean when too much of what I've learned is outdated?
If there were a class called "Utilizing AI for software engineering" or something, even at the university level, that wouldn't bother me.
What bothers me is that people have told me that I should just allow AI for literally everything because it's the future and you should be teaching them the future or something, but I think that's kind of dumb. In those classes, the goal was for them to leave having some competence in with Python, Java, and Object Oriented programming, and I firmly do not believe you can get an understanding of that just by copying and pasting from ChatGPT, and I think even Copilot might hinder the process a bit.
To be clear, I love ChatGPT, I use it every day, it's a very valuable tool that people probably should learn how to use, I just don't feel that it's a substitute for actually learning new stuff.
> What bothers me is that people have told me that I should just allow AI for literally everything because it's the future and you should be teaching them the future or something,
I wonder what "people" told you that. My personal experience is that such advice usually comes from people who understand neither AI nor what I teach. Most of them are university administrators of sorts, and AI is a problem for them more than for me.
Introductory classes teach skills that can be performed reasonably well by AI. Those skills are the foundation you need to build higher level skills. Just like kids need to know how to read to be functional in society and in their later classes, despite screen readers doing an excellent job.
When I teach a foundations class this is my focus, and I don't fool myself or my students into thinking that they will be using those skills directly, but I try to convey the idea that the skills pervade through much of what they will later learn and do.
However that means that I cannot force students to learn. They can cheat, and it's easy, and I prefer spending my efforts on helping the learners than catching the cheaters.
The university administrators, however, are in the business of selling diplomas, which are only worth what the lowest common deminator is worth. So cheaters are a big problem for them. Typically for such people, they just bury their heads in the sand and prefer to claim that teaching students to use chatGPT (that lowest common denominator) is where the value is.
I'm not sure if you're implying that I'm making shit up by putting "people" in quotes, but here's at least a little evidence. [1] [2] [3]
Otherwise it's been with in-person conversations and I didn't record them, there's a spectrum to how completely they suggest I allow AI.
Everything else you said I more or less agree with. Obviously if someone wants to cheat they're going to do it, but I feel that until we restructure the job market to not take GPAs as seriously (which I think I'd probably be onboard with), we should at least have cursory efforts to try and minimize cheating. I'm not saying we have to have Fort Knox, just basic deterrence.
I'm not an adjunct anymore, partly because I took stuff way too personally and it was kind of depressing me, partly because it was time consuming without much pay, but largely because I realized that most universities are kind of a racket (particularly the textbook corporations are a special kind of evil).
I wasn't implying anything. Because your use of the word "people" left the context very vague, I was just cautiously trying to not speak in your name when discussing the people who are pushing me to teach students how to use chatgpt.
With respect to deterring teaching I totally agree that we should go for it. There are ways to mitigate the value of cheating and ways to promote the value of learning, both of which are deterrents. Personally I like having lots of small tasks that build and follow on one another. If the student is working and trying to learn it makes sense and we get to reinforce the high level skills that matter. If the student is cheating it should become increasingly harder to keep a consistent story.
However if we turn this into a cop and robbers game that's what we're going to get.
As for the focus on GPA I think that the tide is turning. Employers need to find an alternative that doesn't eat their time.
And yes universities are rackets and aren't good value employers. They don't even offer job security anymore.
If it’s an introductory course I can see how using generated code would be harmful but I’m guessing the students can’t since they don’t have the experience needed to make a distinction between good and bad code but perhaps that’s the thing you could teach them about LLMs by having a class where they are given an “easy” problem that you might normally assign that an LLM can solve and then also a hard problem that that it will fail to do properly and let the students see the difference for themselves when they have it generate code for both. That may provide valuable insight.
I agree. I attempted to use Copilot more this week for a couple projects at work I had to get done quickly. I hadn’t been using it at all for the past few months, but when a co-worker asked for help getting it working it reminded me to take another look. These particular things seemed better suited for Copilot than some of the other work I had.
I found that having a solid foundation was critical for knowing if something would work, and even just to write a prompt that was semi-decent.
At one point I asked for what I wanted, but it kept only doing half of it and I could tell just by glancing at the code it was wrong. I then had to get very, very specific about what I wanted. It eventually gave a correct answer, but it was the long annoying option I was trying to avoid, so it didn’t change my end result, and I don’t even think it saved me any typing due to all the prompts to get there. It just gave me some level of confirmation that there wasn’t an obvious better way that I’d be able to find quickly.
Knowing syntax is a side-effect of having written a minimum number of LoC, so grading syntax is in effect rewarding experience. Similar to how a large vocabulary is a side-effect of having read a lot.
To me, exams are taken in halls, written on paper, proctored, under deadline. Points may be deducted for syntax mistakes or unclarity as the examiner wishes.
Separately, home work is graded in ways that already makes cheating pointless; usually for 1) ambition/difficulty in the chosen problem 2) clarity in presentation and proofs, argued in person to TAs.
LLM should have no bearing on any part of education (CS or otherwise) unless the school was already a mess.
I dunno I wasn't all that familiar with Python (I'd done a Django project a decade ago) and I picked Python for a recent project writing some control code for some hardware connected to a Linux box and I have learned quite a bit about Python by doing it.
Perhaps the lesson is about how to read and evaluate code and how to test code. If students get good at how and when to spot errors and how to construct test scenarios that ensure the code is doing what it should then perhaps that will lead to even higher quality code than if they were learning, producing bugs, and then learning how to evaluate the code and test what they had written.
> it's very easy to get AI to write some code and copypaste it, not understand anything it said, and be done.
This is why I think that the old way of having tutorials instead of tests is vastly superior. When you are in a tutorial group (typically five or fewer students) and your tutor asks you to explain something to the other members of the group you can't hide behind an AI, a textbook, or even your own notes. Your lack of preparedness and understanding is made abundantly clear.
> Last year, when I was administering a test, I said that they were allowed to use any IDE they wanted, but they could not use any AI assistance. I am pretty convinced one student just copypasted my test questions into ChatGPT or BingAI and then copied the answers. I didn't have a way to prove it so I had to give him the benefit of the doubt and grade it assuming he did it honestly.
If I were a teacher, I would have asked the popular bullshit generators to generate solutions for the test questions, and if a student’s solutions were very similar, I’d have them do some 1:1 live-coding to prove their innocence.
I did actually try and get ChatGPT and BingAI and Google Bard to generate it. I didn’t know about Claude at the time.
The issue is that the thing he submitted was correct, so I was going completely off “vibes”; I never got it dead-to-rights with AI generating a one-for-one match. It got pretty similar, and ChatGPT text does have kind of a recognizable style to it, but I didn’t feel comfortable reporting a student for cheating and risking them getting expelled if I wasn’t 100% sure.
I might have tried to get him to do a one on one coding session, but this was the final exam and literally three hours after it I had to fly to the UK for an unrelated trip for three weeks. Grades were due in one week, so I didn’t really have a means of testing him.
> Just because I can call an Uber doesn't mean I get to say I know how to drive. Just because I can go to McDonalds doesn't mean I get to say I know how to cook.
I hear what you are saying but where does it end? Is Hello Fresh cooking? Is going to grocery store cooking or do you need to buy it direct from a farmer? Do you need to grow the food yourself?
Is renting a car “driving”? Is leasing a car “driving”? If you can’t fix a car and understand how it works are you really driving?
Yes, most of those questions are ridiculous but they sound the same to me as some complaints about using LLMs. Those complaints sound very similar to backlash against higher-level languages.
Python? You’re not a real developer unless you use C. C? You’re not a real developer unless you use assembly. Assembly? Must be nice, unless you’re writing 0’s and 1’s you can’t call yourself a developer. 0’s and 1’s? Let me get out my soldering gun and show you what it takes to be a real developer….
So many strawmen. Following a recipe, even with the ingredients coming from a meal kit, is still cooking. If you’re behind the wheel, you’re driving, no matter who owns the car or if you can perform any maintenance.
A C developer working in notepad.exe is a real developer, as is a Python developer working in PyCharm and using the standard IDE features to improve their productivity. Someone blindly copy-pasting output from a LLM is not a developer.
Good, I’m glad you grasped the point of my comment. I was talking about the absurdity of people “gatekeeping” programming. Those arguments are just as silly as people saying using an LLM (in any capacity) is wrong and means you aren’t programming anymore.
Yes, blinding pasting code from an LLM does not a developer make. However that’s not what I suggested. I believe LLMs can be useful but you need to understand what it’s generating. The same way that SO is useful as long as you understand the code you are reusing (ideally modifying and reusing instead of a straight copy/paste).
> I believe LLMs can be useful but you need to understand what it’s generating
That was never in dispute and not disagreed with in what I wrote. I didn’t say “using an LLM is wrong in any capacity” nor is that implied by anything I wrote. I use ChatGPT daily, I even told students they should use it if they needed help with understanding concepts outside of class”. I didn’t “gatekeep” programming, I just said they couldn’t use AI during an exam.
> I believe LLMs can be useful but you need to understand what it’s generating.
Yeah, if only we had some way of EXAMining if the students understand what they were generating. Like, crazy idea, maybe we could have some kind of crazy test where they aren’t allowed to use ChatGPT or Copilot to make sure they understand the concepts first before we let them have a hand-holding world.
I never said you said that, I said some people. I never said you should or shouldn’t use LLMs on exams, I have no idea why that’s being brought into this conversation.
I can only assume you’ve lost the thread and/or think you’re replying to someone else. This will be my last reply.
Your first post was in response to me claiming that going to McDonalds wasn’t cooking or getting an Uber isn’t the same as knowing how to drive. Specifically, they you were responding to was an anecdote about a student cheating on an exam by using ChatGPT, at least it appeared that way to me. It’s not weird to think your response was in regards to that.
Maybe I misread your intent there, I just got a vibe that you were defending the use of LLMs during exams since that was the thing you were responding to. Apologies if I misread.
> I am pretty convinced one student just copypasted my test questions into ChatGPT or BingAI and then copied the answers. I didn't have a way to prove it so I had to give him the benefit of the doubt and grade it assuming he did it honestly.
I wonder if a more AI-resistant approach is to provide poor code examples and to ask students to improve it and specifically explain their motivations.
They can’t just rely on code output from an LLM. They need to understand what flaws exist and connect the flaw to the fix, which requires (for now, I think) higher-level comprehension.
I would honestly think it doesn't matter. Students, like professionals, should be given the world and all of it's resources to generate a solution. If I can just punch the question into AI and get a valid answer then maybe the question should be modified.
If you always defer to a computer then you never learn to disagree with the computer, you become just as much of an automaton as it.
I love LLMs for coding but only because I have the experience to instantly evaluate every suggestion and discard a double-digit percentage of them. If an inexperienced programmer uses Copilot they aren’t going to have the confidence to disagree and it will stunt their development.
Does that really parse? There's plenty of things that have definite answers that are a Google search away, but we still expect things be able to do.
Did we just stop teaching kids basic spelling because spell check was built into MS Word? No, of course not, because even though spelling has been a more-or-less solved problem for people who already know how to read for the last thirty years, having kids learn to spell helps with their actual comprehension of a subject.
Also, if they do not learn the fundamental concepts then they will be able to differentiate a good solution from a bad solution. There's a lot of really shitty code that does technically accomplish the goal that it sets out for, and until you learn the fundamentals that fact won't be clear.
Do you also think that primary school students should not be taught, say, multiplication? Because they can, of course, use a calculator. No need to ask them what 3 times 4 is.
Early 2023, when everyone started using chat GPT for coding, I thought it would be a big boost because it enabled us to quickly employ a large number of people in a project regardless of language or framework.
An year into the project I am forced to revise my opinion. When browsing my code-base I often stumble in abstruse niche solutions for problems that should not have existed. It was clearly the work of someone inexperienced walking through walls in an AI-fuelled coding frenzy.
Having an oracle that knows all answers is useless if you don't know what to ask.
> It was clearly the work of someone inexperienced walking through walls in an AI-fuelled coding frenzy.
Isnt this what code reviews are for? I catch a decent amount of code that looks AI generated. Typically, some very foreign pattern or syntax that this engineers never used nor is common in the codebase. Or something weirdly obtuse that could be refactored and shows a lack of understanding.
Normally I ask something like, "Interesting approach! Is there a reason to do it this way over (mention a similar pattern in our codebase)?" or if it's egregious, I might ask, "Can you explain this to me?".
This feel similar to early career engineers copy pasting stack overflow code. Now its just faster and easier for them to do. It's still fairly easy to spot though.
You would. And if the juniors don't really _write_ code in the languages you use, if they don't make mistakes and do research themselves, it'll take them a lot longer to learn them sufficiently to be able to do those reviews.
Listening to people talk in Japanese, I picked up enough to have some idea what they're talking about. I can't speak the language beyond some canned phrases. I certainly wouldn't claim to know Japanese. And I definitely wouldn't get a job writing books in Japanese, armed with Google Translate.
For programming, my solution to this right now is lots of pair programming. Really gives you a good idea of where somebody is at, what they still need to learn, and lots of teaching opportunities. I just hired a junior and we spend about 8 hours a week pairing.
Yeah. Are we assuming 100% of the people working on this project are incompetent? I thought OP was talking about new people joining a project producing low quality code that was likely generated by AI.
Code reviews, done by existing more senior members of the project/team, should prevent this. If it isnt then the project has more issues than AI generated code.
There is no substitute to doing something correctly in the first place. The problem is that in the real world, deadlines and lack of time will always cause the default solution to be accepted a small percentage of time even when it is not ideal. The increasing creep of AI will only exacerbate that and most technophiles will default into thinking of a new and improved AI tool to help with the problem, until it will be AI tools all the way down.
I would agree with there being no substitute for doing something correctly the first place, but I would argue that in this case the "first place" is hiring/training better so your employees don't try to throw unrefined AI shit at the walls, but instead take the output of AI and hone it before creating a PR.
If you have an engineering culture that doesn't emphasize thorough code review (at least of juniors, leads and architects emergency-pushing is a different story) that's a problem. In addition to catching bugs, that's a major vector for passing on knowledge.
I agree with you, no doubt there. But still, technology often offers the path of least resistance. So even if you want to hire better people, what of all the people who will grow up training themselves with Copilot? Yes, you can filter them out but it will be harder and harder to find good people. And then, companies will want to go after the bottom line: what if they can get a bigger and more complex product out there with AI assistance but with code that is more unreliable in the long term?
I tell you, it's spiralling out of control. Besides, even if you're doing the hiring, you may not have control if there is a profit motive. Us technical people cannot control the race to the bottom line, especially over the period of decades.
A foundational concept of quality control is to not rely on inspection to catch production defects. Why not? It diffuses responsibility, lets more problems get to the customer and is less efficient than doing it correctly to start with.
Code review isn’t inspection, that would be testing - whether automatic or manual.
Code review is the feedback mechanism that allows quality control to improve the process, teaching the engineers how to do it right in the first place.
Sure. If its generating 85% code that passes a code review and 15% of code that doesnt... I'll take those percentages. That is certainly higher than the average early career engineers pr.
If you're trying to do things efficiently you can't afford not doing code reviews. Only well-funded organizations can afford to write the same code twice after the first attempt has to be thrown out.
I work at a FANMG company and see the value of code reviews for th code we're working on. But, I shipped 20 commercial products before that with no code review, just play testing (games). Our velocity without code review was easily 2x what it is with. For those projects I think that would still be true whereas I'd never suggest getting rid of code review at my current job.
The point being I think it depends on the project, it's size, audience, lifetime, etc whether or not code review is a win. my gut says it usually is a win but not always
At the one company I was where we did code reviews, they took at most one hour per day. The reviewer would go through the changes, ask for explanations when they didn't understand something, suggest small modifications to fit the codebase's style/structure/vibe, and move on.
They definitely didn't take 50% of our time. Half my time wasn't even spent coding, it was spent thinking about problems (and it was a workplace with an unusually high coding-over-thinking ratio, because of its good practices).
> At the one company I was where we did code reviews, they took at most one hour per day.
> They definitely didn't take 50% of our time.
Imo introducing code reviews might very well decrease the velocity by 50% even if the actual time spent on code reviews is much less.
I see at least two reasons for this: 1) increased need for synchronization/communication 2) increased subjective frictions and mental overhead (if only for task switching).
Code reviews are not special in this respect, similar effects can be caused by any changes to the process.
Thorough code reviews are expensive. When I review code, I have to know how to solve the problem well in the first place—that is, do all of the effort that went into writing the original code under review, minus time spent typing.
Less time may be required if you have good rapport and long work relationship with the person, but I would say halving productivity sounds about right if we are talking mandatory reviews.
Using an LLM to generate code means having to do the same. The longer the autocompleted text, the more careful you have to be vetting it. Lower predictability, compared to a person and especially one you know well, means that time requirement would not go down too much.
> When I review code, I have to know how to solve the problem well in the first place—that is, do all of the effort that went into writing the original code under review, minus time spent typing
Agree 100%, even when LLMs aren’t involved.
Certainly not all code reviews are like this—in many cases the approach is fundamentally sound, but you might have some suggestions to improve clarity or performance.
But where I work it’s not all that rare to open up a PR and find that the person has gone full speed in the wrong direction. In those cases you’re losing at least a morning to do a lot of the same leg work they did to find a solution for the problem, to then get them set off again down a better path.
Now in the ideal case they understood the problem they needed to solve and perhaps just didn’t have the knowledge or experience to know why their initial solution wasn’t good; in such a scenario the second attempt is usually much quicker and much higher quality. But introduce LLMs into the mix and you can make no guarantees about what, if anything, was understood. That’s where things really start to go off the rails imo.
> in many cases the approach is fundamentally sound
Sure, but I mean you still have to determine whether it is sound or not in the first place; i.e., you must know how to solve the problem. You can only evaluate it as right or not if you have something to compare against, after all.
If you have worked with a person for a while, though, I can see that you can spot various patterns (and ultimately whether the approach is sane) faster thanks to established rapport. Not so with LLMs, as you probably agree.
I mean, it's something we did once per day, usually towards the end of the day. This was pre-Covid, so our manager was in the room next to ours and we could just pop by his office to tell him we were ready for today's review.
It really doesn't need to be a massive amount of task-switching. And the benefits were obvious.
It probably also depends on the people that you're working with. I can easily imagine the velocity plummeting when the person reviewing your code loves to nitpick and bicker and ask for endless further changes with every next round of alterations that you do.
Doubly so if they want to do synchronous calls to go over everything and essentially create visible work out of code that already worked and was good enough in the first place.
I'm not saying that there aren't times like that for everyone occasionally, but there are people for whom that behaviour is a consistent pattern.
OMG this. Nitpicking code reviews are the worst. I've actually instituted a rule that code must be formatted with an opinionated formatter and linted, and whatever the linter/formatter decide is right, and no formatting/linting comments are allowed on code review to avoid this sort of bullshit review comment.
Code review matters when you have a product that has an indeterminate lifespan. When new people come on, they need to learn the codebase and code review both helps ensure a codebase that is easy for new devs to understand (because hard to understand code doesn't survive review) and also it provides an onboarding experience as new devs can be coached during code review.
"One and done" products like games where you're mostly fixing bugs and releasing new content without major overhauls to the underlying code probably only need code reviews for code that is complex/clever where bugs are likely to be hiding, beyond that QA is fine.
Not just thrown out, but also the customer relations when the bugs affect customers, the investigation to find the cause of the bugs, the migrations to fix any bad/corrupted data because of the bugs. Code review is a far cheaper way to catch errors, bad code, footguns and places where bugs might creep in than letting them into production and having them impact customers. Code reviews also help keep code tidier and cleaner which makes future understanding and changes easier. Small companies can’t afford to omit code reviews.
Can you elaborate on some scenarios you are describing? The only organization I can imagine that can't do code reviews would be organizations of 1. I don't even see how performing code reviews could ever be a net negative in terms of money nor overall productivity.
Even if you're not working in the same language/framework/whatever, you can still do code reviews. You might not know all the syntax and high-level details, but even just an overview of the logic happening in the code is already better than 0 review of any kind
Even if you're not working in the same language/framework/whatever, you can still do code reviews. You might not know all the syntax and high-level details, but even just an overview of the logic happening in the code is already better than 0 review of any kind
usually 'organization' excludes projects consisting of just one person, and if there are two people working on a project that involves writing code, the cheapest amount of that code for both of them to be familiar with is never zero, though it is often less than 100% of all the code they write. so i don't think that 'not all organizations can afford code reviews' is true, interpreted in the usual way. maybe you can be more specific about the scenario you're thinking of?
As mentioned in the other comments, code reviews are typically not done in the game industry, for example, or other write-once, never read scenarios. Very early stage startups typically don’t have formal code reviews either.
while that is true, and some of it is rational, other times it's just another stupid way people waste money and fuck up their projects. because even in a startup or a game there is code that people waste time debugging alone, or waiting for the one guy who knows it to get back from vacation, or out of bed
I recently was stumbled upon a code change done by a colleague who just followed the suggestion as is from Copilot, where it recommend `func (string, string) (error, struct)` whereas everywhere in the code we use `func (string, string) (struct, error)`
When I asked him what prompted him to do that, he said copilot suggest it so I just followed. I wonder if you could hijack copilot's results and inject malicious code as many end users does not understand lot of the niche code it generates sometimes, you could manipulate them to add the malicious code to the org's codebase.
The insidious thing is that quite possibly, it "feels" there should be a couple of bugs like that because all the codebases it was trained on had a few.
It might even take the context of the typos in your code comments, and conclude "yeah, this easy to miss subtle error feels right about here".
That is a problem but thankfully there is a lot of attention on training with highly curated high quality data right now because it is a known problem. Buggy code is still valuable training data if you use it as part of a question and evaluate the response against a corrected version of the code when training the model to perform a task like bug fixing.
It's definitely possible to inject malicious code that humans don't spot, there was a whole competition dedicated to humans doing this in C well before LLMs: https://en.wikipedia.org/wiki/Underhanded_C_Contest
Now I'm wondering, can you put in a comment which the LLM will pay attention to such that it generates subtle back-doors? And can this comment be such that humans looking at the code don't realise this behaviour will be due to the comment?
Almost two decades ago, I saw a junior colleague (this was Java) try to add "static" to almost all of one specific class' methods and members, i.e. making them global instead of attached to each class instance . Obviously this completely broke the software, albeit it did build. When questioned during review, my colleague just shrugged and said "Because Eclipse suggested to do so".
Apparently, if you tried to access a class member without specifying a class instance, one of Eclipse's "auto-fix-it" suggestions was to make all members of that class static, and he just followed that suggestion blindly.
> A year into the project I am forced to revise my opinion. When browsing my code-base I often stumble in abstruse niche solutions for problems that should not have existed.
This is a widespread problem regardless of AI. Hence the myriad Stack Overflow users who are frustrated after asking insane questions and getting pushback, who then dig their heels in after being told the entire approach they’re using to solve a problem is bonkers and they’re going to run into endless problems continuing down the path their on.
Not that people aren’t on too fine a hair trigger for that kind of response. But the sensitivity of that reaction is a learned defense mechanism for the sheer volume of it.
> Hence the myriad Stack Overflow users who are frustrated after asking insane questions and getting pushback, who then dig their heels in after being told the entire approach they’re using to solve a problem is bonkers and they’re going to run into endless problems continuing down the path their on.
The problem is, SO can't tell someone who asks an insane question from someone who asks the same question but has constraints that make it sane. *
So in time, sane people during unusual stuff stop asking questions and you're left with homework.
* For example, "we can't afford to refactor the whole codebase because some architecture astronaut on SO says so" is a constraint.
Or another nice one is "this is not and will never be a project that will handle google-like volumes of data".
> The problem is, SO can't tell someone who asks an insane question from someone who asks the same question but has constraints that make it sane.
Stack Overflow is not there to help you solve your use-case. It's there to create a body of knowledge that everyone can refer to. You need to spell out your specific reasons so that the question and answers become useful to others.
A lot of the friction on Stack Overflow comes from people thinking it's a free help website rather than an attempt to create a collaborative knowledgebase.
I think a lot of SO's problems comes from it trying to pretend it was building a knowledge base when it was really only ever a question and answer site.
It was nice to use early in its life when every answer was fresh. These days, the stale knowledge outweighs the fresh and the top answer is usually no longer correct.
If SO wants to be a knowledgebase, it needs to to be redesigned as it's current structure is not well suited to it.
How ‘bout remembering a million tokens? I’m not feeling too confident about that. Basically my only moat, if there is one, is that I’m able to rely on higher level cognition which LLMs don’t yet have, rather than just on associative memory alone.
I might, actually. Think of where electric cars were six years ago — 2018. Not much has changed. Or, at least, there are still fundamental problems to be solved.
In the same way I can imagine that by 2030 LLMs will still have memory problems and hallucinations. Although I’m sure by then we’ll have something better than pure LLMs.
I've heard claims that context without forgetfulness has already been reached 2 months ago, but as I'm not a domain expert I don't trust that I can differentiate breakthroughs from marketing BS, and I definitely can't differentiate either of those from a Clever Hans: https://arstechnica.com/information-technology/2024/03/claud...
I work in this field, so here's a comment with higher signal-to-noise ratio than you'll commonly find on HN when it comes to LLMs: notice how the demo use cases for very long context stuff deal almost universally with point retrieval, and never demonstrate a high degree of in-context learning. That is not coincidental. The ability to retrieve stuff is pretty great and superhuman already. The ability to reason about it or combine it in nontrivial ways leaves a lot to be desired still - for that you have to train (or at least fine tune) the underlying model. Which IMO is great, because it neatly plugs the gaps in human capability.
So do I, and worse. Look, all I’m saying is I’m thankful for this crutch that helps me deal with the limitations of my associative memory, so as long as it can’t think and can’t replace me entirely
bacopa and lions mane was night and day for my limited memory. But, the obvious, writing simpler code, keeping a notebook while working, etc, and really spending time breaking down problems into much smaller scopes, while simultaneously keeping copious notes on where I am in a given process helped immensely with dealing with peanut brained memory. sure, it's not quick, but my work is usually very readable and understandable for the future reader. I'm not convinced that a tool to help me overcome that memory barrier would actually help me write better code, maybe just write worse code faster. Of course, that's probably the corporate goal though.
Lions mane didn't seem to do much for me. My memory is actually not that bad, though certainly I have seen people with _much_ better memory than mine. I still maintain contact with some of them I met in college, and then later in the various companies. It's just that I deal with so much information and so many streams of it, even writing it down would be a massive chore.
I would pay for a pre-packaged system which could _locally_ and _privately_ make sense of all the emails, PDFs, slack messages, web pages I saw, other documents shared with me, code, etc etc, and make it all easily queryable with natural language, with references back to the sources. Sooner or later someone will make something like that.
The context heads of a LLM are more analogous to the sort of processing that goes on in, e.g., Brocca's Area of your brain as opposed to working memory. You can't have anything analogous to working memory as long as LLMs are operating on a strict feed forward basis[1]. And the fact that LLMs can talk so fluently without anything like a human working memory (yet) is a bit terrifying.
[1] Technically LLMs do have a forget that last toke and go back so I can try again operation so this is only 99% true.
I think humans have better general recall whilst lacking any kind of precision. After reading an entire book, I definitely can’t replicate much (if any) of the precise wording of it, but given a reasonably improbable sentence I can probably tell with certainty that it didn’t appear. LLMs are probably much more prone to believing they’ve read things that aren’t there and don’t even pass a basic sanity check, no matter how long the context window.
>> Having an oracle that knows all answers is useless if you don't know what to ask.
That is a great point. The issue of not asking the right questions has been around as far as I can remember but I guess it wasn't seen as the bottleneck because people were so focused on solving problems by any means possible that they never had to think about solving problems in a simple way. We're still very far from that though and in some ways we have taken steps back. I hope AI will help to shift human focus towards code architecture because that's something that has been severely neglected. Most complex projects I've seen are severely over-engineered... They are complex but they should not have grown to hundreds of thousands of lines of code; had people asked the right questions, focused on the right problems and chosen the right trade-offs, they would have been under 10K lines and way more efficient, interoperable and reliable.
I should note though, that my experience with coding with AI is that it often makes mistakes for complex algorithms, or it implements them in an inefficient way and I almost always have to change them. I get a lot of benefit from asking questions about APIs and to verify my assumptions or if I need a suggestion about possible approaches to do something.
How do you know it was "AI-fuelled"? And what makes it a "frenzy"?
People have been committing terrible code to projects for decades now, long before AI.
The solution is a code review process that works, and accountability if experienced employees are approving commits without properly reviewing them.
AI shouldn't have anything to do with it. Bad code shouldn't be passing review period, no matter if it was AI-assisted or not. And if your org doesn't do code review, then that's the actual problem.
> and accountability if experienced employees are approving commits without properly reviewing them.
You’re putting the entire responsibility on senior employees. So we need much more of them. In fact, we don’t need juniors, because we can generate all possible code combinations. After all, it’s the responsibility of the seniors to select which one is correct.
It’s like how hiring was made crap by the “One-click apply” on LinkedIn and all other platforms. Sure it’s easy for thousands of people to apply. Fact is, we offer quite a good job with high salary, and were looking for 5 people. We’ve spent a full year selecting them, because we’ve receive hundreds of irrelevant applications, probably some AI-generated.
It’s no use to flood a filter with crap, hoping that the filter will do better work because it has a lot of input.
This is almost a special case of the very similar (and common) argument that ‘AI-generated misinformation isn’t bad; we’ve always had misinformation’.
The answer is also the same.
Volume. AI makes it trivially easy to generate vast amounts of it that don’t betray their lack of coherence easily. As with much AI content, it creates arbitrary amounts of work for humans to have to sift through in order to know it’s right. And it gives confidence to those who don’t know very much to then start polluting the informationsphere with endless amounts of codswallop.
> Having an oracle that knows all answers is useless if you don't know what to ask.
Honestly I find this to be the biggest advantage to using a coding LLM. It's like a more interactive debugging duck. By the time I've described my problem in sufficient detail for the LLM to generate a useful answer, I've solved it.
I’ve had a project I’ve been doing for ~6 months learning python through Copilot+ChatGPT. It feels like any other project that accrues technical debt, but that debt is just a lot weirder. It was rough at first, but refactoring has gotten a lot better recently with bigger context sizes.
Good one, however, like Google making memory worse, copilot seems to erode knowledge of programming quite quickly. Last week I was with a colleague at my house; when I hired him during covid he knew these things just fine, but after not having to type in a list comprehension for around 2 years now, when my Starlink was down for a bit, he couldn’t for the life of him create one; he asked Claude opus on his phone instead.
For real. I have the same experience. I'm 19, freelance developer writing Python among other things. Except I never learned Python syntax, and maybe never will.
For a senior-level professional? Yes, it does matter.
I wouldn't be comfortable with an accountant who couldn't do practical arithmetic in their head, or a surveyor who didn't have a fluent grasp of trigonometry and the ratios or function values for common angles.
Of course -- I don't care about the people working for them as a bookkeeper or assistent. They can be button monkeys as much as their boss lets them, but I also wouldn't expect those folk to reach very high in their career. Not everyone's going to, and a disinterest in and lack of technical fluency is a darn good predictor.
Your teacher was giving good advice about building skills and internalizing knowledge, because those are what contribute to mastery of a craft; maybe you were just being too pedantic or cynical to hear it?
I think this advice is backwards, as is the problem. I’d trust an accountant’s mind or a builder’s eye if I knew that they are acting from experience. But you don’t wanna trust someone who believes in-their-head is superior or reliable and makes you good.
The core idea is no, calculating in your head doesn’t turn you into a pro. Being a pro makes calculating in your head safe enough to rely on until you get to your desk and double-check.
That said, I was an integrator half my life and seen some accountants, big and small. Everyone used a calculator, it’s right next to their mouse and there’s another one elsewhere. And no one ever tried to talk about numbers afk, except for ballparks. Most of the times they suggest to go and query a db together. I find it very professional.
I just trust my accountant will use excel and what they are doing isn't rocket science.
I have no interest interacting with someone who is going to get into how great the slide ruler is and how kids these days need to learn the slide ruler. When I was younger I thought this type of person was highly admirable but now older and wiser I see how full of shit they are.
>> If you need Copilot to code in Python, have you really learned Python?
> Does it really matter though? It sounds awfully like when a school teacher said you're not going to have an calculator in your pocket.
Yes. A younger me had teachers say that to me, and that younger me thought they were wrong.
But it turns out they were right, and younger me was wrong. Calculator dependence created a ceiling built from "educational debt" that many years later severely limited my mathematical ability.
The problem is focusing too much on getting the "answer," and losing sight of building the abilities that allow you to get there with your own mind. Eventually the need to manipulate your automated crutch turns into a bottleneck.
It’s also kind of annoying to intuitively get to the correct answer and have everyone claim they can’t distinguish between that and you using a calculator.
I wonder if that would be an issue if nobody had any.
Isn't it obvious? If you can't do arithmetic without a calculator, it makes it hard to do algebraic manipulation. (Un)luckly a calculator that could do that arrived in my lap just in time. Then you get to calculus, and it was the same story. I eked by with a lot of furious typing, but the mental workload of that made it untenable towards the end and I wasn't really gaining much anyway because of that. It would have been far better if I'd just hadn't been allowed to use a calculator from the start.
It didn't affect my ability to reason and prove things, just as long those things don't strongly require the knowledge and skills I should have gotten from the calculator shaped gap in my education. I lack a lot of the background knowledge and/or familiarity and comfort with many skills that I should have.
I think we're using the term “mathematical ability” in different ways. For me, it's an ability to prove a theorem, not to solve a differential equation.
Yes, it does matter. While there are plenty of situations where it's foolish to forbid calculators, that's not universally true.
Forbidding calculators and requiring students to do mental math for absolutely everything is unnecessary. But requiring students to solve integrals by hand when they're learning about integrals? Entirely reasonable.
If your goal when teaching coding is to teach the mechanical process of writing code: sure, go ahead and use LLMs for that process. But if your goal is to develop a deeper understanding of how to code, then LLMs can very easily obscure that. The end goal is not always just the answer.
> But requiring students to solve integrals by hand when they're learning about integrals? Entirely reasonable.
I remember having to write code on a piece of paper in university.
Curiously, this lead to a lot of other students not really learning or caring about indentations and attempting to make code formatted in a generally readable way once they actually got access to IDEs and such.
They'd write it much like you would regular text on a piece of paper and would get absolutely stumped with unbalanced brackets whenever some nested structure would be needed.
Not only that, but somehow a lot of them didn't quite get around to using the various refactoring features of the tools, since they treated them like editors for writing text.
Did it help me memorize the basic constructs of the languages in question? Absolutely not, at least in the long term, since nowadays I have to jump between a lot of different languages and none of the syntax quite sticks, rather a more high level approach to problem solving.
LLMs just make that easier for me, letting me offload more work around language constructs and boilerplate, albeit requiring occasional intervention.
AI is not replacing the same function as a calculator does when solving a math problem.
Using AI to “help” learn to program is replacing the person’s effort thinking through the problems fully and is ultimately going to stunt their learning.
A calculator is fine so long as you understand what the operations actually represent. Remembering the multiplication table is not important, but understanding what multiplication actually does is.
The problem with AI-assisted coding is that, applied uncritically, it circumvents this understanding. And without it, you won't even be able to judge whether the code that CoPilot etc spit out is correct for the task.
yeah, at the moment anyway, because if you don't know what the syntax means, you can't tell when the code means something you don't want. that means you can't tell which copilot suggestions to accept and which ones to reject, and you can't debug your code; asking gpt-4 what's wrong with it only goes so far
imagine if the calculator in your pocket gave you subtly wrong answers most of the time
This is my new capability- I am not a coder or a programmer and I can get things built, in my own time, at my own speed, solo.
Would it be better code if someone with 3 years of university and 5 years of coding practice did it? Yes, very probably, but the gap seems to be narrowing. Humorously I don't know enough about good code to tell you whether what I build with llms is good code. Sometime I build a function that feels magical, other times it seems like a fragile mess. But I don't know.
Do I know "javascript" or "python" or the theory of good coding practice? No, not currently. But I am building things. Things that I have personal, very specific requirements for.
Where I don't have to liaise or berate someone else. Where I don't have to pay someone else. Where I don't share the recognition (if there is any ever) of the thing, I and only I have produced- (with chatGPT, Gemini and most recently llama3).
Folks have been feeling superior for 70 years and earning a good living because they spoke the intermediary language of compute engines. What makes them actually special NOW is computer science, the theory- the languages, we have very cheap (and in the case of open source local models free) translators for those now. And they can teach you some computer science as well, but that is still time and practice.
I'm the muggle. The blunt. And I'm loving this glowy lantern, this psi-rig.
One of the lessons that one learns as a programmer is to be able to write code that one can later read back and understand. This includes code written by others as well as code written by oneself.
When it comes to production quality code that should capture complex and/or business-critical functionality, you do want an experienced person to have architected the solution and to have written the code and for that code to have been tested and reviewed.
The risk right now is of many IT companies trying to win bids by throwing inexperienced devs at complex problems and committing to lower prices and timelines by procuring a USD 20 per month Github Co-Pilot subscription.
You individually may enjoy being able to put together solutions as a non-programmer. Good for you. I myself recently used ChatGPT to understand how to write a web-app using Rust and I was able to get things working with some trial and error so I understand your feeling of liberation and of accomplishment.
Many of us on this discussion thread work in teams and on projects where the code is written for professional reasons and for business outcomes. The discussion is therefore focused on the reliability of and the readability of AI-assisted coding.
hmm.
I ended up with a few 750+ line chunks of js, beyond the ability of chatgpt to parse back at me. So my go-to technique now is to break it into smaller chunks and make them files in a folder structure, rather than existing inside a single *.js
So readability is an issue for me- even more so because I rely on ChatGPT to parse it back- sometimes I understand the code, but usually I need the llm to confirm my understanding.
I'm not sure if this scales for teams. My work has sourcegraph, which should assist with codebases. So far it hasn't been particularly useful- I can use it to find specific vulnerable libraries, keys in code etc, but that is just search.
What I really need is things like "show me the complete chain of code for this particular user activity in the app and highlight tokens used in authentication" ... - something senior engineers struggle to pull from our hundreds of services and huge pile of code. And so far sourcegraph and lightstep are incapable of doing that job. Maybe with better RAG or infinite context length or some other improvement there will be that tool. But currently the combined output of 1000's of engineers over years almost un-navigable.
Some of that code might be crisp, some of it is definitely of llm-like quality (in a bad way)- I know this because I hear people's explanation of said code and how they misremembered it's function during post mortems. Folks copy and pasting outdated example code from the wiki etc. ie making things they don't understand. I presume that used to happen from stackoverflow too. Engineers moving to llm won't make too much difference IMO.
I agree, your points are valid, but I see "prompt engineering" as democratization of the ability to code. Previously this was all out of reach for me, behind a wall of memorization of language and syntax that I touched in the Pascal era and never crossed.
12 hours to build my first node.js app that did something in exactly the way I had wanted for 30 years. (including installing git and vscode on windows- see, now I am truly one to be reviled)
The problem I currently have with AI-generated code, as an experienced programmer who can read and understand the output, isn’t that the code quality is bad, but that it’s often buggy and wrong. I recently recommended in my company that if copilot is allowed to be used, the developers using it must thoroughly understand every line of code it writes before accepting it, because the risk of error is too high.
Copilot may work for simple scripts, but even for that where it mostly for things right, in my experience it still introduced subtle bugs and incorrect results more often than not.
I've been coding Python for 15 years, but I probably couldn't code in Python now without Copilot, or a lot of reference docs. There's so much meaningless trivia and idioms I've purposefully pushed out of my mind so I can focus on other things. If copilot has my back, why do I need to remember that crap?
Learning to be a good programmer is as much about learning how to avoid technical dept as it is about learning to use a programming language. It may take a while until an AI assistant will be able to help with that.
>Having an oracle that knows all answers is useless if you don't know what to ask.
This sentence summarizes the issue with the current AI debacle, along with the whole "just copy/pase code from stackoverflow and earn top bucks" meme that was going around in 2010s.
You're not gonna be a valuable dev if you're just write wrong code faster. Not only does chatgpt/copilot give haphazard code half of the time, it approaches seemingly random syntax and format. Even if LLMs are polished, you're gonna need stand software engineering knowledge to know what's right and wrong.
Coding just keeps getting grosser and weirder decade over decade as new layers of abstraction and complexity get piled on. Just about everyone who does this shit for a living eventually hits a break point where the headassery du jour becomes too much to turn a blind eye to.
LLMs are no excuse for bad code reviews or developers who don’t know what it’s spitting out.
In my code reviews the person who wrote the code needs to explain to me what they changed and why. If they can’t then we are going to have a problem. If you don’t understand the code that an LLM spits out you don’t use it, it’s that simple. If you use it and can’t explain it, well… we are going to have to have some discussions and if it keeps happening you’re going to need to find other employment.
The exact same thing has been happening for pretty much the entire time we’ve had internet. Stack Overflow being the primary example now but there were plenty of other resources before SO. People have always been able to copy/paste code they don’t understand and shove it into a codebase. LLMs make that easier, no doubt, but the core issue has always been there and we, as an industry, have had decades to come up with defenses to this. Code review being the best tool in our toolbox IMHO.
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.
It is exactly what happened to you: it wrote bullshit. Plausible bullshit but bullshit nonetheless.
It’s also just not as good at the task anymore. It frequently gets lazy and gives you an outline with a bunch of vague pseudocode. Compare to when GPT-4 was slower at producing output, but all of that output was solid, detailed work. Some of the magic that made you say “wow” feels like it’s been enshittified out of it.
I sometimes try the free chatgpt when I run into a problem and it's just hilarious how terrible it is. Loves to go around in circles with the same made up solution that has no basis in reality, using functions in libraries that would be great if they actually existed.
I noticed that like since a week ago? Output faster, but not impressive. Now I just skip to stack overflow or docs. The output is also giving error a lot more, as if the libraries on which the example is based off was old. Sometimes it's really trivial task just to save time, and it's just not of any help. Still helpful when you want to start something new, it just doesn't scale that well.
Yes, rather poor but people can always post new answers and votes sort the answers. It might not work all that well but there is a mechanism for improvement and to keep things up to date.
Language models can copy the top answers from SO, ingest docs and specs etc. And then the information is never updated? Or are they going to train it from scratch? On what? Outdated github saved games?
It is difficult to get a man to understand something when his salary depends on his not understanding it. ~ U. Sinclair
A significant problem is the subconscious defense mechanism or bias that compels us to conclude that AI has various shortcomings, asserting the ongoing need for status quo.
The capabilities of GPT-3.x in early 2023 pale in comparison to today's AI, and it will continue to evolve and improve.
I’m surprised to see this comment was downvoted heavily. The quote is very popular around here, to the point where there is an Ask HN if it’s the most quoted quote:
Well before AI co-pilots something happened to the good old admins--they started to disappear only to be replaced by "AWS devops" (their job titles) who have never wired a network using routers, switches, and cables. I noticed that they started lacking basic networking knowledge and couldn't set up networking inside AWS. They just didn't know what a gateway, NAT, or subnet is.
Similar things are happening with AI co-pilots. We have an increasing number of people who "write code", but the number of people who can understand and review code is not increasing. There is also a problem of injecting ethics and politics into those tools, which can produce silly results. I asked Bard to write me a Python function to turn the US Constitution into a palindrome. Bard refused and gave ma a lecture on how the US Constitution is too important to be played with in such trivial fashion. I then asked it to produce code that turns the US national anthem into a palindrome, it refused again. So I asked it do the same but with the Russian national anthem and it spat out code without telling me off. I then asked it to generate code for simple tasks and it did an OK job, except the formatting and the fonts used were different every time, because it just lifted code from different webpages and recombined it like a massively hungover student waking up to realise he's supposed to hand in the assignment in one hour.