I recently was stumbled upon a code change done by a colleague who just followed the suggestion as is from Copilot, where it recommend `func (string, string) (error, struct)` whereas everywhere in the code we use `func (string, string) (struct, error)`
When I asked him what prompted him to do that, he said copilot suggest it so I just followed. I wonder if you could hijack copilot's results and inject malicious code as many end users does not understand lot of the niche code it generates sometimes, you could manipulate them to add the malicious code to the org's codebase.
The insidious thing is that quite possibly, it "feels" there should be a couple of bugs like that because all the codebases it was trained on had a few.
It might even take the context of the typos in your code comments, and conclude "yeah, this easy to miss subtle error feels right about here".
That is a problem but thankfully there is a lot of attention on training with highly curated high quality data right now because it is a known problem. Buggy code is still valuable training data if you use it as part of a question and evaluate the response against a corrected version of the code when training the model to perform a task like bug fixing.
It's definitely possible to inject malicious code that humans don't spot, there was a whole competition dedicated to humans doing this in C well before LLMs: https://en.wikipedia.org/wiki/Underhanded_C_Contest
Now I'm wondering, can you put in a comment which the LLM will pay attention to such that it generates subtle back-doors? And can this comment be such that humans looking at the code don't realise this behaviour will be due to the comment?
Almost two decades ago, I saw a junior colleague (this was Java) try to add "static" to almost all of one specific class' methods and members, i.e. making them global instead of attached to each class instance . Obviously this completely broke the software, albeit it did build. When questioned during review, my colleague just shrugged and said "Because Eclipse suggested to do so".
Apparently, if you tried to access a class member without specifying a class instance, one of Eclipse's "auto-fix-it" suggestions was to make all members of that class static, and he just followed that suggestion blindly.
When I asked him what prompted him to do that, he said copilot suggest it so I just followed. I wonder if you could hijack copilot's results and inject malicious code as many end users does not understand lot of the niche code it generates sometimes, you could manipulate them to add the malicious code to the org's codebase.