This assumes that the companies gathering the data don’t have silent ways of detecting bad actors and discarding their responses. If you’re trying to poison an AI, are you making all of your queries from the same IP? Via a VPN whose IP block is known? Are you using a tool to generate this bad data, which might have detectable word frequency patterns that can be detected with something cheap like tf-idf?
There’s a lot of incentive to figure this out. And they have so much data coming in that they can likely afford to toss out some good data to ensure that they’re tossing out all of the bad.
Not necessarily, not all tactics can be used symmetrically like that. Many of the sites they scrape feel the need to support search engine crawlers and RSS crawlers, but OpenAI feels no such need to grant automated anonymous access to ChatGPT users.
And at the end of the daty, they can always look at the responses coming in and make decisions like “95% of users said these responses were wrong, 5% said these responses were right, let’s go with the 95%”. As long as the vast majority of their data is good (and it will be) they have a lot of statistical tools they can use to weed out the poison.
If you want to pick apart my hastily concocted examples, well, have fun I guess. My overall point is that ensuring data quality is something OpenAI is probably very good at. They likely have many clever techniques, some of which we could guess at, some of which would surprise us, all of which they’ve validated through extensive testing including with adversarial data.
If people want to keep playing pretend that their data poisoning efforts are causing real pain to OpenAI, they’re free to do so. I suppose it makes people feel good, and no one’s getting hurt here.
I'm interested in why you think OpenAI is probably very good at ensuring data quality. Also interested if you are trying to troll the resistance into revealing their working techniques.
What makes people think companies like OpenAI can't just pay experts for verified true data? Why do all these "gotcha" replies always revolve around the idea that everyone developing AI models is credulous and stupid?
It is absolutely fascinating to read the fantasy produced by people who (apparently) think they live in a sci-fi movie.
The companies whose datasets you're "poisoning" absolutely know about the attempts to poison data.
All the ideas I've seen linked on this side so far about how they're going to totally defeat the AI companies' models sound like a mixture of wishful thinking and narcissism.
Are you suggesting some kind of invulnerability? People iterate their techniques, if big techs are so capable of avoiding poisoning/gaming attempts there would be no decades long tug-of-war between Google and black hat SEO manipulators.
Also I don't get the narcissism part. Would it be petty to poison a website only when looked by a spider? Yes, but I would also be that petty if some big company doesn't respect the boundaries I'm setting with my robots.txt on my 1-viewer cat photo blog.
Its not complete invulnerability. Instead, it is merely accepting that these methods might increase costs, like a little bit, but they don't cause the whole thing to explode.
The idea that a couple bad faith actions can destroy a 100 billion dollar company, is the extraordinary claim that requires extraordinary evidence.
Sure, bad actors can do a little damage. Just like bad actors can do DDoS attempts against Google. And that will cause a little damage. But mostly Google wins. Same thing applies to these AI companies.
> Also I don't get the narcissism part
The narcissism is the idea that your tiny website is going to destroy a 100 billion dollar company. It won't. They'll figure it out.
Grandparent mentioned "we", I guess they refer to a full class of "black hats" avoiding bad faith scraping that eventually could amass to a relatively effective volume of poisoned sites and/or feedback to the model.
Obviously a singular poisoned site will never make a difference in a dataset of billions and billions of tokens, much less destroy a 100bn company. That's a straw man, and I think people arguing about poisoning acknowledge that perfectly. But I'd argue they can eventually manage to at least do some little damage mostly for the lulz, while avoiding scraping.
Google is full of SEO manipulators and even when they recognize the problem and try to fix it, searching today is a mess because of that. Main difference and challenge in poisoning LLMs would be coordination between different actors, as there is no direct aligning incentive to poisoning except (arguably) global justified pettiness, unlike black hat SEO players that have the incentive to be the first result to certain query.
As LLMs become commonplace eventually new incentives may appear (i.e. an LLM showing a brand before others), and then, it could become a much bigger problem akin to Google's.
tl;dr: I wouldn't be so dismissive of what adversaries can manage to do with enough motivation.
As someone who works in big tech on a product with a large attack surface -- security is a huge chunk of our costs in multiple ways
- Significant fraction of all developer time (30%+ just on my team?)
- Huge increase to the complexity of the system
- Large accumulated performance cost over time
Obviously it's not a 1-to-1 analogy but if we didn't have to worry about this sort of prodding we would be able to do a lot more with our time. Point being that it's probably closer to a 2x cost factor than it is to a 1% increase.
Normally I balk when commenters go “well they you’re the perfect person to go do it!”, but actually… this is the kind of thing that sounds like it could be a fun project if you’re legit interested. The necessary datasets are likely not hard to gather and collate, a lot of it is probably on places like Project Gutenberg or can be gleaned through OCR of images downloaded from various publicly available archives.
Granted, you’d need to spend about a year on this and for a lot of that time your graphics card (and possibly whole computer) would be unusable, but then if the results were compelling you’d get a cool 15 minutes of internet fame when you posted your results.
I got 15 minutes for basically a useless compiler and programming language that I spent 6 months on. Just for the effort-to-result ratio I feel like it's possible to do quite a lot better.
So here's the thing: Docker is the best way we have to document how to set up a project/application in a way that can be repeated on arbitrary computers. The alternative was "have a README where you list all of the things you need to do/install in order to get this project running".
That failed. Miserably.
Developers always assumed things like "well naturally, if you're playing in the XYZ space, you've already got meson installed. What, do you expect me to teach you basic arithmetic in this README too?" Developers across the board, across programming subcultures, showed themselves unable to get past this sort of thing.
So now we have Docker. You may not like it, but this is what peak install guide looks like. An unambiguous file that describes the exact shell steps required to get the piece of software running, starting from a base distro. The developer can't omit any steps or the container won't work on their machine.
It sucks that this Hegelian situation calls for such a draconian solution, but that's where we're at. Developers as a whole can't be trusted to handle this on their own. If you don't have a better solution to this problem, I'm not sure there's much point in complaining.
I think for the development story, we had vagrant in the 2010s which IMO provided a much better experience for developers to set up reproducible dev environments.
Docker excels at bundling up all the dependencies of a piece of software for deployment.
Devcontainers definitely work these days, but I miss vagrant.
I disagree completely. Vagrant worked for your org or your setup but people hardly ever (in my experience) delivered the recipe, or the steps to setup.
Yes, sometimes the vagrant-configure thing had a few lines, but most people shipped an iso with stuff installed. It could have been done, but wasn't being done.
Speaking as someone with similar views to the OP: my “better solution” is to write an idempotent shell script targeting a specific Debian release/ISO that handles system setup end-to-end.
It is for nearly all intents and purposes functionally equivalent to docker, and it’s pretty trivial to port to Dockerfile in minutes. I use docker plenty for work and am fully aware of its benefits. Like the OP, I just dislike Docker’s iptables fuckery and CLI design as a matter of personal preference.
Of course, context is king, and I only do this for things I’m designing and running myself - but the larger point I’m trying to make is that you can do the whole “unambiguous file that describes the exact shell steps required to get the piece of software running, starting from a base distro”-thing without Docker in the picture.
Dockerfiles were an excellent way of sysadmins getting developers to write down their build steps.
The fact that they're not deterministic was helped by the fact that we can just copy/paste tarballs around (all a docker image is, is just a pile of tarballs in a tarball after all).
I already knew about the whole “mathematically perfect corners” thing Apple does, so I was super curious how someone implemented that in CSS. I figured it was some sort of new CSS feature involving splines, but then I saw there was a folder called “masks” containing PNG files at 3 resolutions and I was immediately transported back to the mid-2000s.
Especially weird since CSS actually has the clip-path property which allows polygons as masks. I think converting a curve to a polygon is still better than having a literal raster image as a mask.
Oh man, the days before blocking net send was a common school sysadmin practice…
Back when I was a highschool freshman (2004-05) I wrote a batch script that would fire off net sends to everyone in the computer lab in rapid succession in an infinite loop, then just sort of left it on a shared drive with a conspicuous name. Sure enough, a few days later, someone ran it out of curiosity and got in trouble, but of course the file had my username in the metadata, and my computer teacher was like “Chris, you knew what you were doing, don’t do this again.”
It was the kind of “good clean fun” sort of prank that doesn’t get you in hot water or suspended, but was hilarious to watch play out.
Edit: Just re-read and saw that your friend got expelled for doing basically the same thing I did. That sucks. I’ll note that I went to an IT-focused votech school, so I think a lot of folks had a better sense of perspective as to how serious net send pranks actually were in the grand scheme of things.
Do you find that it actually generates varied and diverse stories? Or does it just fall into the same 3 grooves?
Last week I tried to get an LLM (one of the recent Llama models running through Groq, it was 70B I believe) to produce randomly generated prompts in a variety of styles and it kept producing cyberpunk scifi stuff. When I told it to stop doing cyberpunk scifi stuff it went completely to wild west.
You should not ever expect an LLM to actually do what you want without handholding, and randomness in particular is one of the places it fails badly. This is probably fundamental.
That said, this is also not helped by the fact that all of the default interfaces lack many essential features, so you have to build the interface yourself. Neither "clear the context on every attempt" nor "reuse the context repeatedly" will give good results, but having one context producing just one-line summaries, then fresh contexts expanding each one will do slightly less badly.
(If you actually want the LLM to do something useful, there are many more things that need to be added beyond this)
Sounds to me like you might want to reduce the Top P - that will prevent the really unlikely next tokens from ever being selected, while still providing nice randomness in the remaining next tokens so you continue to get diverse stories.
They linked to an interactive explorer that nicely shows the diversity of the dataset, and the HF repo links to the GitHub repo that has the code that generated the stories: https://github.com/lennart-finke/simple_stories_generate
So, it seems there are ways to get varied stories.
> Do you find that it actually generates varied and diverse stories? Or does it just fall into the same 3 grooves?
> Last week I tried to get an LLM (one of the recent Llama models running through Groq, it was 70B I believe) to produce randomly generated prompts in a variety of styles and it kept producing cyberpunk scifi stuff.
It's a 3b model so the creativity is pretty limited. What helped for me was prompting for specific stories in specific styles. I have a python script that randomizes the prompt and the writing style, including asking for specific author styles.
I'm going to guess that it's Hacki. It's the one in the OP link, and it's been more recently updated. Plus it's cross-platform with iOS and Android. I've used the Android version previously (I'm now using Harmonic), but couldn't tell you which is the better iOS app.
There’s a lot of incentive to figure this out. And they have so much data coming in that they can likely afford to toss out some good data to ensure that they’re tossing out all of the bad.
reply