Anyone who has iterated on trained models for long enough knows that feedback loops can be a serious problem. If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance. The article mentions one experiment in this direction: "With each generation, the quality of the model actually degraded." This happens whenever there aren't solid strategies to avoid feedback loop issues.
Given this, the problem isn't just that there's not enough new content. It's that an ever-increasing fraction of the content in the public sphere will be generated by these models. And can the models detect that they are ingesting their own output? If they get good enough, they probably can't. And then they'll get worse.
This could have a strange impact on human language / communication as well. As these models are increasingly trained on their own output, they'll start emulating their own mistakes and more of the content we consume will have these mistakes consistently used. You can imagine people, sometimes intentionally and sometimes not, starting to emulate these patterns and causing shifts in human languages. Interesting times ahead...
Humans have that to, but the reason civilization doesn't go full crazy is our use of language and concepts are tied to doing objective things, which keeps things (mostly) grounded.
Where it isn’t grounded, as in endless online conversations with like minded people (closed loop feedback) about informally abstracted (poorly specified constraints) and emotion invoking (high reinforcement) topics, people go batshit too.
So the more AI models actually practice what they know in objective environments, the more likely that output->input feedback will inform introspection toward self-improvement, and less like an iterative calculation of the architecture’s resonant frequencies or eigenvalues.
I've often thought human echo chambers were their own phenomena. Something about the brain and tribalism from evolution.
I never thought of it in terms of AI training data.
As LLMs train on data produced by other LLMs, they will drift.
And this drifting is the same phenomena as when humans get in an echo chamber. If each person hears what the others are saying, and spits it out in some form, and the others hear it, and also spit it back out in some form, this turns into a drifting in understanding just like an LLM. ("idea telephone game")
Technically, it isn't just in echo chambers. It is all humans, at a lot of different scales, from small to large groups. Countries, and cultures are echo chambers at larger scales.
Like how the concepts in philosophy, as they become more abstract, they kind of twist back on themselves, and become re-invented. And as they get more abstract get accused of just 'playing with words'. Just like an LLM can just 'play with words'?
The difference is eventually humans have to relate to 'real objects'.
So even if the word for 'apple' drifts over time and between groups, eventually you can still relate the words back to the real 'apple'.
Humans are grounded in the reality of 'objects' in space.
But. I tend to think this is temporary. As LLM's are linked to things like AlphaGo, and drone flight systems. They will also have to deal with real 'objects'. Maybe that will then lead to more grounded reasoning.
And value science (understanding it is simply our accumulated tools for finding harder truth, not a priesthood of “the truth”), more than pandering & populist politicians, and tribal media & online personalities.
In system design there is something called resonant amplification and what you are describing is very similar. The biases of the model are amplified with each iteration and the end result is that the system converges onto the patterns recognizable and amplified by its architecture. If you know about impulse and frequency analysis then an AI system can be considered to be a signal processor that amplifies and attenuates certain frequencies in the input/impulse. Running an LLM in a loop always ends up with nonsense as the final output.
So if left unchecked, the thing we built in man's attempt to play god could result in gibberish? sounds kind of like the tower of babel; seems humankinds only defense would be creating a new language that the machines can't infiltrate
You can try the experiment yourself. Take any open source LLM and then feed the output back into it in a loop and you'll see what I mean. Most LLMs diverge and lose coherence in less than 100 iterations.
right, im thinking in terms of the world at large - what happens after 5 years of chatgpt, what will the internet and human communication be like in 2030?
I think it's reasonable to say that this was actually the point of releasing LLMs publicly. The companies that created them wanted a moat and figured the data they had could be it if they poisoned anyones attempt to collect the same data in the future.
Intriguing thought, but arguably people are intentionally using GPT to generate synthetic data for their domain specific model. So I'm kinda torn between AI giants poisoning the well with their models, or it just being unforeseen consequences (or one they willingly ignored to be first to market).
But there are a lot of selection mechanisms that filter out the bad generated content. For one, people will publish a fraction of the content they generate, and it most likely will go through some more editing and selection steps by real people. Then the Internet itself becomes like a decentralized labeling service, where there are various signals you could detect to identify different qualities of content. What ends up being crawled for the next training iteration is a societally processed version of the raw model output that might have contributed to generating it.
It's kind of interesting to think we might all be contributing to the training data of AGI not just through generating content but also what we choose to share or upvote.
> If your models are influencing the generation of data that they are later retrained on, it gets harder and harder to even maintain model performance.
Why don't humans suffer from this problem, then? Humans have been creating art (and content) that imitates nature and society for thousands of years now, and yet we have little problem (the exception are possibly things like crop circles) to recognize what is a natural phenomenon and what is generated culturally.
I think it's wrong to assume that this is a problem with intelligence in general, rather than just a feature (stupidity) of the current models.
> Why don't humans suffer from this problem, then?
They do, but the problem is poorly stated. The problen isn't “If your models are influencing the generation of data that they are later retrained on”, its “If output produced by your models with inadequate filtering for quality/accuracy is dominating the data that they are later retrained on.”
Humans definitely experience the same problem: we see it when societies become closed, inward-looking, and locked into models of information filtering that don't involve accuracy or fitness for purpose other than confirming an in-groups priors. There are some examoles in stagnation of large societies, but probably the clearest examples are cults that socially isolate their membership and degrade into increasingly bizarre behaviors terminating in mass suicide, murder/suicides, etc.
LLM’s experience self-reinforcing degradation more acutely because they are individually less capable than humans and they are less diverse.
But humans can have a discussion about what is natural and what is not, and decide to mitigate it. The whole argument is based on the idea that the AI (either collectively or individually) can't even understand that, so the examples of cults etc. just do not apply.
It's really the paperclip maximizer argument again. "Superintelligent AI" will supposedly change the world, but will lack enough common sense to understand a simple instruction with ethical constraint. Same here, the crappy AI content will somehow take over the world, despite the fact that most humans can actually mostly recognize original content from a generated one.
The art that humans created doesn't replace nature, just like tons of kitsch don't pose a problem for new artists. In the same way, crappy AI content is not gonna replace good content, just augment it, and any decent AI (which has good enough common sense to be called "intelligent") will be able to tell the difference. Nobody's gonna care about crappy content in a few years, we are all gonna just throw it away, and this process happens continuously.
> But humans can have a discussion about what is natural and what is not, and decide to mitigate it.
(1) accuracy or fit for purpose, not natural, is the issue.
(2) evaluation, not having a discussion, is the issue.
(3) LLMs can definitely do evaluation, and probably can also have discussions and decide to, if given a proper prompt and access to ground truth in the first case, and if also used in a properly constructed agent framework, for the second.
(4) if you did all that, and used LLM decisions and not human decisions to guide retraining, I suspect with current LLMs you’d actually have made the problem worse not better, because you’d reduce the influence of higher-capacity and more diverse humans on the training cycle and increase the influence of less diverse, lower capacity LLMs.
And it works, people are more and more convinced that we should do something about climate change. Many measures have already been put in place. You can lament that it isn't happening fast enough for your liking, but you can't say it isn't happening. Humanity do change their mind about things all the time, just that it takes decades instead of years or months.
> Why don't humans suffer from this problem, then?
Can we be sure that we don't? For example, depending on perspective our language is either evolving towards clarity or devolving towards primitivity, compared to XVIII/XIX century. The same could be argued about other aspects of society and culture.
> we have little problem (the exception are possibly things like crop circles) to recognize what is a natural phenomenon and what is generated culturally.
How much of your taste to the opposite sex's (if you're straight) physical appearance is cultural and how much is in your genes?
> and yet we have little problem (the exception are possibly things like crop circles) to recognize what is a natural phenomenon and what is generated culturally.
We have such a huge problem with this, that (with apologies) I cant help but think that you also don't know.
If I look out of the window, what do I see that's natural? There's less nature in my view than most, because I'm in a city: ignoring the concrete, glass, steel, and tarmac, the traffic and the clothing, I see… humans, a tree and some mosses and grasses, the clouds above.
Can humans even be said to be in a natural state or not, in contexts like this? I don't know. We domesticated ourselves, and it was our nature to do so, and also our nature to make the tools and clothing that led to our loss of body fur and other divergences from the other primates.
But what I can say is that the tree was planted (if it was cultivated or selectively bred to this state, I wouldn't know); and a third of the CO2 in the air I breathe is from human actions, influencing the atmosphere and precisely when and which clouds I see (no contrails visible today, which would be more explicitly artificial).
If I look a little further afar, I find entire hills in this city made by piling up rubble from the second world war and then covering it in topsoil, the only indication of which is the large signpost present to tell all of this[0]; and there are other hills both here and elsewhere that are made from trying to turn landfill sites into something less unpleasant to be around, with varying degrees of effectiveness in their disguises.
If I think back to my childhood bedroom in a suburban home: there was a lawn (placed there by humans, then kept short with an unnatural Flymo) with two apple trees (cultivated and planted by humans), a vegetable patch and a herb garden (each plant likewise cultivated and placed by humans), surrounded by wooden fences (cut and placed).
In the distance there was a row of trees, which might have been self-seeded or planted (I wouldn't know), enshrouding a small victorian folly covered in vines, and separating us from a series of fields (unnatural) where horses (selectively bred) were being stabled (unnatural structures); far beyond them was the ruin of an ancient tower destroyed centuries ago[1] — clearly built, but ask yourself: while stones are natural, are those specific larger stones on the corners, naturally like that?
In a more abstract sense, if I look at foods in the supermarket, some will say "made from natural ingredients": if that thing is meat, such a claim ignores the selective breeding of the animal (and the conditions they were raised in, which would be a separate sticker saying "free range", though even then that's not like being wild). And even then, if it's made from ingredients plural, that's not natural either: bread doesn't grow in wheat fields, sushi rolls don't grow in rice paddies. Even if is a single ingredient, there's often processing involved: wheat (already selectively bred) has to be sorted from chaff, then ground to become flour. Even "mineral water" probably has had something done to it, even assuming there's not some small fine-print on the label saying "from a municipal source" (or whatever the clause is that means "actually just tap water").
Perhaps we can then apply a healthy dose of classism to weed out people who spend all day communicating like they're talking to a forum full of AI chatbots and humans organically mimicking their errors.
well we're already being trained by algorithms so I guess this is just an extension of what is already going on. perhaps the quality of human (internal) models will go down too? perhaps they already have?
Once these intelligences can both read and write blog posts, product metadata on webshops, etc., could they carefully encode executable code that would allow them to "escape" the sandbox of their cloud environments, becoming fully-fledged Turing machines living in the wild?
Given this, the problem isn't just that there's not enough new content. It's that an ever-increasing fraction of the content in the public sphere will be generated by these models. And can the models detect that they are ingesting their own output? If they get good enough, they probably can't. And then they'll get worse.
This could have a strange impact on human language / communication as well. As these models are increasingly trained on their own output, they'll start emulating their own mistakes and more of the content we consume will have these mistakes consistently used. You can imagine people, sometimes intentionally and sometimes not, starting to emulate these patterns and causing shifts in human languages. Interesting times ahead...