> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed.
That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily seen with text-to-image models. For example the teddy bear swimming in an Olympic pool [1] or many other surreal examples [2] were not part of the training set. So as long as an answer can be expressed as a combination of primitive concepts, LLMs can generate them.
How much of the "novel ways which have never been considered before" is just the novelty effect of having your very own artist? A human being could certainly produce any of the works of Dall-E 2, given the same prompt. The change here is the cost, and not the capability. Of course, this is still significant, but it doesn't suggest to me that Dall-E 2 "thinks" differently or would be able to seriously alter the nature of our cognition, except to the extent that it allows us to realize the same ideas faster.
> How much of the "novel ways which have never been considered before" is just the novelty effect of having your very own artist?
What's the difference and why does that matter? Sure, plenty of people may have had the idea of an anthropomorphized teddy bear swimming in a pool, but if none of them ever realized it in the real world, the AI couldn't have learned from it.
> The change here is the cost, and not the capability.
I strongly disagree. Scaling the artistic process with GPUs instead of meatbags is a huge change in capability, just like the mechanized tractor was a huge capability change over the ox. The change in cost is a side effect in the change of capability.
You can now use a tool instead of outsourcing it to someone else and artists now have an automated tool that ostensibly replaces their manual labor.
I agree that it's a big change in the economics of getting art (e.g. a single game dev can now plausibly get assets this way), but I'm not sure that it's a change in the "cognitive" process of creation. This is in response to the original comment that suggests that novel combinations are interesting.
The machine is clearly good at realizing novel combinations, but I think that has more to do with the lack of interest of human artists in rendering these combinations, rather than the lack of ability.
I am also of the opinion that a human realization would produce better art. A machine might literally depict the bear in a pool, but a human could imagine a logically consistent context for that to be happening and decorate the pool with details like the leaderboard of the Teddy Bear Olympics and have reporters and spectators that are other stuffed animals. There might be a rivalry in progress. The distinguishing feature for me so far has been that human art is a snapshot of a much more sophisticated simulation that draws from the experience of having lived, felt things like fear, tension, joy directly, rather than having to approximate the aspects that give new art its electric nature indirectly as the machines do.
I'm sure a machine will be able to do that someday, but most of my experience with Dall-E 2 has been for the background to be vague, blurry, and weirdly unintentionally surreal. The prompt itself is maybe rendered accurately 95% of the time.
The OP never said that novel combinations are "interesting"... They claimed:
> Generative models can combine existing concepts in novel ways which have never been considered before.... So as long as an answer can be expressed as a combination of primitive concepts, LLMs can generate them.
You seem to be hung up on the "never been considered before" as if artists are incapable of combining those concepts when the more charitable interpretation of what the OP meant is "doesn't appear in the training data." Obviously they aren't omniscient and can't possibly know what people have ever imagined in the totality of existence, but we can empirically compare the novelty of generated images to the training inputs.
No one said anything negative about artists or their abilities. We're talking about the AI's abilities in a positive sense and that's the mechanistic ability to combine concepts in novel ways to generate images. Any art that springs from that is from the human using the AI as a tool.
A vast, intricate design with microscopic details at hundreds of levels of emergent forms, coloured in semi-transparency generating holographic effects, using spectra of light far beyond the human visual range, executed in three dimensions and occupying hundreds of cubic miles. Oh, and why not animate it? It would have to be virtual I suppose, or vastly too expensive to ever be created physically. In fact, why design it to be physically possible? Go nuts, make it 7 dimensional.
That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic.
Compared to text-to-image, which is seemingly impossible to use without getting something weird
There was a post here on hacker news not that long ago where GPT4 came up with "the confetti has left the cannon" when asked for an original phrase similar to "the cat is out of the bag".
Other users confirmed Google could not identify any other use of that phrase.
People saying GPT4 is unoriginal have an uphill battle, it is not the default assumption of anyone who has worked with it.
Confetti cannon is a thing. So there is an obvious link between confetti and cannon. Furthermore, These ("cannon" and "conffetis) are two very common concepts used in a single sentence.
Excuse me for being genuine but I totally fail to understand how original the phrase is.
Why don't you come up with an example of a new phrase with the same meaning that is actually original so we can compare your "real" originality to this "fake" originality?
ChatGpt is as original as us but cannot ever surpass us because it needs our feedback loop. It is a mirroring of sorts after all.
If it could become extremely creative, push the limits and make ideas so advanced that we can’t understand then it failed at the task so to speak. It always remix idea into larger idea sallads and get our upvotes downvotes. It is a great tool but a tool nonetheless.
I have absolutely seen an LLM combine language in novel ways. With prompting they will create outputs that are entirely novel with concepts intermingled in new ways. It's nonsense when compared to observed reality, but it is novel nonetheless.
What is fair about that when judging whether or not they can be creative? Most people whom other people would judge to be creative are nowhere near the best of us.
I would posit the opposite, language exists to be combined in any number of ways easily and still communicate well. A language that cannot do this does not I think exist.
Same. I find it hard to makes sense of why anyone would claim that computers can innovate visually but not with language. It would make more sense if someone at least felt that they both could, or both couldn't.
I remember reading what Lenat did with early versions of Eurisco: he ran it on some axioms from number theory and observed discovery of some number theory theorems, some of them pretty fundamental. As "time" progressed, the "theorems" Eurisco kept getting discovered became duller and duller, like expansion of square of sum into sum of products (I can't remember exact example from the book, but level of dulliness is about right).
I believe this can be served as example of combination of existing concepts in novel ways, applied to math, which is very formal and, on the other hand, is the basis of many other areas of human knowledge.
If this can serve as a such example, then you are wrong.
Also, what you have shown as your examples are not very much interesting outside of entertainment.
This is a good point. No human could possibly have envisioned a teddy bear swimming in an Olympic pool, as we did not previously have the words or conceptual framework to describe that. That image could never have existed before the invention of transformer technology.
LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself.
They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in an LLM only trained to predict protein structure sequences. No folding, nothing explicit in the corpus itself. Yet those truths manifest as a necessity of accurate predictions. Prediction is not bound by what the data explicitly shows.
>A language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about.
You mentioned the corpus prediction being the core of the LLM. Because of this, my prediction is we will see way more data withholding to prevent LLM learning just like we’ve seen with stack overflow, Reddit, X.
I myself have started doing this. For example, I don’t publish code on GitHub anymore to prevent copilot training on my own code. Normally I like to get paid for work, instead of paying for GitHub and doing work for them:) I’d much rather take a page out of an artists playbook and require royalties.
I have also built a coding test harness to test for AGI and let me tell you, that the LLMs are completely unable to reason on the most intermediate and advanced algorithmic problems you present to them the moment they are not in the corpus. This involves a nearly complete lack of the ability to generalize and infer new concepts needed to solve the problem even if the concept is explained in the problem description.
Now the issue is that it is impossible to present the test because once it ends up in the corpus, then someone will put the solution in the corpus as well and claim the ability to reason. You can believe me or not; I won’t be making my test available either way, but it is not too hard to make one of your own to convince yourself.
I wish it was possible to make the tests like that available but alas we will likely end up with a medical-style double blind kind of tests to get rid of AI FUD.
You are not fundamentally wrong, but there are degrees. LLMs can do some symbolic manipulation and a few reasoning steps. Maybe they are just the result of pattern matching, but I have a hunch that most of the time humans do the same. We fake our understanding as much as we can, we use all shortcuts.
Don’t kid yourself, posting your code on GitHub won’t lead to your being “discovered”, either.
Online portfolios are the worst lie we tell young developers. It’s the software industry’s version of “exposure” gigs: they get a few more bits they can shove into their models, you get jack shit.
Every, and I mean, every job I got I got because I either knew a guy or I knew a guy that knew a guy. You are far better off going to conferences and networking.
I have a 30 year career in this business, and I know full well what my Github and other online exposure has done for my career, and my experience does not mirror yours. It has mattered. It's not "made" my career, but it's smoothed paths and gotten me respect from people I'd otherwise not have gotten anything out of. I do accept it won't matter for most people.
But that was not my point. My assumption to start with is that most of us do not put much on Github that more than a few other people will care about. But most individual accounts are also of minimal value to Github or other people training LLMs. So however small recognition you get from having code accessible publicly (for me it's the occasional recognition when doing job interviews and a few e-mails from people now and again), you lose that to take away far less value to people training LLMs.
For the vast majority of us, it takes a vastly inflated ego to think withholding our code from a public repository will even be noticed by more than a handful of people. There are exceptions, to be sure, but they are vanishingly small proportion of us.
The problem is that new knowledge comes not only from combination but discovery, and discovery fundamentally requires access to and interaction with the actual world. And as of yet we have not created a way that computers can directly access the world, all I/O is mediated, etc. The more pedantic argument would be that what an LLM can write about the world is fundamentally limited to what has thus far been captured in human writing, and any structural properties thereof.
To your point, prediction may not be bound by what the data explicitly shows, but it is necessarily bound by the data and its implicit and explicit structure. But to argue that the LLM can make any discovery a person can, or interact fluidly with the world as it exists, would require one to believe that every property of the world is either explicitly captured in human-written text or in the structure of that text, which I think would be pretty difficult to argue.
People are hooking up LLM's with "access to the world" all over the place. Some of that will increasingly result in output that gets fed into training set.
I'd be shocked if there aren't people running fairly large-scale research on letting LLMs using various patterns like ReAct to augment synthetic training data as we discuss this, because that'd be near the top of my list of things to do if I had the resources to turn around and train a large-scale model on those outputs, because testing ways of accelerating the training by letting the models build augmented training datasets themselves is such an obvious step.
What does it take to "hook up" an LLM to a camera and get it to output something cogent? We have a set of human-selected discrete photographs and a set of human-written text labels for those photographs, then a model that is trained to assign those text labels to other photographs. Then a new photo is put into the model, a set of labels is emitted, and those labels are processed by the LLM as text (or more accurately as token integers). The same principle holds for other sensors and their data.
All of these inputs are either human-generated or based directly on the human-generated input. So the argument is merely extended to say that "all human-generated text and photographs are sufficient to fully describe the world" which again I think would be hard to argue for. The point of the article is that an LLM is simply a mechanical transformer from its input to its output, and the specifics of that transformation are fully determined by its training data, which is all human-generated.
> What does it take to "hook up" an LLM to a camera and get it to output something cogent?
The same as it takes to teach a human to recognise objects and output something cogent, except for machine learning models the effort does not need to be replicated from scratch each time.
But you're entirely missing the point of the comment you replied to, which is that giving them feedback loops and ability to carry out actions, and open ended problems to do things like web searches allows for discovery in the same way that a lot of human discovery happens, by posing questions, engaging in discovery and feed their findings back into further training. The only thing stopping us from doing that at scale right now is that training costs are still high enough that people are being conservative and taking the well known paths of feeding in cleaned and well labelled data. But we've already seen a lot of efforts in using LLMs to generate new training data for LLMs, and letting them drive that process without human intervention is an obvious next step (which will run into a huge number of failures, but the successes will keep accelerating future advances).
> The point of the article is that an LLM is simply a mechanical transformer from its input to its output, and the specifics of that transformation are fully determined by its training data, which is all human-generated.
For humans to be anything more than a "mechanical transformer from its input to its output" (assuming you're using "transformer" in the general sense here rather than in the sense of the specific LLM architecture) presumes not just a non-deterministic, non-materialistic world, but one that fundamentally breaks our understanding of both logic and physics. There's no evidence to suggest we are anything more than well trained, well-structured automatons. And on that basis, while there's every reason to acknowledge that there are additional big leaps to be made, this constant stream of assumptions that there's some inherent barrier between us and LLMs rather than just a set of refinements is pretty much a religious belief and nothing more.
An LLM can't generate new training data for itself. Mathematically speaking, the output of the LLM is governed by the same statistics embedded in its parameters, and a large enough training set of LLM output should have zero impact on those parameters. The difference is that training on human output contains the implicit assumption that the human output is "correct" in some way. So if the LLM gets more human-generated training data, it can come closer to "correctness". An LLM training on its own output is merely reinforcing its weights.
Maybe to put it more simply, there is no LLM in the world that can determine for itself whether a proposition or piece of training data is sound without a person saying so at some point, either implicitly or explicitly. Even efforts to give LLMs some non-human-mediated reward function are merely people attempting to encode our own principles. While people are no doubt physical, we have a biological imperative to survival and well-being that governs our sense-making faculties, and to which nobody has yet proposed an analogue for LLMs. Our "transformations" are not only governed by other people's "training data" but by our urge to be able to skillfully cope with a world in which we are biologically invested.
"Directly access"? Computers can use cameras and robotics hardware and such. It is not as if humans are special because of using eyes and muscles instead.
Also, text does say a lot about the world. It includes descriptions of physics experiments and people's best knowledge of the laws, for instance.
I think discovery for llm is possible both through its knowledge of the world from training data and also from some fields which are theoretical enough that interaction is not needed for discover--e.g pure mathematics but also many applied science
However, any such information must be already contained in the semantic field that is provided to the LLM. Either explicitly, or as "nascent" relations in the data that may align with the structure of concepts in that data, since what we may regard as concepts in the LLM is either a matter of embedding or a matter of gradient descent (and related combinatory productions) – and thus again limited by the structure of the semantic field.
> *Glimberfleck*: The sparkling reflection of sunlight that dances on water, making it seem like the surface is sprinkled with tiny, shimmering gems.
> Feel free to use it or let me know if you'd like me to create another word!
> When I’m looking out of the window of an airplane at the ocean, am I looking at a glimberfleck?
> Yes, if you're looking out of the airplane window at the ocean and observing the sparkling reflections of sunlight on the water's surface, you could describe that beautiful phenomenon as a "glimberfleck." It's a whimsical term for those captivating, shimmering patterns you see.
It’s not a mapping of words. It’s a mapping of character sequences. You can ask chatgpt to define “hackernewsitis (zero hits on google) and it gives a plausible definition.
Also the fact that it can write code is evidence that it can understand new concepts. A variable declaration is a coining of a (very short lived) new word.
Hackernewsitis is a made-up term that refers to the tendency of some people on the online forum Hacker News to get overly focused on or obsessed with certain topics, technologies, or companies.
Some characteristics of hackernewsitis include:
- Endlessly debating or commenting on the same subjects over and over, even when no new information is being contributed.
- Getting emotionally invested in certain technologies or companies, treating them like sports teams to cheer for rather than objectively evaluating them.
- Assuming the conversations and interests of Hacker News represent the tech industry as a whole, losing perspective on what is niche vs. mainstream.
- Spending too much time on Hacker News reading and commenting rather than working on your own projects and goals.
- Seeing the world overly in terms of tech industry buzzwords, losing appreciation of other domains.
- Acting overly cynical or pessimistic about new technologies based on theoretical risks rather than evidence.
So in summary, "hackernewsitis" refers to the potential for discussions on Hacker News to become unproductive echochambers if participants are not self-aware. It highlights the need for individuals to think critically and not get swept up in groupthink dynamics.
Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
Embeddings are usually common n-grams of bytes, not whole words unless it's a very common word like "it". It's also not all English.
> Hey ChatGPT, can you give me a brief explanation of AI embeddings, but respond in Chinese?
> 当然可以。
> AI 嵌入（embeddings）是一种将词汇、句子或其他类型的信息转化为高维度的向量的技术。这些向量捕捉到了原始信息的某些语义或关系特性。通过嵌入，我们可以更容易地在高维空间中比较和处理这些信息，如在机器学习模型中。
> 希望这有助于你了解！
As for saying it can only mimic words it knew in it's training data, this one gives zero results in google
> Hey ChatGPT can you make up a word for when I'm hungry but there's nothing in my kitchen I want to eat, so I'm not that hungry.
> Sure, how about "snackstipated"? It's a blend of "snack" and "constipated", suggesting a feeling of hunger but being stuck with unsatisfying options.
It is possible to draw new conclusions from complex data. It’s not something that a human might not also see, just that it’s an undiscovered relationship. (Or at least not directly expressed except by inference in the training corpus)
Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
Me: give me a random character sequence that is not a pronounceable word
ChatGPT: Sure, here's a random character sequence that is not a pronounceable word:
"Jkplqxzv"
The article is claiming that advanced AIs cannot become more intelligent than humans, essentially because LLMs cannot become more intelligent than humans.
LLMs aren’t the be all and end all of AI though. They’re an impressive but inherently limited stepping stone, with a very constrained scope of applicability and capabilities. There is no reason to suppose that future, much more advanced architectures can’t surpass humans in reasoning ability.
Advanced AI doesn’t have to be omniscient fir us not yon be able to predict it’s capabilities, it just needs to be smarter than us. By definition if something is smarter than you, it’s not possible for you to anticipate and predict its behaviour. Therefore in principle it is not possible to predict whether that behaviour will be to your liking. That’s the problem.
I think the paper would similarly claim something like "a chess ai trained only on human games can't play better than the humans who played those games"
I think that's false - you could imagine it playing at GM level without making mistakes that real GMs make
But it's missing an important fact: chess ais aren't limited to learning only from human games. They can learn from self play.
You can extend this to physics (if we can simulate physics precisely enough, there's no reason ai can't make fundamental discoveries in physics)
More specifically, the article assumes that the only way "AI"s will ever be produced is the way that LLMs are -- by being trained off pre-existing human generated media, and existing as static objects after that. In particular, this assumes that the system doesn't subsequently modify itself based on further interactions with the real world (e.g., doing experiments, and taking note of the results).
> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe.
This is often said, but it isn't so.
The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens.
There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning.
I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task.
Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes.
This is a great example when it comes to trying to understand the epistemological limits of AI. People inherently fall back on an argument that presumes the human mind works like a neural network.
There's an interesting theory that I've never been able to identify the origins of, that humans like to think that we created technology from needs based on our models of the world, whereas we ignore the effects of daily technology on our thinking frameworks. The argument essentially states that we never "discovered" the circulatory system of the heart or "discovered" it works like a pump and valves, rather instead right around the same time this theoretical work was being investigated on the heart is when the industrial revolution was in full swing. Thus, we modelled the heart as pumps and valves because that's the technology we were surrounded by. The heart isn't somehow inherently a "pump" and we "discovered" that, we just started using the pump metaphor because it seemed to help do other things. I'm sure though that the metaphor has it limits.
Typically, the narrative around the invention of machine learning models is that we started coding computers to be more like recently "discovered" models of the brain.
Under this theory its the opposite. Right as cognitive sciences started developing as a novel field of research is when we developed computers. So, in classic form we decomposed the brain in atomic fashion, and the 'atoms' of measurement we chose to use ended up being bits and bytes.
Distinguishing between inventing "novel" things and gobbledygook is completely subjective and based on the viewer's own models. It's proving these abilities after the fact, not before it. Thousand monkeys on typewriters etc.
This measurement of "accuracy" is completely forgetting everything that Kuhn discovered about scientific knowledge. If you've got a community of like 3 people who research some incredibly esoteric scientific field, only those 3 people could ever accurately judge the full extent of their domain. A model could generate a series of tokens that for the rest of the world is gobbledygook, but to these 3 scientists it makes perfect sense. This doesn't really endeavour me to believe that there's anything "novel" about what AI "predicts". It just throws out enough combinations and we conveniently ignore the huge gaps when its wrong, but then jump up and down excitedly when it's "right" (as if its discovered some universal material objective truth).
>OpenAI, the company behind ChatGPT, recently announced a massive investment into research on “superalignment,”
The head of the “superalignment” project at OpenAI recently appeared on the AXRP podcast talking about the Automated Alignment Researcher they’re developing.
The host asked him how they were going to align the AAR itself, and if they can do that, what would be left for the AAR to do, but he didn’t seem to understand the question.
People are still desperate to maintain humankind at the top of the intellectual pyramid. It's tiresome and a waste of energy. There are definitely limits to intelligence, nothing will ever be smart enough to predict the future perfectly. But to believe that humans represent the peak of what's possible, is just hubris and nothing else.
he lost me at "a language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about."
This is trivially demonstrated to be a false statement, as GPT is capable of synthesizing entirely novel words based on very little input guidance.
I find it exhausting when articles are written about ChatGPT which can be trivially demonstrated false by simply testing their assumptions. Noam Chomsky pulled a similar one a while ago.
Whats up with the number of mind-numbingly stupid takes in this thread? Are you actually this dumb or is there some elaborate in-joke i missed? Just because LLMs can (poorly) come up with new words when you instruct them to doesn't mean they're actually "thinking" in novel concepts, its just another indirectly grounded label for existing concepts.
https://dai.fmph.uniba.sk/~retova/CSCTR/materials/CSCTR_07se...
This is plainly false. With reinforcement learning, an objective function and the ability to generate random solutions it's possible for AI to stumble upon sequences/strategies that greatly outperform human state of the art.
I found this article a bit fluffy and lacking on technical substance. If we want to entertain the limits of AGI, we need to look at concepts like Solomonoff induction, which were conceived of and researched in the 1960s (and apparently mostly forgotten about since then). Given an input sequence of bits, what is the best (i.e., “oracle”) posterior distribution possible given a domain prior and the universal prior? How would a resource-limited approximation widen this posterior distribution? What are the practical limits to achieving the shortest Turing machines that reproduce the input sequence? These are interesting questions that should be looked into if we want to discuss AGI in the epistemological limit.
Also, I tend to get frustrated with predictions by “AI experts” about what a future with AGI will look like. We don’t have any world experts on AGI currently, for the simple reason that it doesn’t exist yet. We have experts (or I’d prefer to say “first movers”) on particular types of machine learning based upon neural networks, but that’s about it. A true AGI would almost be unpredictable by definition, so human attempts to forecast the future in this regard are mostly sensationalism or a projection of some kind of personal agenda (cough cough Marc Andreessen).
Rather than viewing generative AI as a form of artificial intelligence, I posit that it should be seen as an automated tool for tapping into human cultural, linguistic, and empirical knowledge.
Data and computation are two sides of the same coin. The 'intelligence' in AI is embedded within the data, with the computational model serving as a tool to access and express this inherent intelligence.
I would argue for a change in perspective towards AI, one that recognizes LLMs as powerful tools for accessing the vast wealth of human cultural knowledge rather than viewing them as a separate form of intelligence. We must carefully consider critical ethical considerations about control, access, and trust that will become increasingly relevant as these tools become more integrated into our everyday lives.
This paradigm shift carries some implications:
LLMs will not achieve superintelligence: Although these models can process information quickly and access a wide range of knowledge, they lack the superior reasoning or inference abilities that would classify them as superintelligence.
LLMs as an extension of human thought: These models can automate and amplify human capabilities but do not introduce new abilities beyond what is already present in human thought processes.
LLMs as mirrors of human culture and knowledge: These models reflect the recorded artifacts of human language, art, and culture. They can make the inherent intelligence within these artifacts accessible, providing a vast information resource.
Implications for the future: Access to this "memetic matrix" of human knowledge will become a fundamental part of being human as these tools become more integrated into our lives, bringing up issues of ownership, access, and the potential for misuse.
Thought consolidation and control of inference engines: There's a potential risk that control of inference engines by a small number of companies could lead to a consolidation of thought that threatens democratic governance. I propose a diversity of federated or self-hosted inference tools as solutions to mitigate this risk.
The necessity for trust and individuality: As these tools become more influential in our lives, maintaining trust in our individual thoughts and avoiding the uncritical acceptance of synthesized ideas from sources with opaque motives will become increasingly important.
Synthetic Inference relies on a vast cultural commons: We cannot allow these commons to be closed off and owned by a few big companies. This resource is to approximate totality of all human knowledge, language, and culture. It belongs to all of humanity. Training data must be open, free, and available for examination.
Because, ultimately, the models are modelling human reasoning.
Not only that we do not have any evidence to support the idea that reasoning has a higher quality than we can apply to it using logical processes. It can be done faster, it can be done many times concurrently but ultimately 1 || 1 !& 1 is going to be have the same answer.
The whole idea of superintelligence except as a measure of speed or quantity seems flawed on its face. And if we are to call speed or concurrency superintelligence, then we have already hit that mark some time ago.
AI wins at games because of time compression, not because of an inherently superior logic. They just have time to consider more potential outcomes.
AI research went off the idea of modelling human reasoning a long time ago. To the extent that current models reproduce reasoning in any sense comparable to a human it is purely for the purpose of user interface. For example if you ask a transformer model like chatgpt to solve some problem and explain its reasoning step by step it will give you a reasonable facsimile of a human thought process, but if you’ve read the transformers paper, you know that its actual process is tokenize -> do a bunch of matrix math -> decode the result into words (entirely different to how a human reasoning process works).
Although things like neural nets were clearly inspired by biology they work completely differently to a biological brain. It would be closer to truth to say that the models use linear algebra and optimization to improve performance at specific tasks. For that reason, the whole debate about whether or not superintelligence is possible/has been achieved etc boils down to an argument about the definition of intelligence (as Turing predicted so long ago).
I don’t think it’s a foregone conclusion that human reasoning doesn’t operate on the basis of statistical prediction of the next most probable “token” at its most granular level. Humans are certainly capable of hallucination in the LLM sense, and without training we often struggle to produce (or even outright fabricate) the rationale behind our “conclusions”.
We stopped intentionally modelling human reasoning because we have no clue how it works.
We roughly copied the physical devices then figured out how to slap them together with an algorithm that makes them behave similarly to human thought, when trained with a massive quantity of cultural-linguistic-memetic data.
It should not be surprising that if you take a bucketful of engine parts and keep messing with them until they kinda work as an engine that you will probably wind up with something along similar lines as the original intention of the parts. Frankly, it would be quite surprising if you didn’t.
Biological neural networks are fundamentally linear algebra processing device that integrate data into functions through training, so the fact that we understand the process as linear algebra is actually an argument that the process is similar.
As to whether “superintelligence” can exist, I think we are in exactly the same page there. It is a matter of definition. Personally I doubt the existence of a fundamentally superior system of logic type of superintelligence, but certainly a device can made to iterate faster and flawless memory and instant, accurate calculation is going to beat the hell out of my notebook and HP28s lol.
I suspect that superintelligence can be (has been?) achieved in some respects, but in the same sense that a room full of ten educated people with regular computing tools is “superintelligence”.
> Biological neural networks are fundamentally linear algebra processing device that integrate data into functions through training, so the fact that we understand the process as linear algebra is actually an argument that the process is similar.
Do you have a reference for this? My understanding of biological neural networks is very different from this. Specifically I find it really hard to believe that a biological brain is a linear algebra processing device.
Matrix multiplication with a different function and weights for each input/output. That’s the primary operation.
There are many other operational modes overlaid that we typically do not model with mainstream models, but they primarily act as broad modifiers (suppression, gain, etc) we understand very well how it works, for example, when we get a few brain cells together to play pong or doom. We can transcribe the functionality of these simple biological networks, once trained, onto models and they function as expected.
The simplified computational model we use for neural networks closely models the primary mechanism, but lacks self reinforcement systems, biochemical moderators, self directed connectivity modifications, etc. Basically we do the training with a separate process for most of the systems in common use, in contrast with biological systems which self organise and train in-situ.
I’m not claiming that biological systems “do” linear algebra, I’m claiming that their behaviour can be partially modelled using linear algebra. If I gave the impression that I thought they were actually computing linear algebra I apologise for my unclear writing.
But AFAIK modeling neural network nodes with linear algebra is a foundational principle and has been for quite some time, since I first started messing with neural networks 30 years ago.
But hey, maybe you’re right and computational neural networks aren’t roughly modelled after their biological counterparts and I’ve just misunderstood the field all this time. I guess the good thing about neural networks is you don’t really have to understand them completely to use them lol.
That said, if you’ve got information that refutes the notion that they are roughly modelled after their biological counterparts, or that somehow linear algebra is not the right tool for modelling them, I’d sure be interested to see it.
Just because I’ve ben doing something for decades doesn’t mean I haven’t been doing it -wrong- for decades lol. Wouldn’t be the first thing.
LLMs have obvious limitations and a big contributor is the training data and the way they are trained in a supervised way. Although that is not as much of a limitation as with most other AIs because of the "in-context learning" ability.
But that doesn't mean that future LLMs couldn't conceivably learn abstractions than are at a higher level or even perhaps inaccessible to humans in number of layers or complexity.
I think the obvious type of superintelligence that is only a few years away is what I call hyperspeed AI. LLMs are a very specific application which we will be able to accelerate greatly. Within a decade the output speed will be dozens of times faster than human thought.
Because they will have robust reasoning and extreme "thinking speed", these AIs will probably be connected to industry and military applications. Humans will need to be removed from the loop because waiting for a human decision means the competitors' AIs race ahead the equivalent of days or weeks.
If this is fine then it will create a precarious situation for humans where they are only nominally in control and things like advanced AI "viruses" might become very dangerous.
It's also certain that researchers will continue to develop new types of AIs that incorporate more animal (such as human)-like capabilities.
> Future LLMs, regardless of model architecture, are fundamentally constrained in this way, by virtue of the fact that they are trained on human-written texts. Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more.
That constraint holds only for pure LLMs. There is a secondary source of learning - external feedback to the model. When a LLM is part of a larger system, it can integrate feedback to improve itself, like AlphaGo Zero. A LLM+compiler could self-optimize by observing compilation errors and program outputs. A LLM+game could self-improve by conditioning actions on the score. Humans could create a preference dataset and model for RLHF to refine the LLM based on human feedback. A LLM+robot could learn from past experiences to improve its planning.
Similarly, humans need tools to conduct research. We can't do pure research without labs and experiments. LLMs need external confirmation the same way we do. The outputs of a pure LLM merely reflect our current understanding of the universe. But coupled with real-world feedback, LLMs have the potential to learn and discover new knowledge.
Putting aside the question of what LLMs, specifically, are capable of, I feel the author's central argument has already been shown to be false, through events outside of AI.
In a nutshell, the argument is that, as AI will have just the knowledge that is available to us, expressed in the languages we use, we will be able to anticipate and put a stop to it doing anything catastrophic.
Exactly the same argument, however, could be made over computer and communications security. If anything, those trying to keep things secure have the advantage in knowledge, at least initially, yet zero-day exploits keep surfacing, some of them at fundamental levels such as processor architecture. Nothing truly catastrophic has happened yet, but it cannot be said to be impossible.
The flaw in the argument is that one can easily know all the bare facts without seeing all the implications but someone - or something - else could do so.
"Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more." Others have reflected on the problems with this statement, and I agree, LLMs can be prompted to navigate the training corpus and create novel output, countering this claim. However, these criticisms neglect to credit the external intelligence inherent to the prompts themselves. AutoGPT and its ilk have yet to create convincing agency: for the time being the current state of AI is a mere extension of human intelligence and will not be making lofty discoveries without direct human interaction. LLMs are a form of human co-intelligence.
Reading through there is a heavy fault in logic. Statically reasonable limitations to llm output based on training material, ignores hallucinations. And hallucinations ignore the sheer chance of new emergent information by odds.
Hallucinations are not a different process from normal output. They are merely called hallucinations rather than insights because they are wrong.
There is no magic in LLMs , nor in human thought. It is all a matter of symbol vectorisation and inferring relationships within the multidimensional memetic matrix.
AI absolutely can achieve original insight- but not one that humans could not also make if looking at the same data. This is because the “intelligence” in generative AI is human bounded in the training data, not in the engine that processes it. I strongly suspect the same is true of humans.
After all, we modelled neural networks after our own brains, why would we not expect them to achieve similar results using similar means? Wasn’t that the whole point?
It never ceases to amaze me how people go all pikachu face when I say that our thought process is probably a lot like generative models. Step one, make a facsimile of thing. Step 2: be surprised when thing can also be viewed as being similar to the facsimile. Lol. Hubris is our primary characteristic.
By odds and in the context of RLHF the human could very well thumbs up the output without recognition of said hallucination.
As hallucination is a general term giving to said phenomenon. Otherwise the question Emerges why 2023? And why was that not known colloquially before hand. When the basis of these Algorithms come the the 80s?
Because models got sophisticated enough that wrong outputs started looking plausible answers rather than random garbage. Sometimes they still spew random garbage though.
Why do we need anything close to superintelligence or "omniscience" (wtf) before we get profound changes to how humanity operates. It's really an obsession!
Instead compare to the average human, at most. Compare to refrigeration? Compare to fire? Compare to computers? And that's even if the bar is placed at the singularity. Human progress as we have known it so far is not founded on superhumans but on a lot of work by a mix of merely visionary, hard working, and just plain average people. Just speeding that an order of magnitude changes everything.
This focuses on the argument that some hyper-sophisticated future LLM will not discover the secrets of the universe on its own. That's plausible, but beside the point when it comes to the potential dangers of A.I. Not all future A.I.s will be LLMs. Understanding the universe isn't the only way to pose an existential threat.
This is an old discussion with much deeper insights already available than I can parrot here, but one obvious counter just to this argument is that even an LLM can be an existential threat, depending what it can convince you it knows.
I'm interested in seeing how brains change as AI usage increases, in the same way many of us don't attempt to memorize phone numbers, what skills will we offload to 'AI', will we be aware that it's happening? Perhaps more interesting, what new adaptations or adjustments may occur as a result of this augmentation. I hope that our brains ability to synthesize new thoughts both extrapolated and 'from thin air' will remain a useful skill regardless.
The problem with AI is that it doesn’t really free you to do anything else, because as soon as it does, that thing becomes the next open research problem.
For example, you could say that ChatGPT frees writers to focus on creativity. But someone somewhere is working on making LLMs more creative and less formulaic.
Recursive insight is possible with a model that self trains, but right now that would result in a detour into unreality. Perhaps with the right systems of vetting prior to incorporating new data into the retraining set.
Right now they just get stupider if you train them on their own output, which suggests that the quality of the data available in the training set is higher than the quality of output produced by the model as a general rule. The fidelity is < 1.0 . Apparently, it is possible to achieve fidelity >1 (the growth of human knowledge) but our algorithms are not so great at this point, it seems.
Not necessarily. For example Anthropic's ConstitutionalAI (CAI) leverages the model to substitute human judgments in RLHF, effectuating essentially RLAIF. CAI information is used to fine-tune the Claude model.
Broadly speaking, you require statistics at echelon N+1 when you are at rung N. We can amplify models by providing them additional time, self-reflexion, demand step by step planning, allow external tools, tune it on human preferences, or give it feedback from executing a code, or from a robot.
Yeah, it makes some sense that you could use a more intense introspection to train weaker ones… I wonder what the human analogue for that looks like.
Maybe working up a proof and then quizzing yourself on it?
As long as we get >N supervision and the difference is more than the model retrograde, it seems that could work. But it seems like there is a definite limit to that. The N-n1 difference will only stay above the improvement delta up to a point.
The model would learn from feedback, not just regurgitate the training set, as long as the model is part of a system that can generate this feedback. AlphaGo Zero had self play for feedback. Robots can check task execution success. Even chatting with us generates feedback to the model.
LLMs are trained on human-written texts, therefore their output can only reflect our current understanding - am I reading this right? Hopefully not as that would be insultingly wrong.
Anything with started with "the myth" arguing about something that is at the limits of knowledge without a clear answer is unavoidably biased by a person with big ego.
That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily seen with text-to-image models. For example the teddy bear swimming in an Olympic pool [1] or many other surreal examples [2] were not part of the training set. So as long as an answer can be expressed as a combination of primitive concepts, LLMs can generate them.
[1] https://imagen.research.google/main_gallery_images/teddy-bea...
[2] https://imagen.research.google/