"no major AI technology breakthroughs in decades.everything we are seeing is larger compute scaling." This is false. Everything from the transformer to advancements in state space models have been foundational breakthroughs
People who say things like "major breakthroughs" often imagine a cliff/steep rise. The reality is that most "breakthroughs" are small, incremental, almost invisible steps in all aspects of the field, and potentially even in fields that are only tangentially related.
Then, one day, they hear about it in the news, because there's now some hype, or some event that makes it newsworthy. This makes it feel like the breakthrough was instantaneous or steep, but in fact has been in the making for decades or even more.
I have no doubt AGI is coming, but it will be gradual and slow. It will be the accumulation of more advances in everything, including hardware, as well as software. It might even include economic changes.
From nothing to ChatGPT opened to the public and billions students' lifes rely on that in the next month, isn't this the biggest step in the tech history?
It's a major societal failure that education has been reduced to turning in coherent enough strings of characters, that's what I've learned in the past year.
You mean the use of a writing system to share knowledge amongst ourselves?
I find that absolutely wonderful and it worked decently well for me (and possibly you.) Now we have a never seen before technology and society will adapt, that's it. No failure.
> You mean the use of a writing system to share knowledge amongst ourselves?
No. And what's with the "it is so bad yet you used it"? I am very much allowed and required to denounce a system even if I cannot escape it or if I could have somehow profited from it or chosen to use it.
I very much reached the point where I am despite the educational systems I was exposed to. And a system geared towards memorization and regurgitation of data in textual format where pupils can successfully use a chatbot to avoid doing work is certainly failing its goals of educating the youth.
I would point you to the complaints of American teachers about the reading and mathematics levels of students, if only because that is widely accessible. I did not grow up in the US and the school system in my home country is leagues behind the US.
excluding... antibiotics, electricity, refrigeration, the combustion engine, the digital computer...
like even restricted to the domain of strictly computation, I'd say it barely scratches the surface... like even if we ignore computer engineering ("the transistor," "silicon microprocessors", etc.)... foundational tech like "compilers" are more significant.
even restricted to modern applications, GPS is more useful and life-changing.
so, no. It's not the biggest step in tech history.
Curious about how many kids in India and China are relying on it.
I'll accept the prospect of hundreds of millions, generously.
And "relying on it" is a strong phrase. Using it as a curiosity, sure. The ones relying on it seem to keep ending up in the news because of how, well, unreliable it is.
There were hundreds of competitive, even SOTA LLMs before ChatGPT existed. You're basically just proving the parent comment right in how small of a leap ChatGPT is from t5-flan or BERT.
I beg to differ. Transformers are purely an optimization. It’s not exactly right to call everything “compute scaling” but we are still, at the end of the day, fitting polynomials.
And frankly, that’s probably not what our brains are doing.
It is a bit weird though… I mean, you could just as well say that the last breakthrough in all of computing was the transistor (Shockley 1946) – all we’ve been doing ever since is just “scaling” and combining them in new ways.
AI has been a 'moving aim' as opposed to moving target. It's a label slapped on whatever slice of software engineering seeming most magic-like at any given decade.
Early computers in late 40s were called 'electronic brains' by the media...
It’s difficult to comprehend impact of something that’s widely adopted. Like, batch normalization alone was probably mind blowing when it came out. Yet it seems so simple and self-explanatory now.
> And frankly, that’s probably not what our brains are doing.
I think that it is! It's much more likely to me that our brains are doing something big and simple than small and complicated. That's the way that nature tends to work. Fitting low-order million-dimensional polynomials would meet that description.
Our brain is however the very definition of small and complicated. It's the most complex known organic object known to humans and for all that any given normal brain handles in a given day, it uses just 0.3 kilowatt hours (kWh) to do it. No computer we have comes close to handling what a brain handles with that power consumption.
ChatGPT by contrast, consumes roughly a gigawatt hour per day serving its users. Yes, to do this it's handling a colossal amount of queries, but that's all it's doing. Your brain handles everything in your body and consciousness, in ways we don't even fully understand, while also letting you think and communicate and reason as a conscious being with self direction.
Moreover, there is evidence that at least part of our brain's functions may be exactly as the other reply here mentions, weird, subatomic and deeply complex in ways that are difficult to get a clear grip on.
From the double slit experiment, to particle-wave duality, to the particle zoo of the 70s, to quantum chromodynamics, to asymptotic freedom, to more exotic theories like string theory, etc. tells us the complete opposite. Every major discovery in physics in the past 150 years seems to disagree. Things are extremely weird and complicated when we get extremely tiny. Why would our brains be different?
If we lived at the quantum scale, then classical physics would be the weird one. Quantum chromodynamics is only confusing for two reasons: it differs from our everyday experience so we don't have an intuition for it, and because it has a large number of mutually-interacting (but basic) components.
Richard Feynman put it very well:
"The world is strange, the whole universe is very strange, but see when you look at the details then you find out that the rules are very simple, of the game, the mechanical rules by which you can figure out exactly what's going to happen when the situation is simple. It's again this chess game; if you're in just the corner with only a few pieces involved, you can work out exactly what's going to happen. And you can always do that when there's only a few pieces. And so you know you understand it. And yet, in the real game there's so many pieces you can't figure out what's going to happen.
"There's such a lot in the world, there's so much distance between the fundamental rules and the final phenomena that it's almost unbelievable that the final variety of phenomena can come from such a steady operation of such simple rules... But it is not complicated, it's just a lot of it."
