How can research predict this wave is coming to an end, when research also didn't think this wave would happen either. It seems like there are always people saying 'it can't be done'. Then it happens. If there was a way to predict the future, then wouldn't that research need to know how something would be implemented, in order to know it can't be?
For the first time with GPT4, OpenAI as been able to predict model progress with accuracy:
> A large focus of the GPT-4 project has been building a deep learning stack that scales predictably. The primary reason is that, for very large training runs like GPT-4, it is not feasible to do extensive model-specific tuning. We developed infrastructure and optimization that have very predictable behavior across multiple scales. To verify this scalability, we accurately predicted in advance GPT-4’s final loss on our internal codebase (not part of the training set) by extrapolating from models trained using the same methodology but using 10,000x less compute:
> Now that we can accurately predict the metric we optimize during training (loss), we’re starting to develop methodology to predict more interpretable metrics. For example, we successfully predicted the pass rate on a subset of the HumanEval dataset, extrapolating from models with 1,000x less compute:
> We believe that accurately predicting future machine learning capabilities is an important part of safety that doesn’t get nearly enough attention relative to its potential impact (though we’ve been encouraged by efforts across several institutions). We are scaling up our efforts to develop methods that provide society with better guidance about what to expect from future systems, and we hope this becomes a common goal in the field.
Isn't this all based off self-attestation? There is no comprehensive audit of their research data and finances I am aware of. If I was OpenAI and blew millions of dollars training models that showed exponentially worse performance for incrementally more resources expended training the model, my next step would not be to publish about it.
I am a mild LLM skeptic. But I find the response of "oh, it's all just post-crypto scamming" really weird. Crypto was a total scam. There was never a concrete, non-criminal (important caveat) application where crypto was easier than just using PayPal or whatever. LLMs are very imperfect and still have a lot of work to do, but they do actually do some job related tasks today. If I have JS snippet and I wish it were in Go or vice versa, I can ask an LLM and get a good translation. If I have a document, I can ask the LLM if there's something I should add or simplify. Are these world changing capabilities? No, not yet. But they do exist and they are real and they are new. So yes, of course investors are interested in a new technology whose future limits we don't know yet.
The LM industry valuation would be way smaller if they were not laundering behavior that would be illegal if a human did it. If "AI" were required to practice clean-room design (https://en.wikipedia.org/wiki/Clean_room_design) to avoid infringing copyright, we would laugh at the ineptitude. If people believed the FTC-CFPB-DOJ-EEOC joint statement was going to lead to successful prosecutions, the industry valuation would collapse. https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CF...
If you spend weeks drilling flash cards on copyrighted code, then produced pages of near-verbatim copies with copyright stripped, any court would find you to have violated the copyright. A lot of people right now are banking on "it's not illegal when AI does it", and part of that strategy is to make "AI" out to be something more than it is. That strategy has many parallels to cryptocurrency hyping.
As someone who have been very anti-crypto for a long time, it wasn't always a complete scam.
The first wave of the crypto boom, before anyone that wasn't a programmer had even heard of it, there was a lot of real work being done that very much mirrors current AI work. Lot's of very sharp developers learning about block chain, figuring out how to implement things, experimenting with ideas. Back then everyone owned their own wallet and you would meet at coffee shops to exchange cash for BTC.
Most of the serious engineers that were really into crypto during the first crypto boom of 2012 left in disgust when the second boom came around.
Having worked in AI/ML for a long time, I myself can start to see how they felt. We do have some really cool technology in front of us, I think it has a lot of potential, but so many of the loudest voices in this space are entirely out of touch with what's possible, and far more interested in hype and making money than the underlying technology.
> There was never a concrete, non-criminal (important caveat) application where crypto was easier than just using PayPal or whatever.
That isn't true.
You can use it anywhere that irreversibility matters. Suppose you're going to commit significant resources to the customer's request, so you charge them, commit the resources, deliver the goods, and then discover that they gave you a stolen credit card and you get a chargeback. Cryptocurrency avoids that.
You can use it to accept payments from all over the world. Someone in Asia or Africa may not be able to open a US bank account or get a US credit card, but if they can find a Bitcoin ATM to put their local currency into, they can pay you, or vice versa.
It allows you to pay for something over the internet without giving your name. There are situations where this is important.
The main impediment to using it is, ironically, regulatory. The IRS decided that it's an investment and not money so every time you want to use it for what it's actually supposed to be for, they treat it like a securities transaction where you have to fill out paperwork, even if you're just buying a pack of gum. Which makes it much less convenient for ordinary people to use than cash or credit cards which don't require this -- presumably on purpose in order to destroy its utility in the US.
But it can still be useful for people in countries that don't do this, or in the US if a less explicitly antagonistic regulatory environment could be established.
I apologize, because HN does tend to automatically go to crypto when discussing blockchain.
I was not talking about crypto. Blockchain “solutions” in enterprise were spinning up all over the place for non-crypto applications. In particular, in you work financial, supply chain, government or random startups, you probably heard blockchain a lot in non-crypto contexts.
There was incredible hype. That doesn't mean there was 'nothing to get hype about'. Just that the hype might have overshot, not that GPT isn't amazing.
It doesn't follow that when new tech arrives, the post hype everything goes back to status quo. Typically after the hype, the new tech just grows or gets absorbed a little more quietly, in un-foreseen ways, and does end up having a big impact. Just when the impact is stretched out a little, people stop noticing.
Like replacing drive through ordering with GPT like tools. Kind of under-radar, not fancy, not flashy. At some point you'll notice that the drive through you are talking too isn't a human, and go 'huh, that's interesting'. But, big impact on jobs, so nobody is hyping it.
People/Companies are already using AI to replace or augment Graphic Artists, Coding, etc... etc... That is happening, not just a power point from a middle-manager.
"Research" is not some monolithic single concept. One might also ask, "How can research produce ChatGPT when for decades research failed to produce ChatGPT?"
Exactly. So why now are we trusting 'Research' that is trying to predict the future of other 'Research'. The linked article is just some estimates on the error built into the current LLM model.
How can we extrapolate that to be "well, gosh darn, these LLM's are already played out, guess we're all done"
