Given the cost of gpus it would be negligent if they weren’t at least looking for alternatives. A story like this could also help negotiate prices with their suppliers. And everyone is looking at the success Apple had with custom silicon. But I suspect they’d prefer to partner or find alternative (cheaper) suppliers
It’s interesting to think, when a company starts to vertically integrate, how deep do you go?
Seems like OpenAI is exploring its own devices/OS as well, which makes sense to me, but it’s a vertical integration bet. This seems to be another big bet, but they could benefit from having their own optimized chips regardless of whether the device/OS bet wins out.
As far as gives you a competitive edge! For AI, to win you need better data and compute than your competitors.
Going up the stack to consumer devices seems like a somewhat speculative move, though I understand the underlying desire to secure a data moat.
Going down the stack to chips makes a lot of sense; if you can secure an edge in compute efficiency then you will beat anybody that doesn't have substantially more data than you do.
Very exciting (from an engineer point of view), but there is much much more in a chips/devices/OS than "AI". It's a trap!
To me, it seems like a distraction for them to go into devices stuff, rather than making their stuff so relevant that device vendors (Android/iOS) can't ignore. At the moment they can because they got good enough competitor solutions.
If they found a theoretical way to infer something like GPT-3.5 without using so much RAM and can build a chip which makes this feasible in laptops or (holy grail) phones, they've got their moat for a 12-24 months, possibly more if they manage to patent it. Big if though.
Considering the unit prices of NVidia chips, and the growing cuda software lock-in, it seems “obvious” to work on something to hedge against price increases. That might be the performance or cost edge they need.
If they just make android phones better, they risk Google benefitting off their efforts and replacing the models. Besides that’s a lot more work to partner and manage across the industry.
It seems risky to bet they can release consumer hardware (instead of server side) but that’s the ultimate big bet. Ask Meta how it feels to be “just an app”.
This makes a lot of sense. When ChatGPT initially broke the mold, I was hoping someone would find a way to repurpose all the silicon the crypto-bros' nonsense has commandeered -- alas, the problems are too different.
Making specialized chips to run LLMs is the logical next step.
When building out an initiative like this, how do companies avoid IP issues? They are looking to build technology that competes with the best in class to make it worth the effort without having to reinvent the wheel.
This is not answering question, on what OpenAI will earn money. As even using only Nvidia chips, industry need somewhere gather ~$1000B (half of this is OpenAI share now), but with custom chips will be at least 3-4 times larger numbers.
The economy is in an interesting place when in house chip making efforts or startups are popping up now. It used to be a task like landing on the moon. Still a difficult effort but it looks like this industry is expanding. It speaks to the rapid nature of technology in general where insurmountable tasks over time become closer to trivial.
I said a while back that I expect the major cloud vendors (Azure, AWS, GCP, etc) to start trying to develop their own chips for AI work. Google already does to some extent with their tpus. At the very least, this is saber rattling trying to convince nvidia to lower prices.
I think that endgame for generative language models is models embedded directly into chips. Computers that run english language instead of machine code and for which CPU, GPU and what is currently known as PC is more like a peripheral IO device.
> I think that endgame for generative language models is models embedded directly into chips. Computers that run english language instead of machine code and for which CPU, GPU and what is currently known as PC is more like a peripheral IO device.
That may be the endgame, but I think if it is there is a long time before attempts to jump to it aren't going to fail like every high-level-system-in-hardware for other than very niche applications, because general purpose (comparatively, even if specialized for running AI models) hardware will be good enough that the value of being able to upgrade the models it is running will outweigh any marginal temporary edge that current-models-in-hardware have.
That might not be a serious issue if we keep updating our personal hardware at the present frequency. So you’ll be able to decide whether to update to the new iPhone with English 4.7, or rather stay with English 3.5 for another year.
That's basically how our neurons work. New neuron growth and connection isn't much of a factor in learning. Rather it's the synaptic restructuring (equivalent to AI model weights) that change relatively quickly.
So we need to figure how how to "grow" mechanical brains. I envision this being done with a new generation of FPGAs tailored to this task.
Are we watching the same company? They have been shipping new features like crazy. Code interpreter blew peoples' minds, they are training GPT-5 and I bet they are leaning into multi-modal more than with GPT-4V. Strategically multimodal is the big frontier on which they are going to continue expanding datasets.
With that in mind, expanding their apps to ingest more audio and image data is an obvious strong move. (And you can see why a consumer device would help them get even more data from the real world, though it's less obvious to me that this is a win vs. just shipping apps.)
4 months is an eternity in AI right now. I would be shocked if they are not training a new GPT (at least in experimentation/research mode, if not full pretraining) - else what are their GPUs spinning on? That is not a capital investment you just let sit idle.
But I concede that I don’t have any concrete proof so I should modulate my certainty of tone.
Still feels like they're laser-focused on achieving AGI/aligned ASI. Improving chips and evolving the UX feel like cohesive (possibly requisite) intermediate steps
I guess they are trying to find a path to profitability and scalability: the current setup will not to scale much further, and they need some more energy efficient solution.
I was just thinking the inference cost could be reduced by making hardware with less error correction in specific areas to get higher density, and let the NN work around the limitations.
Can't we have hardware companies that make hardware, software (AI) companies that make software, and data companies (or government institutions) that run the software on the hardware and deal with our data?
We had that, then we realized that companies who understand all the pieces lead to better user experiences, which is why we all have MacBooks and iPhones.
Using my Apple Silicon Macbook is a way better user experience than my Intel Macbook ever was. It certainly feels like they invested in user experience well after the walls were up around the garden.
> It also creates walled gardens and reduced incentives to invest in user experience once those walls are built.
> Not because the UX is bad
I don’t see the connection. The first sentence says that walled gardens create bad UX, but Macs are the premier mainstream walled gardens and you don’t find their UX bad.
I dunno, the last two generations of MacBook pro pretty much feature for feature provide fixes for all the widely criticized issues of the last Johnny Ive models. Apple has plenty of faults but they seem to actually be listening to user feedback in some areas despite a captive audience.
This is a funny example because Apple has disproven this logic just as often as they have proven it. Carplay and the App Store are two examples. People generally prefer the software Apple makes over the software car manufacturers make. People also like to install their own software on their devices beyond the software that Apple makes.
Just because vertical integration occasionally works doesn't mean it is actually good for the consumer.
“good for the consumer” is as subjective as the consumer.
the eu disagrees in that it views itself as representative of the consumer. similar to how us states set laws that may be more strict than others.
get a large enough government of a populace with a large enough portion of the sales, and companies can be made to act. which can be good, but isn’t a guarantee.
just because companies can be forced to act doesn’t mean the forced actions are actually good for the consumer.
Honest question, is Google the one making it impossible to set Google Maps as the default maps option on Apple? Or making it impossible to one-click delete backed up photos?
I fully believe Google holds back differentiating features and DSPA is certainly mismanaged, but I'd hope these basic functionalities are not the ones since there's always the tradeoff between serving 50%+ of your mobile users and trying to differentiate Android/Pixel...
Apple does shit like make it impossible for Google Fi to set up easily on iPhone, so I just assumed most of the UX idiocy came from Apple's anticompetitive review process.
When hardware companies fail to provide sufficient competition to an extortionate monopoly, the layer1 companies react after feeling the burn for too long.
If Nvidia, Amd and Intel were in a battle for offering the best VFM, none of these companies would be hopping into hardware. Apple's strong commitment to chip making coincides with years of stagnation from Qualcomm and Intel.
From my experience, companies love nothing more than a 3rd party that solves your problem for you, better than you and at a price that's easily cheaper than what I'd cost to build it in-house. This is especially true when the 3rd party product is an internal spec (gpu, cpu) rather than a competing platform (android auto)
There is a reason car companies don't build their own speakers or tires....but still try to build their own UI (no matter how bad)
Moores law is dead when it comes to power consumption, that means compute is getting more capital intensive.
If you have an algorithm that works and need scale, you must vertically integrate to maintain an edge over those using more general compute architectures.
The vertical integration allows them to corner the market and destroy the competition or prevent them from entering the market, which is bad for the consumer and bad for society, eventually.
It's lack of anti-trust laws and (more importantly) enforcement that allow companies to corner the market. nVidia and ARM should be worried about competition from OpenAI and Google: that's the good kind of competition that we want. If only "hardware companies" are legally allowed to make hardware, that increases their moat, not decreases it.
Let's instead go back to when we actually enforced the anti-trust laws that we have. That was nice.
Would OpenAI be competing with them, though? If they just manufacture their own and use their own, but don't sell them, then nVidia and ARM are just losing a client, not getting competition.
Ideally what we would see is OpenAI investing in a new but independently operated chip manufacturer that makes chips to their standards. Though is it also possible that a chip of such standard would be so specialized that it wouldn't be usable by others? I'm not a hardware person, so that's a genuine question.
I'm very hesitant to define competition as "can't sell to them because they make theirs in house". Losing a sale isn't competition. Competition is someone threatening to take your clientele from you with their own offering. Apple can afford to design and use its own chips, but they're in a pretty unique position to do so. It's not like every other consumer of their chips is going to say "hey, if it's that easy, I'll do it too".
I would go even further. I was wondering why most chip companies are so good at being mediocre. Like TI OMAP dropping out of smartphones and so on. But I think it's worthwhile to invert the question. It is quite extraordinary in the silicon space for there to be 1 clear winner like Nvidia or Qualcomm (or Intel not so long ago). So much so, that we can assume they got there by anti-competitive means and rent-seeking measures.
Ironically, Open AI are going to face the same gigantic barrier to enter the market.
As of today, I say if they go into consumer market they will fail.
If they go into specialised server chips, then they would have a chance, with some sort of accelerator over some arm-based chip - similar to what Nvidia is doing with Grace. Still big money to be spent on supporting the existing ecosystem on their hardware.
Vertical integration is already under scrutiny by the antitrust folks. Even more so now after everything Google was able to get away with.
OpenAI wanting to vertically merge to make their own AI chips may seem harmless enough (it's a good business move, we can cut expenses)! But we can't forget that Sam Altman just a few months ago told Congress he supports making an organization that companies need permission from to being creating/utilizing advanced AI systems. And he's such a kind man he's willing to lead that organization himself.
Obviously someone integrating the chips to train AI, and having the ability to approve/deny his own competition is a huge red flag.
Well kind of. You are ignoring the elephant in the GPU hardware room right now, which is NVidia.
NVidia has a near monopoly on the AI hardware market right now, so some vertical integration of alternative AI hardware doesn't seem nearly as big if a deal if it is needed to fight that current monopoly.
That's true, but again, him attempting to put himself as the gatekeeper of who can and cannot train/utilize advanced AI systems at scale gives him (almost) unilateral control over Nvidia's own AI chip manufacturing business. Deny startups the ability to train AI systems, and you deny Nvidia the opportunity to sell them their chips. Nvidia loses that revenue stream and stops producing AI chips due to high production costs and "lack" of demand. Ultimately leading to a market where only the obscenely wealthy companies who can manufacturer their own in-house chips can train AI, and even then, Sam Altman can deny even that.
I bet OpenAI is trying to figure out how to have self-improving AI which suggests modifications to the hardware that runs it. You’d need some hardware expertise in-house for that (in addition to software of course), though not necessarily a big chip company.
Dealing with this professionally for DNNs. It just doesn't work. The large, important DNN models are so complicated, the toolchains for optimized execution don't do sane things unless you do some sort of vertical integration. The community tried with things like TVM, Halide, ONNX and others .. it is just crazy if you don't have a fully opinioned pipeline. Just my personal opinion.
Every company is going to want to avoid paying rent on key infrastructure that they rely on for their business, thus the push to own their own IP and means of production in that regard.
It doesn't work out all the time, for sure. In fact it probably fails more than it succeeds. But the motivation is pretty clear.
This is especially true when you are beholden to a single supplier, with basically no competitive options.
Right now nvidia completely owns the AI market, and is exploiting that absolute monopoly with ever escalating pricing, licensing and restrictive usage models. Everyone keeps trying to escape this -- see Tesla and their super-hyped and now apparently abandoned Dojo thing, while they put in their orders for tens of thousands of H100s -- but instead they keep being beholden to nvidia.
That may seem to be an advantage. But because there is only 1 (or a few) companies now controlling the whole stack, you have less choice in the products you can buy. And since the incentives of the vendor may not be aligned with yours, which is increasingly the case if they are a monopoly, then the product is not actually better from the consumer's point of view.
Don't like the way OpenAI treats your data, or how you can only run it in the cloud and not on an on-premises server? Or what dataset they used for training? You're out of luck!
But if the market were more modular, and lots of small companies could use the same hardware in their products, you'd have something to choose from!
