The market segmentation is likely a result of Nvidia's monopoly position. They double the RAM and flops, improve the thermals and housing and sell for ten fold the price. It doesn't make sense to me. A cheap 4090 theoretically outperforms even the A6000 RTX Ada. https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...
Nvidia needs to satisfy gamers, who individually can't spend more than a few $k on a processor. But they also have the server sector on lockdown due to CUDA. Seems they can easily make money in both places. Maybe those H100s aren't such a good deal...
If someone understands these dynamics better I'd be curious to learn!
Nope, this is about it. They try to force the larger users into the expensive cards by prohibiting datacenter use in the driver EULA. This works sufficiently well in America, but it also means that you can find German companies like Hetzner that will happily rent you lots of consumer cards.
(There are also some density advantages to the SMX form factor and the datacenter cards are passively cooled so you can integrate them into your big fan server or whatnot. But those differences are relatively small and certainly not on their own worth the price difference. It's mostly market segmentation.)
The main limiter in the data center setting is licensing, interconnects, and ram.
By contract - you can’t sell 4090s in a data center. You’ll find a few shops skirting this, but nobody can get their hands on 100k 4090s without raising legal concerns.
Likewise, nvidia A100s have more than a few optimizations through nvlink which are only available on data center chips.
Lastly, per card memory matters a lot Nvidia has lead the market on the high end here.
Has that been happening? I guess there's been a bit of a dip after the crypto crash, but are prices staying significantly lower?
> or they develop at ASIC or more efficient model
This seems likely. Probably developing in partnership with Microsoft.