On: Google TPU v5p beats Nvidia H100

[wmf]

Your numbers are still off. A GH200 pod has 144 terabytes of RAM; it isn't even measured in gigabytes. It looks like TPUv5p may have 851 terabytes which would be a generation ahead but you didn't show those numbers.

[versteegen → wmf]

nabla9 said "one Grace Hopper" not "one GH200 pod".

Actually, DGX GH200 seem to come in different sizes, something that I can't find clearly stated on Nvidia's website, but what I do see are entirely inconsistent specs. I'm thinking they've changed it a few times.

https://developer.nvidia.com/blog/announcing-nvidia-dgx-gh20... describes the DGX GH200 as:

-256 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU)

-each Grace CPU has 480 GB LPDDR5 CPU memory

-each H100 GPU has 96 GB of HBM3

-each GPU can access all of the CPU and GPU memory in the whole system (144TB), at 900 GBps

At https://www.nvidia.com/en-us/data-center/dgx-gh200/ (linked to from the DGX page from the website) which has a Datasheet pdf they say the DGX GH200 has:

-32 NVIDIA Grace Hopper Superchips (1 CPU + 1 GPU)

-each Grace CPU has 72 "ARM Neoverse V2 Cores with SVE2 4X 128"

-each GPU can access "19.5TB" shared memory

-that's 624GB per superchip, which is weird. I expect it's actually 96GiB HBM3 + 512GiB LPDDR, a total of 19456GiB = 19.0TiB

And other people have found completely different specs elsewhere on the website!