On: Karpathy's MinGPT

[stainforth]

You're saying your project costed millions of dollars, or the big boys' projects did?

[shmageggy → stainforth]

If "4 days across a couple dozen machines" cost millions, something is very wrong.

[T-A → shmageggy]

Not if it was a couple dozen of these machines:

https://www.hardwarezone.com.sg/tech-news-nvidia-dgx-a100-su...

[GaryNumanVevo → shmageggy]

Running your own DC is quite expensive with GPU hardware. One DGX-2 is $400k and draws something like 24kW.

[rajnathani → GaryNumanVevo]

> draws something like 24 kW.

That number is off. The DGX-2 consumes 10 kW at peak [0] and the DGX-2H consumes 12 kW at peak [1].

[0] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...

[1] https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Dat...