On: Llm.c – LLM training in simple, pure C/CUDA

[simonw]

The size of CUDA really is astonishing. Any chance someone might figure out how to slim that down?

[jsheard → simonw]

Taking a peek inside the package it seems to mostly be the libraries - CuFFT alone is about 350MB for example, twice over for the debug and release versions. I'm guessing those are probably fat binaries pre-compiled for every generation of Nvidia hardware rather than just the PTX bytecode, which would help to speed up fresh builds, at the expense of being huge.

[phdelightful → simonw]

Here’s a blog that breaks down how large different pieces of CUDA are:

https://carlpearson.net/post/20231023-cuda-releases/

[xiphias2 → simonw]

Talking directly to the kernel / driver / firmware.

As others have said, George Hotz is doing his best in reverse-engineering and skipping layers.

[maille → simonw]

Raise your voice on their forum: https://forums.developer.nvidia.com/t/how-to-overcome-the-hu... Tried my luck 2 years ago but it keeps increasing.

[dartos → simonw]

Nvidia is the only one who could, since they own it.