On: YaFSDP: a sharded data parallelism framework, fast...

[dayeye2006]

Any idea on what are the main tricks used to achieve gains over fsdp?

[albertzeyer → dayeye2006]

The blog post seems to contain more details and the core ideas: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-train...

[az226 → albertzeyer]

Odd that they don’t expand on this:

In Yandex’s pre-trainings, the implementation of YaFSDP along with other memory optimization strategies resulted in a speed gain of 45%.