> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks?
It’s several things:
* Cutting-edge code, not overly concerned with optimization
* Code written by scientists, who aren’t known for being the world’s greatest programmers
* The obsession the research world has with using Python
Not surprising that there’s a lot of low-hanging fruit that can be optimized.
I’m not sure this is fair - a lot of the performance optimisations have come from applied mathematicians rather than programmers, and python is not generally considered to be the bottleneck (it is the interface rather than what is running the computation - it calls a C API which then often uses CUDA and may also run on hardware specifically designed for ML).
Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
In ML python is effectively the interface rather than the bit that is doing the heavy lifting.
The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).
You're completely correct that the speed-sensitive parts are written in lower-level libraries, but another way to phrase that is "Python can go really fast, as long as you don't use Python." But this also means ML is effectively hamstrung into only using methods that already exist and have been coded in C++, since anything in Python would be too slow to compete.
There's lots of languages that make good tradeoffs between performance and usability. Python is not one of those languages. It is, at best, only slightly harder to use than Julia, yet orders-of-magnitude slower.
In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.
It’s several things:
* Cutting-edge code, not overly concerned with optimization
* Code written by scientists, who aren’t known for being the world’s greatest programmers
* The obsession the research world has with using Python
Not surprising that there’s a lot of low-hanging fruit that can be optimized.