Ask HN: Why are a lot of AI projects written in Python?

qsort · on Oct 28, 2022

If you're talking about the industry, machine learning and data science projects are inherently experimental processes that rely on setting up quick feedback loops to analyze your data and see what works.

The ideal setup for that is the interpreted shell/compiled core approach that languages like Python have championed forever. Competitors to Python would be the likes of R, Lua, Ruby, Perl, Javascript; not C, C++ or Rust.

Why Python in particular? The answer is that it has an excellent scientific stack with Numpy, Scipy and Pandas.

R is very competitive for statistical work.

Julia is an interesting, if unproven, alternative.

Once you're done with your model, you write it in C (or whatever) for production, but that's a more traditional software engineering project.

In academia, the incentives are such that most people don't really care about code. Their careers depend on them publishing papers, so that's where they're going to focus their efforts. Code is a way to prove a point for publication, not a product that needs to generate value.

> a lot of things are popularly rewritten in Rust or Zig

Beware of the HN bubble. Rust and especially Zig are tiny, fringe players in the industry, regardless of their technological merits.

jstx1 · on Oct 28, 2022

> Once you're done with your model, you write it in C (or whatever) for production, but that's a more traditional software engineering project.

This doesn't always need to happen IRL, most large ML systems are running TensorFlow/PyTorch (a few run C++ without the Python layer).

jstx1 · on Oct 28, 2022

The best libraries are in Python.

> Curious that since a lot of things are popularly rewritten in Rust or Zig, why has this not happened on these types of projects?

1. It's a lot of work and you need to good hardware support. There are lots of people putting in a lot of work into PyTorch, TensorFlow, JAX etc.

2. Some things have been rewritten, mostly as hobby projects.

3. Even if you rewrote most of this sutff, it would be likely be more difficult to use. Writing Rust is a completely different experience from writing Python + PyTorch/TensorFlow.

4. Zig is a thing only on HN and in similar mini-bubbles. Hardly anyone has even heard of Zig, let alone considering to rewrite anything in it.

username-404 · on Oct 28, 2022

> Curious that since a lot of things are popularly rewritten in Rust or Zig

Are you referring to personal or professional codebases rewritten in Zig? Personally I haven't seen the Zig language used professionally in 1 company. And my impression from the last discussion about Zig:

https://news.ycombinator.com/item?id=33331549

was that the compiler was still too unstable to be used for real work.