Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does.
I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch.
JavaScript is so much faster than Python, and the package management experience is so much better, that I really think a JS library would have a chance to take market share from PyTorch in the long term if it had those features.
What's wrong with creating a function that does those things? It would be less surprising to people new to the library, would be self-documenting by having a name and an easily inspected declaration with named arguments, and it would be idiomatic JS. You could also have variants that are purely functional and return a new value or ones that mutate in place that you could use depending on your needs.
What's wrong is it obscures simple math expressions behind tons of dots and parentheses. The thing is that the core of deep learning algorithms is usually very simple math. It's useful to be able to translate that math directly from research papers into straightforward expressions that mirror the structure in the paper like a = b / c + d * e rather than something less similar like a = b.divide(c).add(d.multiply(e)).
In python when I try a new gpu accelerated array library, to write norm_arr = arr / lib.sqrt( lib.sum( arr*2, axis=-1, keepdims=True) , I have to read documentation for sum to see whther they use axis or dim in sum.
In javascript, to write the same thing I need to read documentation for a sum function, a broadcasted division function, a multiply function. I can probably assume that the sqrt function behaves and is named as I would expect
If each of those operators were implemented as functions then you'd have different names for different implementations in order to avoid confusion over what type of division or multiplication they were performing. It's more verbose but that's a good thing since it prevents you from making incorrect assumptions about what's going to happen when you do a * b.
Because there is nothing to make assumptions about. In the example code, both multiplication and division have a scalar on one side, there's no possible ambiguity of behavior. But there is the eternal question of terminology: do you specify dimensions by "axis" or "dim" and does your API actually use both terms in different places?
I'm glad that proposal was withdrawn; it would've been by far, in my opinion, the worst implementation of operator overloading, in any (mainstream) language
Could not agree more hahaha! I tried to work around it building methods like “torch.add(a, b)” for operator overloading and “torch.at(index)” for slicing. But it’s definitely not as seamless as these features you proposed.
You should do it! If you actually had a solution for operator overloading you'd really stand out from the other various JS deep learning libraries. Save me from pip and conda please :)
I can try implementing it in the future lol
It would surely be a quality of life improvement.
But with the current tools, I tried my best to make the syntax as similar as possible to PyTorch’s!
If you do want to add evaluation of mathematical expressions you should check out Math.js since they provide a parser among other utilities. Please make it optional though, it would be a nightmare to debug if everything was written in strings.
Curious, why do you need to construct these as class instances, like operation = new Exp() ? Seems like a lot of extra overhead constructing those objects. Why not just have Exp contain static methods for forwards and backwards?
[edit] nevermind, I missed the cache step. Still not sure it wouldn't be more performant to centralize caches as plain objects somewhere rather than to call new() on every op...?
I centralized the entire backpropagation around the Operation objects. They store the data about the forward prop in the cache, and serve as the connections in the graphs between tensors. Each tensor has a “parent”, “child” and “operation”. These store who generated the tensor, what tensors it generated, and how it was generated (what operation). I could store the backward function inside of each tensor instead of an Operation object, but I chose the slightly more verbose option because I think it is a little more interpretable and simpler to add new operations.
Would JS be faster than Python when it comes to Pytorch? For example, I seriously doubt that would be the case for Numpy, since it's a wrapper for C code, with the ability to use Fortran libraries for optimization.
There is often a lot of work that has to be done before and after PyTorch gets involved. For example the code I'm working on right now involves reading and parsing a bunch of files, filtering and extracting a bunch of data based on various criteria, formatting that data, passing it to a PyTorch model and then taking the results from PyTorch, validating it, reformatting it and then writing it to disk. The PyTorch part is probably as fast as it can get, but most of the overall runtime is spent doing all that other stuff and if you can speed that up then that is a clear win in many cases.
The benefit of having it in JS is not speed but portability and access to the JS ecosystem of tools. Having the code run in the browser without needing a complex setup is a huge benefit for sharing demos. Node.js provides a way to use native code as well and it's quite commonly used https://github.com/nodejs/node-gyp so there's no reason you couldn't use those same or similar libraries in a JS implementation.
C running on the CPU isn't fast enough for ML. You need to run on GPUs or TPUs if you're serious.
Yes, most of the tensor operations in PyTorch do their math in native code. However, Python still does orchestration and other tasks like data loading and because it is so slow it still ends up causing a ton of overhead in many cases despite offloading most of the work. It's very common for the GPU to sit idle between kernels while Python spins. So JavaScript being faster could still be a big advantage.
Very much the opposite, since this is pure JS. PyTorch uses tuned native-code and GPU components for the heavy lifting and can in some cases compile Python code using PyTorch JIT / torch.compiler / torch.fx.
You could use eval to expand whatever syntax you like into function calls. It could be done once at the start of the program and act like a js preprocessor.
I know eval is not kosher but this problem is also not real, so why not.
I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch.
JavaScript is so much faster than Python, and the package management experience is so much better, that I really think a JS library would have a chance to take market share from PyTorch in the long term if it had those features.