@trekhleb: Can you modify the images used in the README.md - they seem to be PNG files with a transparent background and the formula (or whatever is there) in black; on github in "dark mode" (or with a stylus or similar css modifier for a dark mode) the black text becomes unreadable. t/y
Nice work op. There is some stuff at fast.ai that you might also enjoy. I recently learned about Swift for TensorFlow from one of their videos which modifies Swift to allow for placing attributes over functions and then having the compiler automatically generated the derivatives of those functions.
Sadly, no. This OP's writeup is just about the simplest exposition that doesn't miss the point, and it misses the major transition when you go to layers with nonlinearities between them.
I think there might be a fun idea in that. I wonder what kind of learning materials we could generate by holding competitions on who could explain a topic the simplest
> This OP's writeup is just about the simplest exposition that doesn't miss the point, and it misses the major transition when you go to layers with nonlinearities between them.
Well - the activation function appears to be linear, and thus close to RELU - and so (maybe?) there wouldn't be any issue with the so-called "vanishing gradients" problem? But I am just a hobbyist who dabbles in this topic, so my opinion doesn't carry much weight...
What this particular model shows is probably about as simple as you can get; the activation function (what is called "predict" in the code) is a simple linear model (very similar to RELU - which has other important properties valuable for multiple layer models vs other activation functions). You probably couldn't make it more simple unless you made it a binary function (ie - set it to "1" if the value is anything over zero, otherwise set it to "0").
Everything else basically stays the same, though, and it can't get much simpler.
Here's the thing though: As far as anyone knows - and research is ongoing on this, so this may change at any time - heck, it may be different now than the last time I looked into it - but anyhow, as far as anyone knows, in nature (that is, actual neurons in a brain), backpropagation is not done, nor is there an apparent mechanism to allow for it.
This has led Geoffrey Hinton to think maybe things should be rethought (and he has gone down the road of something called "capsule networks" which I have yet to understand - I can't tell if they are actually simpler or more complex from what I have read):
This doesn't mean that backpropagation doesn't exist in nature, just that we haven't found a mechanism for it (although there are some competing theories).
I have also read research papers on other ways of making a neural network learn - one was something akin to, instead of backprop, randomly picking neurons in the layers and re-assigning them with random weights - and amazingly, the network would still learn.
All I can really say is to dig in and think about it, try to learn all you can about both artificial neurons and network implementations, along with the history of the field.
For instance, many don't know that Alan Turing played around on paper with neural networks a few years after the Mcculloch-Pitts model was released; I tend to wonder if there is anything in his notes that is worthwhile or has been overlooked - his network models, from what I recall, were not structured, and I believe his neurons were simple binary functions - likely, nothing is there, as people had to have studied this work, as obscure as it is.
Also study and learn what we know about actual neurons and the networks they form in "brains" (from simple things like C.Elegans to humans); some researchers believe that so-called "spiking neural networks" more closely model actual neurons than the commonly used style of mathematical model of the sort shown by this tutorial/github.
And keep in mind that all of this is "models" - not the real thing, and shouldn't be expected to behave the same as the real thing. But - maybe something simpler can be done that comes close - much like an airplane is a simpler machine than a bird, and looks and works nothing like a bird. Perhaps there is something simpler for a model.
But then again, nature seems to have come across a better solution than what we currently use - after all, our brains use no where near the amount of energy a GPU uses - and that GPU is used to simulate much simpler models than what the brain is. So what is the secret?
Nobody as of yet knows - but the first to crack it may become a very wealthy person (that, or depending on what happens, they may inaugurate the technological singularity with an advanced GAI that runs roughshod over mankind, viewing us as inconsequential as we do ants and insects on a construction site today - and as such, that person would become the most reviled human ever, for as long as the human race remains in the face of the changes)...
