I need to digest this but it is a seductive idea. My quick take: there may be a connection between back-propagation and reversibility, both computational and physical. For a system to be reversible implies conservation of information.
It also makes me think about the surprising success of highly quantized models (see for example recent paper on ternary networks, where the only valid numbers re 0, 1, and -1.)
Artificial Neural Networks were originally conceived as an approximation to an analog, continuous system, where floating-point numbers are stand-ins for reals. This is related to the ability to back-prop because real functions are generally differentiable. But if it turns out that we can closely approximate the same behavior with a small, discrete set of integers, it makes the whole edifice feel more like some sort of Cellular Automaton with reversible rules, rather than a set of functions over the reals.
Finally (sorry for the rabbit-holing) - how does this relate to our brains? Note that real neurons "fire" -- that is, they generate a discrete event when their internal configuration reaches a triggering state.
Yes, that is roughly correct. You nay want to look up "reversible computation". It's a fundamental part of quantum computing, for one thing.
The key insight is that a (finite) discrete, reversible system will always eventually cycle back to its original state. This fact has very interesting follow-on implications for the concept of entropy and the Second Law. If it is guaranteed that a system will return to a prior state, how can it also be true that entropy (disorder) always increases?
It also makes me think about the surprising success of highly quantized models (see for example recent paper on ternary networks, where the only valid numbers re 0, 1, and -1.)
Artificial Neural Networks were originally conceived as an approximation to an analog, continuous system, where floating-point numbers are stand-ins for reals. This is related to the ability to back-prop because real functions are generally differentiable. But if it turns out that we can closely approximate the same behavior with a small, discrete set of integers, it makes the whole edifice feel more like some sort of Cellular Automaton with reversible rules, rather than a set of functions over the reals.
Finally (sorry for the rabbit-holing) - how does this relate to our brains? Note that real neurons "fire" -- that is, they generate a discrete event when their internal configuration reaches a triggering state.
Lots to chew on...