It's important to recognize that the space of all programs that take x,y as input and then output 1 or 0 is equivalent to the mathematical space of all possible functions. We interpet the programs as indicator functions.
Neural networks are usually smooth because they must be differentiable. Stochastic binary neurons and other techniques take us into a larger space of possible mappings, discontinuous ones.
But yeah, ultimately the model we solve for, will affect what kind of interpolation the solution ends up having.
The biggest question is what kind of interpolating techniques the brain uses. The search for the magic model....
https://en.wikipedia.org/wiki/Indicator_function
Neural networks are usually smooth because they must be differentiable. Stochastic binary neurons and other techniques take us into a larger space of possible mappings, discontinuous ones.
But yeah, ultimately the model we solve for, will affect what kind of interpolation the solution ends up having.
The biggest question is what kind of interpolating techniques the brain uses. The search for the magic model....