Scary possibility: what if there is no good formal theory to explain how it works? What if intelligence, both animal and machine, is purely random trial and error and "this thing seems to work"?
I don't believe that's true necessarily, but it will sure hamper the authors hopes.
In the same way that there is no formal theory for the exact shapes of proteins? I think it’s possible. But as with proteins, there are probably some general aspects of the problem that can be explained in simpler terms.
This might be indeed true for deep learning in its current shape. However, in the long run I think we will see more models that (1) are more composable and (2) better approaches to engineering such models, however, these, engineered models will look a bit different than what your research scientist throws over the fence today.
At the moment we are in a phase were, to stick to the optics metaphor, we stack up lenses until we see the object on the screen. This means we end up with models that are sprawling, instead of having models that were engineered.
Another trend that seems to start in deep learning is that layers become more constrained. I expect, that in 20 years, we will see much more constrained models and much generative models.
2. hopefully with time we'll have better approaches to engineer all things that are engineered
No, at the moment we go for the biggest and shiniest lens that we can get our hands on and hope that it's capable enough to tackle our problem. If it is we can waste time designing a smaller, more constrained, lens to ship to consumers.
I'm curious if you two are going to be talking past each other with the first point. Any chance I could get you both to explore what you mean by composable?
These days any method that uses gradient descent to optimize a computational graph gets branded as deep learning. It's a very general paradigm that allows for almost any composition of functions as long as it's differentiable. If that's not composable then I don't know what is.
My guess for what they meant was that you can't compose the trained models.
For example, a classifier that tells you cat or not can't be used with one that says running or not to get running cat.
The benefit being that you could put together more "off the shelf" models into products. Instead, you have to train up pretty much everything from the ground. And we compare against others doing the same.
Scary possibility: what if there is no good formal theory to explain how it works?
It's scary, but to my thinking, inevitable. It was all well and good for the early atomic scientists to say that "Math is unreasonably effective at explaining Nature," [1] but our level of understanding of both mathematics and natural law is still superficial in several important areas. The universe doesn't owe us a formal theory of anything, much less everything.
It seems likely that we will soon start building -- and relying upon -- software that no human actually understands. The math community is already having to confront this dilemma to some extent, when an outlying figure like Mochizuki releases a massive work that takes months for anyone else to understand, much less prove or refute.
At some point we will have to give up, and let the machines maintain our models for us.
> what if there is no good formal theory to explain how it works?
We don't need a theory that is perfect. Each theory was partially wrong but still lets you make useful predictions about the world. We need useful models that let you reason about the world. All models are wrong, some are useful.
> What if intelligence, both animal and machine, is purely random trial and error and "this thing seems to work"?
Evolution could be just considered random trail and error. However until we reach the singularity, we need people to speed up the evolution process by adapting and remixing pieces that worked before. We need models for what each level does so have ideas of what to try for a new application.
I think the gp is referring to something a little more general. Let's pretend for a moment that our minds can be modeled by a formal system. Every thought has a chain of axioms grounding it. So, the scary part is, Godel showed us there are true things that can't be represented by a formal system.
Maybe the useful models exist, but we can't comprehend them, because they're true outside of the set of rules we happened to get built into our minds?
generally though, i'm on board with you. all models are wrong, some models are useful.
> Godel showed us there are true things that can't be represented by a formal system.
This is stretching things a bit. Specifically, it defines truth as 'does not lead to a contradiction in (some formal system that extends) Peano arithmetic'.
Then, as there are statements that are 'true' in this sense in such a system A but not probable by that system, there are 'unproveable truths'.
But is that satisfactory as a definition of truth? It used to be because we had hope for a complete and consistent formal system, which feel very truthy. When Godel proved that cannot exist, perhaps the conclusion is that formal systems aren't the 'base' for truth.
I am saying there is no connection between the two. You don't need a single complete formal system for a useful model of what is occurring. Godel was saying there can't be one true complete set of axioms for mathematics. This does not mean all mathematical models, theories are suddenly impossible.
Extrapolating this concept in various and biased directions... you could say that intelligence, behavior, and evolution (defined as physical iterative behavior across successive reproductive generations) are products of random changes/mutations, reinforced by gradient descent in the form of Survival of the Fittest.
Then maybe we should put more resources to study other approaches and higher-level concepts, like statistical learning research done by Vladimir Vapnik.
I don't believe that's true necessarily, but it will sure hamper the authors hopes.