I don't think generative models are sufficient to model the kind of creativity that the author is thinking about. See this quote from the article:
The prevailing misconception is that by assuming that ‘the future will be like the past’, it can ‘derive’ (or ‘extrapolate’ or ‘generalise’) theories from repeated experiences by an alleged process called ‘induction’. But that is impossible.
He is looking for solutions outside of the distribution that has been previously observed.
What the author is looking for is a kind of induction. If I'm seeking a new theory of gravity, I'm not going to start looking at the colors of hats. Why not? Because I very strongly suspect it will lead absolutely nowhere. But if I think some ideas have potential and some others don't, isn't that a distribution of sorts?
Theories that have relevance or potential obey a certain distribution, and it is this distribution that you are trying to sample from. Sure, theories may not be directly derived from the extrapolation of sense data, but they are nonetheless derived from the extrapolation of theories in general. So what you're looking for is not a fundamentally new paradigm, it's more like an additional level of indirection. But there's no point to experiment with multiple induction levels if we can't even make a single one work well enough.
I agree that higher-order induction might very well suffice. But just to clarify the author's argument on the point, our rational justification for induction -- as you have just done -- cannot be inductive, because there cannot be an inductive justification for induction, and therefore induction cannot produce knowledge (which must be justified).
I don't think induction requires justification at all. I mean, if you do it and it works, great! If you do it and it doesn't work, well, what else were you supposed to do anyway? Ultimately, I use induction because I have no better idea, not because I think it necessarily works.
Also, generative or inductive mechanisms have a wider scope of application than prediction. They can be used to inspect your own belief systems and pinpoint inconsistencies: the easiest way to know if your model of the world is inconsistent is to generate ideas and examples that fit the model but trigger contradictions.
> if you do it and it works, great! If you do it and it doesn't work, well, what else were you supposed to do anyway?
But, as the author explains, epistemology doesn't work this way, and is certainly not inductive (maybe high-order inductive -- whatever that means). We don't treat physics as simply "something that works", but as knowledge based on assumptions (codified in symmetry laws) which are not inductive by any means. The laws of symmetry (assumptions, really) are a justification for induction, but can't be a result of induction alone. In fact, all of mathematics is a set of justifications for inductions that humans have developed.
Sorry but forming hypothesis based on observations, which is what assumptions means to me, is induction.
They are fallible so they need to be proven in order to be accepted. Usually that's done by proof of contradiction.
But you don't need to prove them to use them. Many people go around believing unproven theories.
In the sciences the preference is for verified theories(and theorems in math) and we prove them by deduction.
But it seems that you know that. I don't understand why you don't like induction in a general algorithm. We don't have to restrict ourselves to a single type of reasoning. Induction, deduction, abduction are all valid and used by humans for generating new knowledge.
I might be misunderstanding something in which case please correct me.
The article says that current research focuses on achieving intelligence by means of induction alone, but induction cannot explain all of intelligence, because we reason in ways that contradict induction (although maybe they're a result of higher-order induction).
The prevailing misconception is that by assuming that ‘the future will be like the past’, it can ‘derive’ (or ‘extrapolate’ or ‘generalise’) theories from repeated experiences by an alleged process called ‘induction’. But that is impossible.
He is looking for solutions outside of the distribution that has been previously observed.