We are incrementally replacing human reasoning with computation in the present day. For instance, most static type checkers are weak but fast theorem provers, and type inference replaces some of the human reasoning involved.
Granted, static type checking is a very minor corner case, but manifold small incremental changes add up. It's untrue that human reasoning has never been replaced with automation, and it's untrue that human reasoning isn't currently in the process of being further replaced.
I agree that total replacement of human reasoning is likely any time soon. However, I'd argue that total replacement of human reasoning implies removal of human desires from the input. (What are programs, if not incredibly formal expressions of what humans desire computers to do? How can we divorce human desires from human reasoning about what is good/desirable?) Science fiction provides numerous examples of how a complete decoupling of computers from human desires can go terribly wrong.
Granted, static type checking is a very minor corner case, but manifold small incremental changes add up. It's untrue that human reasoning has never been replaced with automation, and it's untrue that human reasoning isn't currently in the process of being further replaced.
I agree that total replacement of human reasoning is likely any time soon. However, I'd argue that total replacement of human reasoning implies removal of human desires from the input. (What are programs, if not incredibly formal expressions of what humans desire computers to do? How can we divorce human desires from human reasoning about what is good/desirable?) Science fiction provides numerous examples of how a complete decoupling of computers from human desires can go terribly wrong.