Sorry I didn't imply we didn't agree but that programmers were and are going to be impacted as much as writers for instance, yet I see an environment where AI is generally more accepted as a tool.
About your last point sometimes I think that in the future there will be models specifically distilling the climax of selected thinkers, so that not only their production will be preserved but maybe something more that is only implicitly contained in their output.
That's a good point: the greatest value that we can glean from one another is likely not epistemological "facts about the world", nor is it even the predictive models seen in science and higher brow social commentary, but in patterns of thinking. That alone is the infinite wellspring for achieving greater understanding, whether formalized with the scientific method or whether more loosely leveraged to succeed with a business endeavor.
Anecdotally, I met success in prompting GPT-3 to "mimic Stephen Pinker" when solving logical puzzles. Puzzles that it would initially fail, it would succeed attempting to mimic his language. GPT-3 seemed to have grokked the pattern of how Stephen Pinker thinks through problems, and it could leverage those patterns to improve its own reasoning. OpenAI o1 needs no such assistance, and I expect that o2 will fully supplant humans with its ability to reason.
It follows that all that we have to offer with our brightest minds will be exhausted, and we will be eclipsed in every conceivable way by our creation. It will mark the end of the Anthropocene; something that likely exceeds the headiest of Nick Bostom speculations will take its place.
It seems that this is coming in 2026 if not sooner, and Alignment is the only thing that ought occupy our minds: the question of whether we're creating something that will save us from ourselves, or whether all that we've built will culminate in something gross and final.
Looking around myself, however, I see impassioned "discourse" about immigration. The merits of DEI. Patriotism. Transgenderism. Religion. Copyright. Vast herds of dinosaurs preying upon one another, giving only idle attention to the glowing object in the sky. Is it an asteroid? Is it a UFO that is coming down to provide dinosaur healthcare? Nope, not even that level of thought is mustered. With 8 billion people on the planet, Utopia by Nick Bostrom hasn't even mustered 100 reviews on Amazon. On the advent of the defining moment of the universe itself, when virtually all that is imaginable is unlocked for us, our species' heads remains buried in the mud, gnawing at one another's filthy toes, and I'm alienated and disgusted.
The only glints of beauty I see in my fellow man are in those with minds which exceed a certain IQ threshold and cognitive flexibility, as well as in lesser minds which exhibit gentleness and humility. There is beauty there, and there is beauty in the staggering possibility of the universe itself. The rest is at best entomology, and I won't mourn its passing.
About your last point sometimes I think that in the future there will be models specifically distilling the climax of selected thinkers, so that not only their production will be preserved but maybe something more that is only implicitly contained in their output.