The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years.
But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.
> The median person in field thinks 5-10%, not 2-5%
The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages.
> median timelines are shorter than 40 years.
The median person suggested a 50% chance in 39 years.
> since unaligned ASI is the default case
I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.
I work in the space (doing tech stuff that isn't direct research). The best argument I've seen for alignment by default is something like "morality comes from training data, and therefore the fact that LLM training data sets contain human moral intuitions will mean that an ASI stemming from such a training regime will share enough human values" (Quintin Pope believes something like this, as far as I can tell), which is deeply unconvincing, since it contradicts the evidence we _do_ have from human & animal value formation.
Happy to entertain other arguments that alignment-by-default is reasonable; most arguments I've seen are much worse than that one. I haven't seen many people make an active case for alignment-by-default, so much as leave open a whole bunch of swath of uncertainty for unknown unknowns.
But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.