You don't need AGI to do serious damage. I think it's just easier for the layperson to reason about the ethics and implications of AGI than it is to reason about how various simpler ML models can be combined by bad actors to affect society and hurt the common good.