I was under the impression that if there were any aliens in universal paperclips that they were so inconsequential as to be classified as "matter". The adversarial "drifters" are drones from your swarm that have had their values drift since they themselves, of course, are hyper intelligent optimal energy matter converters.