What do you actually think the IR spectrum would look like given that it would still be very close to a star that could potentially already saturate the measurement?
So the telltale sign is a significant IR source with little to no visible light. Assuming the star is sufficiently encompassed by the swarm then it's still outputting energy, the swarm heats up and it needs to radiate that heat away. So a stellar level IR source that isn't visible is pretty weird.
And if you wonder if the Swarm uses all the light such that little escapes, that's the whole point. Think of it like a fog. A fog can block light with a sufficiently large number of tiny water droplets but it isn't strictly "solid".