Isn't lower class work interchangeable to some extent ?
Less illegals to do construction means a larger portion of construction it will be done by legal employees, which reduces the available pool for manufacturing.
Construction is not particularly more interchangeable than programming. Inexperienced workers can do some tasks, which have been prepped by more experienced workers. A worker operating out of their depth will cause problems that cost 10x to fix.
In other words: you can fill part of your staff with interchangeable workers, but not all of it.