Legally, they could sue you for the same value that they were charging the customer for your work until the end of the contract, which is the money that they lost by the contractor not working.
In practice, they will probably not do that as the HR team is busy with other things and the lawsuit might be bad PR, although a hiring manager that wants to get back to you for doing that might have the HR department sue you, legally they could do that.
This seems like it's begging for a test case to strike this down on humanitarian grounds. I can't imagine any western judge of any serious level (possibly some county judge in West Texas or something) would take seriously the claim that this wasn't a violation of basic human rights (in US terms, the 14th amendment, though I assume similar principles apply elsewhere), and that such contracts were unenforceable. IANAL and I'm certainly not a limey lawyer, but our notions of justice on both sides of the pond share a lot of DNA.
In practice, they will probably not do that as the HR team is busy with other things and the lawsuit might be bad PR, although a hiring manager that wants to get back to you for doing that might have the HR department sue you, legally they could do that.