If physics had somehow involved taking stuff from India they would have done it, which shows that the problem was with the attitude of the colonial era conquerors to the countries they took over rather than with archeology. In fact, a lot of the "they took the crown jewels" stuff has nothing to do with archeology at all.
