That would certainly work for relatively small or close-knit groups where it is easier to collectively decide to move. I would expect that impact factor has a significant amount of gravity associated with it so even if the leaders in a field decided to move to open-access journals unless the (up and coming) majority also made the same move then the impact-factors of both journals may not change much.
