It's certainly a point of trust, but ultimately it's the miners who decide how Bitcoin works. So far this has predominantly worked by them using software from bitcoin.org and upgrading it relatively frequently, but this would probably change very quickly once they stop thinking of the bitcoin.org developers as working in their interest.
