Because it doesn't take infinite time from our perspective. The object falling into the black hole experiences strong time dilation due to the extreme gravitational field, but that doesn't affect a remote observer.
In the twin experiment, the observer sees his twin age slower, so it _does_ have a remote effect. From the perspective of the object falling into the black hole, the local conditions do not change, time is progressing normally etc.
