That's not entirely true, if your decrement results in a cascading free of an entire tree of objects you will pay for the deallocation time for that entire tree -- decrement, free, decrement children, free, etc..
And unless your RC algorithm is more clever than most that's going to be stop-the-world.