I think there's more to it than that. Due to a combination of OCD/anxiety/chronic pain, a lot of the things I would enjoy doing aren't currently possible - I rarely leave the house, and I can't cook for myself, for instance. Despite that, each time I've stopped lengthy (>6 months) periods of opiate treatments, I've just stopped (no tapering) and had no ill effects (withdrawal etc.).
the trouble is when someone do not have that.