Humans will always $hit where they eat, so I say, go full hog on the Universe. We are less than a rounding error in time and space in terms of the damage we can do to it. It's like peeing in the ocean.
The Earth is unique and precious, but the vast out there - we cannot do our worst even if we tried.
So does every other living thing, in fact. It's not like every animal and plant is a nature conservationist and humans are the mindless exploiters - it's actually the other way around. We talk about restraining ourselves because, unlike the rest of the nature, we can choose to be selfless, or at least thing longer-term and at ecosystem scale.
Also, you're absolutely correct. Earth is a gem. Everything else in space we know of is just rocks and deserts and clumps of gases. And there's so much of it that we aren't going to make a dent even if we rode the exponential growth for a while longer.
In the past I've also wondered if it wouldn't be too destructive to mine asteroids, but it makes no sense to worry about that while we continue to destroy our planet. This is the only place we can live, with vibrant, complex, diverse ecosystems, and we're causing a massive extinction event here, while making this one habitable planet less habitable for ourselves.
Meanwhile asteroids are completely uninhabited and dead, and there's millions of them. They have all the same minerals the Earth has, and often much closer to the surface. We could even just mine a few and easily replace all the destructive mining on Earth.
The Earth is unique and precious, but the vast out there - we cannot do our worst even if we tried.