This is the main reason I'd like to see asteroid mining take off. Let's tear some uninhabited asteroids apart, instead of the only planet we can live on.
Humans will always $hit where they eat, so I say, go full hog on the Universe. We are less than a rounding error in time and space in terms of the damage we can do to it. It's like peeing in the ocean.
The Earth is unique and precious, but the vast out there - we cannot do our worst even if we tried.
So does every other living thing, in fact. It's not like every animal and plant is a nature conservationist and humans are the mindless exploiters - it's actually the other way around. We talk about restraining ourselves because, unlike the rest of the nature, we can choose to be selfless, or at least thing longer-term and at ecosystem scale.
Also, you're absolutely correct. Earth is a gem. Everything else in space we know of is just rocks and deserts and clumps of gases. And there's so much of it that we aren't going to make a dent even if we rode the exponential growth for a while longer.
In the past I've also wondered if it wouldn't be too destructive to mine asteroids, but it makes no sense to worry about that while we continue to destroy our planet. This is the only place we can live, with vibrant, complex, diverse ecosystems, and we're causing a massive extinction event here, while making this one habitable planet less habitable for ourselves.
Meanwhile asteroids are completely uninhabited and dead, and there's millions of them. They have all the same minerals the Earth has, and often much closer to the surface. We could even just mine a few and easily replace all the destructive mining on Earth.
I just can't imagine mining the moon being in any way shape or form easier than mining the earth. The list of workplace hazards would be impressive. I imagine we're talking surface mines to keep the logistics manageable, but then:
1. How do you get astrominers there and back safely and on a regular basis whilst still keeping costs down?
2. What would it cost to keep your astrominers well fed and rested in a comfortable environment (warm, enough water, entertainment, etc)
3. How much more in bonuses (danger of hazard, long way from home, etc) do you even have to pay an astrominer vs a terraminer?
4. Most mines have some form of processing on site, e.g. break down rocks and sift through shit. How do you get those massive machines up there on the cheap and service them frequently on the cheap? Do we even have the machines that can work in those conditions?
5. Gravity is weak on the moon, I can imagine rocks of considerable mass flying or tumbling about being an issue. Sifting doesn't work as it does on earth without normal gravity and abundant water.
6. etc. etc. etc.?
I mean I think it'd be cool if we could pull it off as a civilisation but I just can't imagine how out there in scope and complexity moon mining would be.
But I think mining asteroids will be easier than mining the moon. Because on asteroids, everything is right at the surface. There's an asteroid that seems to be just the nickel-iron core of a former planet. That's more nickel and iron than you could ever hope to mine on Earth, and there's no planet around it.
Mining the earth is of course the easiest, because we're already there and there's no space travel or vacuum involved. But it's also very destructive, and once you tackle the space travel and zero-g mining issues, I think asteroid mining is going to be way more profitable. As well as saving our planet.
I can imagine dragging smaller asteroids and letting them "drop" into earth (e.g. attach massive parachutes and then guide them to land on some mining site. Totally sci-fi but I still find it less so than the moon. The reason I don't imagine robots mining the moon is that minerals, as you say, are not readily on the surface, nor is the moon mainly made up of a specific mineral.
I once heard that the one thing we might mine on the moon is He3, once/if we get to that level of fusion power. There's apparently a lot of it, and it's right at the surface.
But the advantage of asteroids is that everything is at the surface.