1 gram at an appreciable amount of c is about as much energy as a nuke. Getting hit by a swarm of these while on a Sunday drive would fuck you up, no matter how powerful you are.
The real reason there isn’t a moon colony? Person gets fired and loses their shit, starts tossing 1-2 km sized rocks down the gravity well and … we all die.
Throwing shit in space is a sure fire way to piss off any species.
To throw a 100 meter rock that would wipe out NY city (and then some), we could do that with modern technology... assuming the rocket never had to slow down.
Same with a nuke or an engineered pathogen. Throwing rocks from the Moon is less practical though, people would see you setting up the universe's biggest launch pad or railgun for years. It would be the least stealthy, most telegraphed apocalypse possible.
