You could be right, I don't know. Maybe the authors can tell us about their experience.
My intuition is that the dataset will be too imbalanced to learn anything useful. Even if you crawl only decensored images, the area you truly care about inpainting is still pretty small. If you don't focus on it somehow, it might learn how to inpaint anime-style geometry correctly (from the rest of the image) but produce "barbie doll" style anatomy.
The demo in the repo only shows inpainting of anime-style geometry. In the 4chan thread linked by nathansmith, someone tried to apply it to a mosaicked penis, but it disappeared (NSFW, probably: https://i.4cdn.org/g/1540580868203.png). When they used the mosaic decensoring mode instead, it worked slightly better (NSFW, definitely: https://imgur.com/a/PwAunbc/). Other people complained about missing genitals as well.
So I'm guessing that this model doesn't have any specific training for filling in only genitals. I think there are category-aware inpainting algorithms that could be used, but they'd need tagged data, as you say.
My intuition is that the dataset will be too imbalanced to learn anything useful. Even if you crawl only decensored images, the area you truly care about inpainting is still pretty small. If you don't focus on it somehow, it might learn how to inpaint anime-style geometry correctly (from the rest of the image) but produce "barbie doll" style anatomy.