I'm glad I clicked this because it's awesome, but it wasn't at all what I was expecting. Maybe change the title to PIFuHD: High-Resolution 3D Human Digitization?
So I am building a farm robot, and for various reasons I think having a 3D reconstruction of the plants would be useful. I’m hoping that this technique might be useful for reconstructing models of plants from a small number of images. The model doesn’t need to be perfect, just close. I think this will help in doing reinforcement learning on simulated data, among other things.
Great research coming out of Facebook Reality Labs!
I’m guessing this was trained on images and their 3D scanned models so it kinda fits the unseen parts of the body.
I can nearly guarantee you that if you try to to use an image with something on the back it won’t be able to fit it, backpacks don’t work and probably something like the Victoria’s Secret angel wings won’t either.
You can also see where it really fudges things like the shoes on some of their examples, their examples are also the best case scenario (similar to the StyleGAN2 results) and not necessarily representative of what the model achieves regularly.
I’ve tried using some pictures of my partner in order to 3D print her but it didn’t really work well I’m guessing you need a really clean background.
It is not doing photogrammetry and it works on a single image. It has learned what people look like (and training may have used photogrammetry in some way) and it can imagine a good 3D model from even a single image. The last line of the abstract:
“We demonstrate that our approach significantly outperforms existing state-of-the-art techniques on single image human shape reconstruction by fully leveraging 1k-resolution input images.”
It’s not about being “accurate” as in representing exactly what it can’t see but rather be “authentic”, since you don’t care about things like patterns or logos clothes aren’t that hard to figure out and so are body shapes.
Backpacks, and other attachments that are on the back or occluded ofc can’t be predicted or fitted with-in the model to represent their real life counterparts well not unless you train the model on every possible backpack.
That said training this model on people’s photographs and their 3D scans and then possibly on a dataset of clothes and accessories from big brands might actually allow you to much more accurately predict the unseen parts of the image by not only filling the gaps based on predictions but by actually specifically matching the items of clothing and accessories to draw additional information that way.
Most fashion brands already have detailed images of their products from multiple angles and even have 3D views of many items.
Combine that dataset with this model and you’ll probably be able to increase the accuracy of these models considerably.
