This is probably less a critique of your project and maybe more a celebration of cubism: The clock in the Picasso portrait is not "harmonized" in terms of cubist motifs at play in the composition. The contours appear merely dilapidated. Maybe this passes for someone with little concern for painting or art history but it was like a sort thumb for me. Maybe this is a good study reference on the gap between pattern recognition and human perception. Maybe not. I'd be interested in other thoughts.
Plenty of them fail to stylistically harmonize even if they visually harmonize. Of course, in some cases, it's debateable whether stylistic harmonization is even possible at the human semantic level. The bicyclist on the bridge in the scream painting was quite bad, and the Eiffel Tower in the cityscape was noticeably degraded by the fact the painting didn't have the colors that the artist would probably have actually used, so it's rather bluish.
What's more impressive is that several of them work quite well, and barring anachronism, at the sizes given in the samples you might not know what was added if they didn't point it out. The park bench I found particularly impressive, for instance. I suspect if we increased the resolution it would break down, but at this size it works pretty well.
I've measured the "realism" of graphics in terms of resolution for many years now, because nowadays we generally have realistic graphics, as long as the resolution is small enough. (Compare with, say, an NES game, where no matter what you do to that image nobody will ever find it realistic.) There's a lot of techniques/artists/etc. I've seen that are perfectly realistic looking at 320x200 (say), but couldn't sustain the illusion of realism at 1920x1080. This way I get a reasonable metric of whether or not graphics are "photorealistic", or in this case, match the style of the painting.
It would be interesting to see them do a higher resolution version, or if they have them. It's possible it would indeed fit right in even so; now that I think about it the microscale of these paintings are probably very stereotypical and the algorithm might be able to reproduce them well. It would be especially interesting to see if it would correctly reproduce brush strokes, which have a lot of context to them. Given that "deep learning" can do things like reproduce the structure of a TV script accurately, it doesn't seem out of the question.
I think the gap in this case is not as wide as you think given that it is only looking at this one image. Had the network been trained for one art style in particular it may be able to pick up on that detail similar to if a person had studied many paintings in that style.
Way cool. I would personally mistake many of the results for a real painting if I didn't have the original in front of me.
Getting the colors right is half the problem, and I guess the histogram loss function I saw mentioned at the bottom of the page does that. A couple of the results had strange looking color transformations though - the little girl whose red shirt turns blue, even though the bed has some red in it, and the red rose that turns yellow even though there are orange flowers in the background. So it's not always choosing colors that are the closest to the source while being available in the target's palette. Anyone know why the colors sometimes go so far of course?
Also, this is one of the first style transfer papers I've seen that has a pretty obvious built-in and seemingly plausible business idea. I'm sure poster stores in malls everywhere could sell versions of your favorite painting or poster with your or your own face or something else of your choice added to the composition. It's like the new version of the picture board painting with face cutouts, but way better.
With these examples it always feels like a very good usecase would be automatic rotoscoping. A movie like "A Scanner Darkly" looks like it could've used these techniques, instead of having all the original footage redrawn manually.
Why would they need/want to add the Disclaimer? ("This software is published for academic and non-commercial use only.") By fear of some patent violation? Or do they plan for commercial use (e.g. licensing to Photoshop) and this way they keep the license non-OSS (a.k.a. the "share source" euphemism https://en.wikipedia.org/wiki/Shared_source)? Is it some kind of a well known legalese in academic circles?
They use VGG-19 pretrained weights which are derived from ImageNet which consists of mostly copyrighted images. It is currently untested whether this is a legal problem or not.
Wow, ok, this is seriously insightful and mindblowing for me! I'd never think of such a danger vector, but with your hint I can now totally understand the reason for caution. However painful it feels. Thanks for the reply! :)
Maybe they simply don't have the rights to the whole source code, but were allowed to make it public if they added that disclaimer. Usually the university will have the ownership of code produced in academia and if you find a commercial model based on their research code, they most likely want some money from that.
Consider for example a picture of a street and a rendered car overlaying it. It would be interesting to see how well the technology bakes in the car so that it matches the environment. This would go in direction of realistic rendering.
This is really amazing. Very impressive. Would love to see this published on the web somewhere to try it out quickly without requiring folks to have the necessary components to build it.
I guess that would be more difficult, because there has to be coherence between subsequent frames (i.e., it has to avoid wildly changing random effects between frames).
This is probably less a critique of your project and maybe more a celebration of cubism: The clock in the Picasso portrait is not "harmonized" in terms of cubist motifs at play in the composition. The contours appear merely dilapidated. Maybe this passes for someone with little concern for painting or art history but it was like a sort thumb for me. Maybe this is a good study reference on the gap between pattern recognition and human perception. Maybe not. I'd be interested in other thoughts.