The examples on the landing page are not representative of the quality of the service.
Try it with internet photos of people. What it does is just morph the face a little: larger eyes, shorter forehead, elephant ears, narrow and taller mouth. The only thing it tries to generate is hair, and it fails horribly at it if you try with completely bald heads.
The Toonify produces avatars which match the source photo in terms of detail and feature correspondence. In that regard, at first take, the algorithm might appear to be working at a near pixel level of granularity. However, at least in the examples I tried, there is a loss of similarity and recognisability, to the extent that I don't think my friends would get that my avatar is me, and I don't get that feeling about the avatars generated from pictures of my friends. For that reason, I am wondering if the algorithm is based on feature recognition (eyes, nose, mouth, eyebrows etc), followed by a feature level of replacement/distortion rather than pixel level?
The celebrity examples might be a different case in terms of our recognition sentiment, because of their familiarity, and the juxtaposition of a number of phenotypically diverse celebrities in one set, which gives a correspondence in the diversity of the input set and the output set.
What modifications would be required to the algorithm to score consistently higher in facial recognition of the avatar by people who view the avatar cold without access to the original photo?
I'm asking rhetorically, of course. Such research might take a long time. Toonify looks like a good foundation for what might be an open ended project.
Specifically, there's this snippet:
"These StyleGAN face models can produce a huge diversity of faces and it’s actually possible to find basically any face inside the model. It’s actually a straight forward process to search for any image of a face in the model. So given an example image you want to find you can find a “code” (aka latent vector) which, when given as an input to the model, will produce an output which looks almost exactly like the face you’re looking for."
So it seems like they first generate an image that is similar to yours with one model, and then translate that too its more cartoony counterpart.
1. Clicks on the browse button to find a good selfie.
2. Uploads the pic to Toonify.
3. Clicks on 'Toonify' button:
> Failed: Didn't find any faces in the supplied image.
Ack what? Ok. The machine is still learning.
(P.S: I like the project though. Hope it will improve over time. But toonification of any pic without deep learning/machine learning thrown into the mix is already a solved problem, no?)
We're using dlib's frontal face detector for the face detection. It's ok, but certainly not state of the art, maybe in the future I'll find a better face detector...
The way I solved this for Faceweave is, if no face detected, let the operation proceed anyway. This gives the user the ability to line up the face themselves.
The logic is basically: ask dlib for face landmarks; if no landmarks, resize image to 1024x1024 and proceed; otherwise align face.
By default the portrait image will be rotated, causing the face detection to fail. I know this because I‘ve run in to the same problem before, had to correct it server side :)
A perfectly-normal program that might only load/run on a Google Cloud TPU or a Titan V, mind you; and probably one held together with tape and bailing wire (hardcoded paths, "works on my machine", an undocumented and unreproducable build env, etc.)
Y'know, the same reason that the "perfectly normal computer programs" that generate the data for scientific journal papers aren't usually published.
I understand that some networks might require the compute power that you mention, but this just seems like an average CNN mapping images to Disney characters. And I don't have much experience on that front, but like qayxc said, this should run fine on consumer hardware.
And yeah, many journal papers suck specifically for the reason that their results are not reproducible.
StyleGAN of this size runs fine on your average CPU.
I know that for a fact since I wrote paint.net plugin that does exactly that. Said plugin works perfectly fine with images of the size they use (512x512) and generates output (CPU-only, mind you) in less than 5 seconds on a dual-core laptop...
I remember reading reviews on this app (KPT Bryce [0]) in MacFormat in the 90s or so. The software seemed to use a really unique UI for development when compared to most creative Mac apps of that time.
Presumably there is some kind of bias in the training set causing this. It does seem subtle but noticeable, although I wouldn't say it's just the males.
Yeah, I was hoping it was going to be trained on photo -> cartoon of same subject by some known cartoonist. Could be a fun profile photo that makes it look like x paper's cartoonist has thought you significant enough to warrant a cartoon.
There are two plausible measures of success for something like this:
1) Whether the generated character is readily identifiable by an observer on its own.
2) Whether the generated character is a reasonable manifestation of the chosen art style.
I think it does a reasonable job of #2 and a mediocre job on average of #1. I think greater eyebrow variability would help some so that it doesn't look like all the women get "woman's pencil eyebrow" and all the men get "man's extremely bushy eyebrow".
I know HN is not normal, but I immediately tried my pets since I'm not happy to give out photos.
Didn't work, it can't find the face.
Otherwise it worked ok on the fascist leaders I tried. Black and white got color added. And one photo had an artefact put on top of both the original and the Toonify, a mirror of the top of the image to fill space perhaps?
Well done.
Also Startup idea, facial recognition for pets and sell the data to the Russians for addition tracking ability on people through proxies.
...wait what. I need to know more about these nightmares that involves whales. Did you stumble across a particularly vivid bit of whaler fan fiction that combined Melville with Lovecraft or something?
This works really well when it comes to making scary pictures of my three year old niece.
I look forward to seeing if she recognises herself and what she thinks. I don't believe any adult likes a cartoon version of themselves, rarely are critiques truly flattering, a true cartoon hurts!
I tried cartoonifying my niece's favourite cartoon characters. The program failed. So you can't toonify a toon.
I made this and can assure you I'm not doing anything with the images. This is just a silly side project of mine (along with Doron). I'm certainly not hoping to gain any money or data from it!
You should probably put this in the disclaimer. These days I assume all toys like this are data gathering tools in disguise. And I'm right more often than not.
Could he do that though? I mean, hypothetically. I'm based in the EU so GDPR applies, and a picture of my face is definitely personally identifying info. So, unless I clicked something saying I consent to the data be stored/sold, I assume it would be illegal to do so.
I'll be really impressed when one of these that actually retains some of the original facial features instead of just replacing them entirely. Cool side project though.
Or rather, the answer is "no" right now, but I will bet you $500 that it'll be "yes" within ... hm ... 2 years max. And if I lower my bet to $100, I'll do 1 year max.
Let's just say it's a very lucrative, very active area of research.
One observation: facial detection gives you landmarks, and those landmarks are essentially UVs for a texture. So if you do that process repeatedly, and rotate the face slightly using latent directions, then you can back-project the result onto a 2D texture.
Very very cool. I could see this being intergrated into something like fortnight in the very near future.
If I was a better coder I'd being trying to get an Epic Grant for this. I can't imagine getting a full blown model would take more than a few hundred thousand
this is cute, but to be honest, the celebrity ones are pretty bad - they don't really look like the person nor capture their spirit, with a few exceptions: Tim Allen and Gal Gadot are quite good and Will Smith is alright. Some especially bad ones: Leo DiCaprio, Barack Obama, Johnny Depp, Keanu Reeves, Idris Elba, Ryan Gosling, Robert DeNiro
Try it with internet photos of people. What it does is just morph the face a little: larger eyes, shorter forehead, elephant ears, narrow and taller mouth. The only thing it tries to generate is hair, and it fails horribly at it if you try with completely bald heads.