Facebook AI Can Digitally Open Your Annoyingly Closed Eyes In Photos
Photography is a tricky thing, and we're not even talking about principles like the rule-of-thirds or using Bokeh effects to spice up a snapshot. No, the real trick is dealing with that subject that has a tendency to close his or her eyes. In my neck of the woods, we call that subject "Tammy," and if it's a group photo, there's a 99.9 percent chance (rough estimate) she'll close her eyes right as the photo is taken. This leads to laughs and groans, followed by a retake, but what if there was another way? Facebook is working on that very thing.
Over at Facebook Research, the dudes and dudettes in white coats (that's how we envision them, anyway) have developed an artificial intelligence (AI) scheme that can 'paint' over closed eyes so that they appear open. So rather than retaking a photo when someone like Tammy blinks at the most crucial moment of a photo shoot, it artificially generates an open set of eyeballs, and does it in realistic fashion (they don't look cartoony, in other words).
Reference images (left), closed eye photos (center), AI-generated image (right)
Facebook Research calls this process "Eye In-Painting with Exemplar Generative Adversarial Networks," which is also the name of the paper it published. We call it really freaking cool. Either way, it's some advanced stuff that tackles the challenge of generating human features, and that's not easy to do. As the researchers note in their paper, "humans ave very sensitive to small errors in facial structure," especially if the subject is well known to the person viewing the photo (like a friend or family member).
The use of generative adversarial networks (GANs) is the key. These are a "specific type of deep network that contain a learnable adversarial loss function represented by a descriminator network." Or in plain English, it's fancy technology (a machine learning technique) that is well suited to painting human images that look real.
"The generator network in a GAN learns to fill in missing regions of an image, and the discriminator network learns to judge the difference between in-painted and real images, and can take advantage of discontinuities between the in-painted and original regions. This forces the generator to produce in-painted results that smoothly transition into the
original photograph, directly sidestepping the need for any pixel blending," Facebook Research explains.
Faceook's AI scheme is generating open eyeballs out of thin air. Instead, it examines other images of a subject, ones where Tammy has her eyes open, and then uses that data to erase that untimely blink. The results can vary, but there are several examples of it working really well.
It will be interesting to see where this goes. We could see Facebook implementing a version of this tech for photos that are uploaded to the social network, as well as imaging programs adopting this or similar techniques. It could also mean less retakes for Tammy.
Images Sourced From Facebook Research
