Gas station without pumps

2016 September 6

srez: Image super-resolution through deep learning

Filed under: Uncategorized — gasstationwithoutpumps @ 09:28
Tags: ,

I found an interesting project that uses deep learning to reconstruct faces from very-low-resolution images at GitHub – david-gpu/srez: Image super-resolution through deep learning.  Here is what the github main page for the project says about it:

srez
Image super-resolution through deep learning.
This project uses deep learning to upscale 16×16 images by a 4x factor. The resulting 64×64 images display sharp features that are plausible based on the dataset that was used to train the neural net.

Here’s an random, non cherry-picked, example of what this network can do.

From left to right, the first column is the 16×16 input image, the second one is what you would get from a standard bicubic interpolation, the third is the output generated by the neural net, and on the right is the ground truth.

As you can see, the network is able to produce a very plausible reconstruction of the original face. As the dataset is mainly composed of well-illuminated faces looking straight ahead, the reconstruction is poorer when the face is at an angle, poorly illuminated, or partially occluded by eyeglasses or hands.This particular example was produced after training the network for 3 hours on a GTX 1080 GPU, equivalent to 130,000 batches or about 10 epochs.

The results are pretty impressive, except for the low-angle profile and the glasses.  The glasses problem could undoubtedly be fixed with a slightly larger range of training images (ones that included a large number of people wearing different styles of glasses).  Generalizing to cover the low-angle photo would require a much larger training set, with a lot of different camera angles for a lot of different faces.  There also seems to be some difficulty with skin tone—all the reconstructions are coming out more tanned than the originals.  I suspect that for dark-skinned people the reconstruction will lighten their skin—moving everyone closer to average for the training set.

I suspect that images that aren’t so carefully framed and cropped would also cause problems, as would ones that have some form of blurring different from the down-sampling used to create these images.  The results are impressive enough, though, that they almost make me want to go into research in deep learning.

2 Comments »

  1. I find that study interesting because I have some digital images from antiquity that are low resolution. Just doubling them without the bicubic errors (or going to 4x and than sampling down to 2x) would be a vast improvement over what has been preserved, but that would require a much bigger training program. This project is probably just for the security world.

    Comment by CCPhysicist — 2016 September 6 @ 11:04 | Reply

    • The deep-learning approach is not going to improve arbitrary images—it is more an attempt to add knowledge of what faces look like to low-resolution face images. People do this themselves—it is part of how we recognize faces even in random blobs and swirls. It might be interesting to see what the network would provide given a 16×16 rendition of a face-like image (a cloud or piece of toast that people see as looking like a face). I suspect that even a “smiley face” would get converted into something that looks more like a human face.

      Note that the input to the neural network here is very small (a 16×16 image)—the training set required probably grows exponentially with the size of the image. A lot would need to be done to the system even to make it useful for security work, as cameras are not likely to provide precisely sized images that are aligned the way the neural net expects.

      Comment by gasstationwithoutpumps — 2016 September 6 @ 11:34 | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: