Along with flying skateboards and food replacement pills, the ability to enhance a highly pixelated image is something futuristic crime and sci-fi has promised us for decades. Now, it seems, there may be some progress on the latter front, with the nerds at Google Brain devising new software capable of making sense of tiny, pixelated images.
According to this new piece from Ars Technia UK, the secret is “a clever combination of two neural networks.” Here’s how they explain it:
The first part, the conditioning network, tries to map the the 8×8 source image against other high resolution images. It downsizes other high-res images to 8×8 and tries to make a match.
The second part, the prior network, uses an implementation of PixelCNN to try and add realistic high-resolution details to the 8×8 source image. Basically, the prior network ingests a large number of high-res real images—of celebrities and bedrooms in this case. Then, when the source image is upscaled, it tries to add new pixels that match what it “knows” about that class of image. For example, if there’s a brown pixel towards the top of the image, the prior network might identify that as an eyebrow: so, when the image is scaled up, it might fill in the gaps with an eyebrow-shaped collection of brown pixels.
To create the final super-resolution image, the outputs from the two neural networks are mashed together. The end result usually contains the plausible addition of new details.
So, no, the enhanced photo isn’t real, per se; rather, it’s a collection of details from similar classes of photos. But tests have yielded promise, with a modest, but still impressive group of human observers being “fooled” by the upscaled photos. Even if the technology is never able to upscale to perfection, it could still be incredibly helpful when trying to discern fuzzy blobs from grainy security footage, or specters from the night-vision footage ghost-hunting shows like to use as evidence.