Photo editing will never be the same again.
In what has been described as a Google approach to understanding digital photos, researchers at Carnegie Mellon University have come up with a radically innovative idea to add or remove content from digital photos. The idea is that if you have enough information at your hands, you can act smart without knowing what you’re up to. Sounds familiar? Yes, anyone who has used Google knows that it doesn’t really know the ‘meaning’ of your search query, yet appears to give you ‘meaningful’ search results.
“Whether adding people or objects to a photo, or filling holes in an edited photo, the systems automatically find images that match the context of the original photo so they blend realistically. Unlike traditional photo editing, these results can be achieved rapidly by users with minimal skills.”
Adding Content — Photo Clip Art
This system “uses thousands of labeled images from a Web site called LabelMe as clip art that can be added to photos. A photo showing a vacant street, for instance, might be populated with images of people, vehicles and even parking meters.”
“Instead of trying to manipulate the object to change its orientation, color distribution, etc. to fit the new image, we simply retrieve an object of a specified class that has all the required properties (camera pose, lighting, resolution, etc) from our large object library. We present new automatic algorithms for improving object segmentation and blending, estimating true 3D object size and orientation, and estimating scene lighting conditions. We also present an intuitive user interface that makes object insertion fast and simple even for the artistically challenged.”
Editing Content — Scene Completion
This system “draws upon millions of photos from the Flickr web site to fill in holes in photos. The system looks for image segments that match the colors and textures that surround the hole on the original photo. It also looks for image segments that make sense contextually ? in other words, it wouldn’t put an elephant in a suburban backyard or a boat in a desert.”
“The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. For many image completion tasks the system is able to find similar scenes which contain image fragments that will convincingly complete the image. The algorithm is entirely data-driven, requiring no annotations or labeling by the user. Unlike existing image completion methods, the algorithm generates a diverse set of image completions and allows users to select among them.”
The bizarre simplicity of using these systems momentarily shook me. Are there any security or other ramifications? Will they make Photo ID fraud simpler? Will they make it easier to create obscene photos of celebrities or private persons you know? Will they evolve into film editing tools? Only time will tell.
Photo Credits: CMU