An Unquiet Mind

Photo edit­ing will never be the same again.

In what has been described as a Google approach to under­stand­ing dig­i­tal pho­tos, researchers at Carnegie Mel­lon Uni­ver­sity have come up with a rad­i­cally inno­v­a­tive idea to add or remove con­tent from dig­i­tal pho­tos. The idea is that if you have enough infor­ma­tion at your hands, you can act smart with­out know­ing what you’re up to. Sounds famil­iar? Yes, any­one who has used Google knows that it doesn’t really know the ‘mean­ing’ of your search query, yet appears to give you ‘mean­ing­ful’ search results.

“Whether adding peo­ple or objects to a photo, or fill­ing holes in an edited photo, the sys­tems auto­mat­i­cally find images that match the con­text of the orig­i­nal photo so they blend real­is­ti­cally. Unlike tra­di­tional photo edit­ing, these results can be achieved rapidly by users with min­i­mal skills.”

Adding Con­tent — Photo Clip Art

This sys­tem “uses thou­sands of labeled images from a Web site called LabelMe as clip art that can be added to pho­tos. A photo show­ing a vacant street, for instance, might be pop­u­lated with images of peo­ple, vehi­cles and even park­ing meters.”

“Instead of try­ing to manip­u­late the object to change its ori­en­ta­tion, color dis­tri­b­u­tion, etc. to fit the new image, we sim­ply retrieve an object of a spec­i­fied class that has all the required prop­er­ties (cam­era pose, light­ing, res­o­lu­tion, etc) from our large object library. We present new auto­matic algo­rithms for improv­ing object seg­men­ta­tion and blend­ing, esti­mat­ing true 3D object size and ori­en­ta­tion, and esti­mat­ing scene light­ing con­di­tions. We also present an intu­itive user inter­face that makes object inser­tion fast and sim­ple even for the artis­ti­cally challenged.”

Edit­ing Con­tent — Scene Completion

This sys­tem “draws upon mil­lions of pho­tos from the Flickr web site to fill in holes in pho­tos. The sys­tem looks for image seg­ments that match the col­ors and tex­tures that sur­round the hole on the orig­i­nal photo. It also looks for image seg­ments that make sense con­tex­tu­ally ? in other words, it wouldn’t put an ele­phant in a sub­ur­ban back­yard or a boat in a desert.”

“The algo­rithm patches up holes in images by find­ing sim­i­lar image regions in the data­base that are not only seam­less but also seman­ti­cally valid. For many image com­ple­tion tasks the sys­tem is able to find sim­i­lar scenes which con­tain image frag­ments that will con­vinc­ingly com­plete the image. The algo­rithm is entirely data-driven, requir­ing no anno­ta­tions or label­ing by the user. Unlike exist­ing image com­ple­tion methods, the algo­rithm gen­er­ates a diverse set of image com­ple­tions and allows users to select among them.”

The bizarre sim­plic­ity of using these sys­tems momen­tar­ily shook me. Are there any secu­rity or other ram­i­fi­ca­tions? Will they make Photo ID fraud sim­pler? Will they make it eas­ier to cre­ate obscene pho­tos of celebri­ties or pri­vate per­sons you know? Will they evolve into film edit­ing tools? Only time will tell.

Photo Cred­its: CMU

Related posts:

1. Intel­li­gent Image Resizing
2. Evoca­tive Images from India
3. What Is Wrong With This Picture?