Photoswap, not Photoshop

Pho­to edit­ing will nev­er be the same again.

In what has been described as a Google approach to under­stand­ing dig­i­tal pho­tos, researchers at Carnegie Mel­lon Uni­ver­si­ty have come up with a rad­i­cal­ly inno­v­a­tive idea to add or remove con­tent from dig­i­tal pho­tos. The idea is that if you have enough infor­ma­tion at your hands, you can act smart with­out know­ing what you’re up to. Sounds famil­iar? Yes, any­one who has used Google knows that it doesn’t real­ly know the ‘mean­ing’ of your search query, yet appears to give you ‘mean­ing­ful’ search results.

Whether adding peo­ple or objects to a pho­to, or fill­ing holes in an edit­ed pho­to, the sys­tems auto­mat­i­cal­ly find images that match the con­text of the orig­i­nal pho­to so they blend real­is­ti­cal­ly. Unlike tra­di­tion­al pho­to edit­ing, these results can be achieved rapid­ly by users with min­i­mal skills.”

Adding Content — Photo Clip Art

This sys­tem “uses thou­sands of labeled images from a Web site called LabelMe as clip art that can be added to pho­tos. A pho­to show­ing a vacant street, for instance, might be pop­u­lat­ed with images of peo­ple, vehi­cles and even park­ing meters.”

abbeyBackground abbeyComposite

Instead of try­ing to manip­u­late the object to change its ori­en­ta­tion, col­or dis­tri­b­u­tion, etc. to fit the new image, we sim­ply retrieve an object of a spec­i­fied class that has all the required prop­er­ties (cam­era pose, light­ing, res­o­lu­tion, etc) from our large object library. We present new auto­mat­ic algo­rithms for improv­ing object seg­men­ta­tion and blend­ing, esti­mat­ing true 3D object size and ori­en­ta­tion, and esti­mat­ing scene light­ing con­di­tions. We also present an intu­itive user inter­face that makes object inser­tion fast and sim­ple even for the artis­ti­cal­ly chal­lenged.”

Editing Content — Scene Completion

This sys­tem “draws upon mil­lions of pho­tos from the Flickr web site to fill in holes in pho­tos. The sys­tem looks for image seg­ments that match the col­ors and tex­tures that sur­round the hole on the orig­i­nal pho­to. It also looks for image seg­ments that make sense con­tex­tu­al­ly ? in oth­er words, it wouldn’t put an ele­phant in a sub­ur­ban back­yard or a boat in a desert.”

NoBoatsWeb SailboatsWeb1

The algo­rithm patch­es up holes in images by find­ing sim­i­lar image regions in the data­base that are not only seam­less but also seman­ti­cal­ly valid. For many image com­ple­tion tasks the sys­tem is able to find sim­i­lar scenes which con­tain image frag­ments that will con­vinc­ing­ly com­plete the image. The algo­rithm is entire­ly data-dri­ven, requir­ing no anno­ta­tions or label­ing by the user. Unlike exist­ing image com­ple­tion meth­ods, the algo­rithm gen­er­ates a diverse set of image com­ple­tions and allows users to select among them.”

The bizarre sim­plic­i­ty of using these sys­tems momen­tar­i­ly shook me. Are there any secu­ri­ty or oth­er ram­i­fi­ca­tions? Will they make Pho­to ID fraud sim­pler? Will they make it eas­i­er to cre­ate obscene pho­tos of celebri­ties or pri­vate per­sons you know? Will they evolve into film edit­ing tools? Only time will tell.

Pho­to Cred­its: CMU

This entry was posted in photography, Science and tagged . Bookmark the permalink.