When we first sneaked content-aware fill in Photoshop CS5, there were more than a few misbelievers. Gizmodo called it, “the craziest thing I’ve ever seen.” VentureBeat called it, “the power of the gods,” and suspected, “there just has to be sorcery involved.” ProLost called it, “actual witchcraft.” YouTube commenters said, “grab the torches.”
Of course, we were flattered. And actually, a little frightened. So here we are to tell you: It’s not witchcraft – it’s only science! Content-aware fill began in 2007 when Eli Shechtman joined the team and introduced a patch-based synthesis algorithm he had developed at the Weizmann Institute of Science. The speed issue was solved in our research labs in collaboration with our colleagues at Princeton University. PhD student Connelly Barnes (now a post-doc in our Cambridge, MA satellite lab) developed the initial prototype code during his Adobe Seattle internship in the summer of 2008. Our joint work became the foundation for a technical academic paper published at SIGGRAPH 2009, entitled “PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing.”
Previous state-of-the-art academic methods for filling missing pixels in images were impractical in the real world – taking hours to fill just a few hundred pixels! Our work showed how to accelerate the bottlenecks of these methods by factors of 20-100, which suddenly made it seem feasible to use them in real shipping software. Even before the paper was in print, we began discussing how we could bring it to Photoshop. Thanks to lots of hard work and tight collaboration between our lab and the Photoshop team, we were able to ship the feature the following spring – a pretty short time for a research paper to have an impact on the real world! In addition to shipping in Photoshop, that little summer internship project became the foundation for Connelly’s PhD thesis, which he successfully defended last spring.
Of course, any technology has its limits, and content-aware fill doesn’t always work as well as you (or we) would like. So we aren’t resting on our laurels. In CS6, we shipped content-aware move and content-aware patch tools, which add more user control to the underlying technique – a frequent user request. And we have continued to research new fundamental algorithms to improve quality and expand the capabilities in new directions. We presented some of that new work in two technical papers at SIGGRAPH and SIGGRAPH Asia last year, and you can bet we’ll be looking closely at how those technologies can fit into products like Photoshop in the years to come.
Sometimes I am asked why a company like Adobe would publish some of its best work in the academic community, rather than keeping it a trade secret – like some other big software companies do. I can think of three important reasons: First, by encouraging publication, we make it attractive for the best minds in the business to come work in our labs – we count several former and current University professors among our ranks. Second, our researchers draw on the wealth of knowledge in the academic community as well – a great deal of our research is done in collaboration with graduate students like Connelly. And third, the rigorous demands of peer review keep us motivated to try truly new things – rather than being content to simply do all the old things better.
Like the Wizard of Oz, I’ve pulled aside the curtain to reveal the mere mortals sitting behind it. But undoubtedly some people will still call it sorcery. As Arthur C. Clarke said, “Any sufficiently advanced technology is indistinguishable from magic.” I hope we can bring you more magic in the years to come!
To learn more about other work I’ve been doing at Adobe, visit my webpage.