PHOTOSHOP.COM BLOG

See what's possible.

Meet the Team: Dan Goldman — Principal Scientist

Dan Goldman

When we first sneaked content-aware fill in Photoshop CS5, there were more than a few misbelievers. Gizmodo called it, “the craziest thing I’ve ever seen.” VentureBeat called it, “the power of the gods,” and suspected, “there just has to be sorcery involved.” ProLost called it, “actual witchcraft.” YouTube commenters said, “grab the torches.”

Of course, we were flattered. And actually, a little frightened. So here we are to tell you: It’s not witchcraft – it’s only science! Content-aware fill began in 2007 when Eli Shechtman joined the team and introduced a patch-based synthesis algorithm he had developed at the Weizmann Institute of Science. The speed issue was solved in our research labs in collaboration with our colleagues at Princeton University. PhD student Connelly Barnes (now a post-doc in our Cambridge, MA satellite lab) developed the initial prototype code during his Adobe Seattle internship in the summer of 2008. Our joint work became the foundation for a technical academic paper published at SIGGRAPH 2009, entitled “PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing.”

Previous state-of-the-art academic methods for filling missing pixels in images were impractical in the real world – taking hours to fill just a few hundred pixels! Our work showed how to accelerate the bottlenecks of these methods by factors of 20-100, which suddenly made it seem feasible to use them in real shipping software. Even before the paper was in print, we began discussing how we could bring it to Photoshop. Thanks to lots of hard work and tight collaboration between our lab and the Photoshop team, we were able to ship the feature the following spring – a pretty short time for a research paper to have an impact on the real world! In addition to shipping in Photoshop, that little summer internship project became the foundation for Connelly’s PhD thesis, which he successfully defended last spring.

Of course, any technology has its limits, and content-aware fill doesn’t always work as well as you (or we) would like. So we aren’t resting on our laurels. In CS6, we shipped content-aware move and content-aware patch tools, which add more user control to the underlying technique – a frequent user request. And we have continued to research new fundamental algorithms to improve quality and expand the capabilities in new directions. We presented some of that new work in two technical papers at SIGGRAPH and SIGGRAPH Asia last year, and you can bet we’ll be looking closely at how those technologies can fit into products like Photoshop in the years to come.

PatchMatch Goldman

Sometimes I am asked why a company like Adobe would publish some of its best work in the academic community, rather than keeping it a trade secret – like some other big software companies do. I can think of three important reasons: First, by encouraging publication, we make it attractive for the best minds in the business to come work in our labs – we count several former and current University professors among our ranks. Second, our researchers draw on the wealth of knowledge in the academic community as well – a great deal of our research is done in collaboration with graduate students like Connelly. And third, the rigorous demands of peer review keep us motivated to try truly new things – rather than being content to simply do all the old things better.

Like the Wizard of Oz, I’ve pulled aside the curtain to reveal the mere mortals sitting behind it. But undoubtedly some people will still call it sorcery. As Arthur C. Clarke said, “Any sufficiently advanced technology is indistinguishable from magic.” I hope we can bring you more magic in the years to come!

To learn more about other work I’ve been doing at Adobe, visit my webpage.

TAGGED AS: , , , , , , , , , ,

Adding a fourth reason: publication keeps you sharp. Sooner or later some competitor will find a similar solution that is better. So, better not rest on your laurels and ‘deliver, publish and move on’. (preferably in that order :-) )

Kudos all around! Everything you said in this post warms the cockles of my heart (as the saying goes). Keep up the GREAT work.

Allen

I just purchased an HP ENVY dv7 Notebook PC with Windows 8 and IE9. Then I got the Microsoft 2010 and installed it on this PC, then I downloaded Adobe. Now I want a program to see my pictures, and I have an older copy of Photoshop, on another computer, with pictures dated back to 1945 on it. I want to transfer those pictures to this PC and I want to know how to do it, and what Adobe Photoshop should I purchase from you. There are too many Photoshop programs and I am confused. Which one should I buy?

    Get the creative cloud subscription. It has the latest tools you will need including Lightroom

i thinking write very good!

i thinking write very good!

gsdkf,’askg/falsnf/dlkmafnjknam,,f;a,sd

Adding a fourth reason: publication keeps you sharp. Sooner or later some competitor will find a similar solution that is better

cs5 is good realy i handel this softwear i like us

photoshop

The example, you have shown is quite an eye opener. I never knew “Content aware” can do this much amount of manipulation. this looks awesome. Dont know how to gain this expertise. But I am hopeful.

nice photo shots. more to come and share to view.

publication keeps you sharp. Sooner or later some competitor will find a similar solution that is better. So

great article and nice photo, thanks for sharing. Keep up with good work

or later some competitor will find a similar solution that is bett

or lasdompetitor will find a similar solution that is bett

Very awesome post. And I am glad, that you aren’t resting on our laurels and each new version is better. Thank you.

Well, I guess the latest version, the Photoshop CC, takes it to an all new level entirely!

or later some competitor will find a similar solution that is bett

COMMENTS

(required)

(required)


7 + four =