Peek Behind the Sneaks: Hand-Drawn Art in Three Dimensions

There’s something uniquely expressive about hand-drawn art.  The variations in pencil pressure, the length and thickness of a brush stroke or even the subtle imperfections in a hand-drawn line all combine to aid in the expressive intent of the artist and result in a certain organic, human quality in the finished product.

Digital drawing tools, for all their benefits and efficiencies have often struggled to incorporate these very human, hand-drawn qualities into the design process.  That’s been especially true for artists and animators who need to work with 3D objects and design tools.  Existing techniques for applying textures, or simulated brush and pencil marks to 3D objects are often too perfect, too uniform, too standard or even too repetitive to capture the unique qualities or style of an individual artist.

But what if 3D design tools could be made as simple as drawing pen to paper?  That was the goal of Project StyLit — a new technology shown at MAX 2016 for incorporating an artist’s hand-drawn style into the 3D design process.

Paul Asente applies a hand-drawn texture to a 3D model of a dinosaur in real time at Adobe MAX 2016

“Rather than stylizing a 3D object based on a specific technique and its associated algorithm — such as cross-hatch or oil paint — we wanted to stylize by example,” explains Paul Asente, senior principal scientist at Adobe.

Project StyLit was born out of collaboration between Adobe Research and the Department of Computer Graphics and Interaction at the Czech Technical University in Prague, including Daniel Sýkora, associate professor, and Mike Lukáč, who studied there, and is now a full-time research scientist at Adobe.

Although the technology behind Project StyLit is advanced, the result is deceptively simple.  Simply draw a ball on a piece of paper, in whatever hand drawn medium and style you desire, and Project StyLit will apply that style and texture to 3D objects.  Add a shadow, light reflection to the ball, or contrasting colors, and Project StyLit will include those elements in the illumination effects as well.

Although easy and extraordinarily practical for an artist to use, the technology behind Project StyLit is rather complex.  Taking a hand drawn image, capturing all of its organic qualities, and then applying them to a 3D model still has the same challenge of making things too perfect, or too repetitive.

“Any example-based texture synthesis runs into problems where what the objective function says is optimal and what actually looks natural are very different things.  We didn’t just want to sample one part of the image that was mathematically convenient and repeat it.  You might end up overusing a part of the sample, resulting in a finished product that looks repetitive and artificial,” notes Mike.  “What we did was recast the optimization problem so that we are always careful to use the entire input.  Everything is there for a reason.”

That’s why the ball shape the artist draws in their example is so important.  It’s simple enough that almost anybody can draw it.  It still provides a lot of freedom for the artist to employ their unique palette and personal style, but it also incorporates all the information Project StyLit needs to interpret lighting effects and texture accurately.

A few examples of the style from a ball drawn applied to different 3D models

“All 3D design programs have tools to render the light path expression of a scene appropriately: highlights, shadows, the way light bounces around and reflects … combined they represent the entire set of lighting information for a scene.  By using the ball and table example we know the illumination angles of the scene and exactly what part of the artists drawing corresponds to these effects,” adds Paul.

“It’s incredibly fun to play with.  Everywhere we go to demonstrate this, people line up. Pretty much anybody can do it, and there’s this instant reward of seeing their drawing style transferred onto another scene in real time.”

In the future, the Project StyLit team plans to see just how far they can push this technique.  “Today we’re limited to just a few scenes.  The Holy Grail would be to create a complicated theme with multiple objects and textures, and transfer them to another scene.  It’s something we’re working on, but not quite there yet,” Mike notes.

In the meantime, curious artists and animators are invited to try the Project StyLit demo by visiting here.  You’ll need a recent PC with a relatively powerful GPU, Windows 10/8/7 and a camera with DirectShow support.

This story is part of a series that will give you a closer look at the people and technology that were showcased as part of Adobe Sneaks. Watch other Sneaks from this year’s MAX here and read other Peek Behind the Sneaks stories here.