Kevin Goldsmith

May 21, 2009

Adobe Romania Sponsors Pixel Bender contest for BEST

Mihai Pricope, a platform evangelist from Adobe Romania posted about this on his blog.

Adobe Romania helped sponsor the 2009 Bucharest Best Engineering Competition for the Board of European Students of Technology. Miti taught introduced the students to Pixel Bender and below you see what they produced two hours later. He has a zip of the pixel bender kernels on his blog.

May 14, 2009

Editing Pixel Bender in Eclipse with PBDT

I've often been asked if Adobe was going to do an eclipse plug-in for Pixel Bender. It's been something that we definitely wanted to do, but I wasn't sure when we would get to it. Joa Ebert decided to take advantage of the PBUtil command-line application that we ship with the Pixel Bender Toolkit SDK to create his own eclipse plug-in, PBDT. He even came up with a way to get Pixel Bender development working with Linux (another often requested feature).

I've been playing with it a bit (it integrates with Flex and FDT pretty well) and it is a cool alternative way to do Pixel Bender development.

Thanks to Joa for his contribution to the community!

April 19, 2009

video of my talk to the Seattle Flash User's group last June

I spoke to the Seattle Flash User's group last June, and I hadn't realized that they'd posted video of it until just now...

Live Videos by Ustream

April 3, 2009

Pixel Bender Toolkit v1.5 released!

The newest version of the Pixel Bender Toolkit has been released on Adobe Labs. This is pre-release 6, but it is also version 1.5. This new version includes the ability to edit, compile and run Pixel Bender Graphs (supported in Photoshop and After Effects). It also has a number of bug fixes, specifically in areas around PBJ generation.

You can get it from the Pixel Bender Technology area on Adobe Labs, and please let us know about any issues or suggestions for the next release in our forums, our Adobe Group or on twitter.

April 2, 2009

More Cool Pixel Bender Experiments

Ralph Hauwert wanted to see what he could do with Pixel Bender and Alchemy to get the most possible performance from the Flash Player. He created this experiment showing a 3D particle system using over 300,000 particles rendered in 3D in real time. More info and the demo on his blog.

Rob Skelly has a post on his blog showing how to use Pixel Bender for general purpose math. "So, does using Pixel Bender for math really help? Yes! In my simple tests, using Pixel Bender for collections of Numbers larger than a few hundred is at least 3x as fast and much better still for larger sets and more complex calculations. There is some overhead associated with setting up the shader objects, but depending on what you’re doing, the savings are definitely worth it."

Tom Beddard (the creator of the Esher Droste effect and the sweet Mandelbrot renderer in Pixel Bender) has dropped some new Pixel Bender coolness with a landscape raytracer! This is pretty neat and he also includes the source and some background on how he created it. Very nice.

Frank Reitberger has created a very trippy sphere generator with Pixel Bender and also includes the sources and a demo on his blog.

Aaron Conran decided to see what he could do using Pixel Bender, AIR and Explorer. The result is the Pixel Bender Explorer application. It also allows you to reference pbj files as effects in the wild to play with them. This app is a ton of fun.

David Lanaert has been experimenting with physical simulation in Flash using Alchemy and Pixel Bender. He created an web app called Farbe that lets you do watercolor simulation in Flash using Pixel Bender. It is really impressive, especially the performance he achieved. Really inspiring to play around with!

Thanks to everybody in the community who continues to push the bounaries using Pixel Bender. We have some very exciting stuff that we're working on that will help you create a whole new generation of awesomeness! Stay tuned!

March 2, 2009

Open position on the AIF team

My team is hiring! We're looking for a white-box QE with C++ and scripting experience.

For more information: http://cooljobs.adobe.com/frameset.html?goto=er-viewjob&refnode=440636

February 13, 2009

NVidia's Pixel Bender contest update: You've got two more weeks!

The NVidia Pixel Bender contest was extended to the end of February. You've got two more weeks to submit your filters for some pretty sweet prizes!

More info in my previous post or go straight to the contest page on the NVidia website!

Great Pixel Bender instructional videos on Adobe TV

Ryan Taylor presented at the Adobe MAX conference in San Francisco last November on Pixel Bender. I didn't get to see it because I was teaching the tutorial at the same time, so I was really looking forward to watching the video. It is finally up! I'm really impressed because Ryan got a lot of the info at the very last minutes (it was changing right up to MAX) and still put together a great talk. There are a couple of minor errors in it on some of the background, but nothing major. Unfortunately, the recording gets cut off, but it is still a good talk if you want to get a good view of Pixel Bender for CS4.

Lee Brimelow (who also created the great Pixel Bender tutorials at gotoandlearn.com) has a good shorty tutorial as part of the ADC Presents series and does a really nice walkthrough of a Pixel Bender kernel. It's basically the 10 minute condensed version of the Pixel Bender tutorials we did at MAX.

Paul from Flash Downunder also does a great tutorial on using Pixel Bender in Flash as well that goes into a bit more depth.

January 16, 2009

Some cool experiments with Pixel Bender

Since the first posting of the Pixel Bender toolkit, developers have been pushing the boundaries, moving beyond image and video processing filters. Some really cool things have been posted, so I thought I would call attention to some of the things I've seen. There is a ton more out there to check out, this is just some of the things I've seen posted lately.

David Lenaerts has created a smoke/fluids simulation in Pixel Bender utilizing 49 Pixel Bender kernels per frame! Also, he has posted the source, which is always cool.

"the chuck" has created a sawtooth synthesizer in Flash using Pixel Bender to generate the waveforms.

Joe Cutting has created a wind tunnel simulator using Pixel Bender.

Mike Welsh created a cool ray tracer in Pixel Bender and was cool enough to post it in the Pixel Bender exchange.

Tom Beddard has posted a really pretty Mandelbrot generator in the Pixel Bender Exchange.

There has been literally a flood of cool posts on people's blogs about Pixel Bender, in the future, I'll try and collect more links to show you here. We're pretty busy working these days on the next gen of cool stuff for you to use... :)

December 23, 2008

NVidia hosting a Pixel Bender writing contest!

NVidia is really jazzed about the way that Pixel Bender shows off their GPUs in After Effects and in the Photoshop Pixel Bender plug-in. So much so that they've decided to hold a contest to see what Pixel Bender filters you guys can think up!

There are some really sweet prizes including an Alienware Area-51 PC and NVidia cards ('natch). I know I've seen some pretty killer stuff that you guys have been posting to your blogs and to the Pixel Bender Exchange. Now is your chance to really show off!

Here is the info from the press release:
NVIDIA Hosts Adobe Pixel Bender Creation Contest
NVIDIA and Adobe are joining forces to encourage artists and developers to write Adobe Pixel Bender™ Kernels for Adobe Creative Suite 4. The first place winner will receive an Alienware Area-51 Personal Computer. Category winners will receive a NVIDIA Quadro CX graphics card. The categories are coolest Pixel Bender kernel, most useful Pixel Bender kernel in the Adobe Photoshop® workflow, and most technically compelling.

Pixel Bender is a technology created by Adobe that enables a new way of enhancing and adding functionality to Photoshop and other Adobe applications like Adobe Flash and Adobe After Effects. The Adobe Pixel Bender technology delivers a common image and video processing infrastructure which provides automatic runtime optimization on heterogeneous hardware. You can use the Pixel Bender kernel language to implement image processing algorithms (filters or effects) in a hardware-independent manner without having to work with each application’s plug-in SDK. The Pixel Bender graph language is an XML-based language for combining individual pixel-processing operations (kernels) into more complex Pixel Bender filters. The most recent version of the Pixel Bender plug-in (1.1) for Photoshop CS4 was recently released by Adobe and is accelerated by the graphic processing unit (GPU).
More information on Pixel Bender can be found here.

The contest runs from January 1, 2009 to January 31, 2009.

Contest details here

Interview on Pixel Bender from MAX Milan

Michaël Chaize filmed this interview with me at MAX Milan. It is sort of a quick intro to Pixel Bender.

Materials from our MAX "Creating Effects With Pixel Bender" Lab

For those of who couldn't make it, I'm happy to post up the exercises and slides from our "Creating Effects With Pixel Bender" Lab that was presented at MAX 2008 in San Francisco and Milan. The feedback from the lab was great, thanks to all who attended!

The goal of this lab is not to make you a Pixel Bender whiz, but to get you started. It is assumed that you know your way around Flash Authoring a bit and that you are at least familiar with ActionScript. No Pixel Bender or image processing knowledge is assumed (that is what the lab is for!). A lot of people who took the lab had been a bit intimidated by Pixel Bender, but realized that it is very easy and fun to play with.

In order to go through this lab, you need to have Flash Professional CS4 installed. If you don't have it on your computer, you can download a trial here. You also need the Pixel Bender Toolkit. If you purchased Flash, it should already be installed. If you don't have the Pixel Bender Toolkit, you can install it from here.

You can download the presentation slides from this link: Download file. Without us talking, they may not be that useful, but I'm posting them anyway.

You can download the exercises from this link: Download file.

A big thanks to Lee Brimelow from whom I liberally stole when writing the exercises and to Bob Archer and Brian Ronan, who helped me write the slides and who co-taught the class with me in San Francisco.

November 20, 2008

Kerry Garrison demos the Pixel Bender Gallery Plug-in for Photoshop

Pixel Bender Demo from Kerry Garrison on Vimeo.

He gets the name wrong a few times, but it is a nice video anyway :)

November 16, 2008

New Pixel Bender releases for MAX!

Hey everybody! I've been laying a bit low for a little while because the team has been heads-down preparing for MAX and polishing our latest releases.

It seems crazy that only around a year ago we first announced Pixel Bender publicly! At that time we were only taking about Flash, although we had been working to make Pixel Bender plug-ins work with After Effects and Photoshop (John Nack had an excellent hint on his blog that day). This year, we're doing a 90 minute lab on Pixel Bender every day and there are two additional Pixel Bender sessions from the Flash developer community! I'll also be doing the lab twice in Milan at MAX Europe.

Tonight, we posted a new release of the Pixel Bender Toolkit, version 1.1. This version is newer than the one that shipped with CS4. There are some bug fixes and a great developer guide and reference put together by the developer documentation team. Also included is the promised Pixel Bender command-line compiler, PBUtil. Those of you using other tools or looking to automate your Pixel Bender workflows can use it, and those who want to create 3rd party tools can use it for both kernel and bytecode compatibility.

The most important release tonight is the Pixel Bender Plug-in for Photoshop Cs4. It supports both Pixel Bender kernels and Graphs and is accelerated for both GPUs and multi-core CPUs.

You can now use the same Pixel Bender filter in Photoshop, After Effects and Flash!

All of these releases are now available from our new labs page:
http://labs.adobe.com/technologies/pixelbender/

Don't forget:
The Pixel Bender Exchange: http://www.adobe.com/cfusion/exchange/index.cfm?event=productHome&exc=26&loc=en_us
the Pixel Bender Forum: http://www.adobe.com/cfusion/webforums/forum/categories.cfm?forumid=72&catid=661&entercat=y
Pixel Bender on Twitter: http://twitter.com/pixelbender

For those at MAX SF: Brian Ronan, Bob Archer and I are will be at the Meet The Team session for the Flash team tomorrow night. Come by and say hi or find us in the halls of Moscone or the Marriot! We always want to find out what folks are doing with Pixel Bender! Unfortunately, all of the labs are full, but we will post our content after all the MAXes are over.

October 3, 2008

Conduit creates the first 3rd party tool for authoring Pixel Bender filters

Lacquer is a plug-in developer for Adobe products like After Effects and Photoshop. They have a great UI and their tools are really easy to use. Pauli contacted me shortly after the Pixel Bender launch at MAX last year asking if they could extend their Conduit compositing and effects tool to Pixel Bender. My response? "Go for it!" The result is better than I could have hoped for. For those who are intimidated by writing Pixel Bender code, this is worth checking out.

September 26, 2008

CPU, GPU, multi-core

There has been a lot of confusion around Pixel Bender and GPUs in CS4. Admittedly, some of it was caused by me :). I wanted to do a clarifying post about GPU, Pixel Bender, and multi-core and how apps in CS4 do different things.

One thing I wanted to correct is the assumption that GPU = FASTER. I've seen this misconception a lot, and I think it is confusing some people. The chips on graphics cards (GPUs) are extremely efficient processors capable of doing lots of math in parallel and have the benefits of fast local memory with a super fast connection to the processor. This makes them ideal for the kinds of things that Pixel Bender does. However, this super-efficient processor is connected to the main computer processor by a not-so-fast connection, the bus. Moving data on and off of the GPU is expensive relative to doing things on the GPU directly. What this means is that if you want to do something on the CPU with some data then do something on the GPU and then use the output of the GPU on the CPU again, it might be more expensive than having just done the whole thing on the CPU in the first place. The overhead of the bus transfers can overwhelm the benefits of the fast GPU computation. The busses are getting faster, and when things will work better in one place vs. another is very different from machine to machine. There are a ton of other details I'm glossing over. I'm just trying to make a central point here: that the GPU is not always faster than the CPU.

Pixel Bender is designed to run very efficiently on the GPU, but that design also allows it to execute extremely efficiently on a multi-core CPU. In Flash Player 10, Pixel Bender does not run on the GPU, it does run multi-threaded and executes really fast, especially on multi-core and multi-processor chips (see Tinic's post for more info). The Flash team really has done an outstanding job with their JITter and their multi-threading and Pixel Bender runs pretty darn fast on every machine I've tried (from a lowly single core based laptop to an 8-core Penryn MacPro).

In After Effects CS4, all the OpenGL effects including the new Cartoon Effect, Turbulent Noise, and Bilateral Blur effects are written in Pixel Bender and can run on the GPU or CPU. When don't they run on the GPU? When you have a non-GPU effect following them in the effects chain on the layer. In those cases, it isn't clear if you would have a performance gain by running on the GPU. Cartoon is the exception. The algorithm is complex enough that AE assumes it is always faster on the GPU. All 3rd party Pixel Bender filters run multi-threaded on the CPU. This was an architectural decision.

In the Photoshop plug-in, Pixel Bender filters always run on the GPU if you have a graphics card that is supported by CS4. In other cases, the filters run multi-core. The new canvas rotate-pan-and-zoom and the gigantor image support are all done using the GPU. John Nack has lots of details on his blog. One thing I wanted to correct about Photoshop CS4: it is not using CUDA. Not sure how this rumour got out there, but it isn't true. Not that we aren't fans of CUDA, we just aren't shipping anything that uses it in CS4.

There are other apps in CS4 with GPU support, but I wanted to keep this post to the ones that support Pixel Bender, just to clear up the confusion.

September 25, 2008

Pixel Bender Toolkit bug with AIF_FLASH_TARGET

This came up on the Pixel Bender forums and on the pre-release forums. I figured I would blog it so that people would catch it in their web searches.

There is a bug in the current version of the Toolkit where the AIF_FLASH_TARGET is ignored when trying to export a PBJ file from the toolkit (although it is not ignored when testing in Flash-compatibility mode). This is an embarrassing bug, we know.

It will be fixed in the next release posted to Adobe Labs. If you are using this pre-processor variable, you can work around the bug for now, by adding the following code to the top of your pbk file when exporting to pbj:

#undef AIF_FLASH_TARGET
#define AIF_FLASH_TARGET 1

September 23, 2008

Spilling the beans... Pixel Bender in CS4

Oh, it is sooo nice to be able to use the letters CS4! Finally, I can let the cat out of the bag on some of the stuff that the Adobe Image Foundation team has been working on since CS3 shipped.

First, I'll cover the stuff that you might already know because we'd announced it before yesterday:

• Flash 10 supports Pixel Bender bytecode for custom effects, blends, and number crunching. It runs very efficiently multi-threaded thanks to Tinic. You can export Pixel Bender bytecode from the Pixel Bender Toolkit.


• We will be shipping a free extension for Photoshop CS4 soon that will let you run Pixel Bender filters as well. John Nack demoed this at Photoshop World and blogged about it. These effects run on your computer's Graphics card for super fast performance if you have a card that supports them (most do).

• After Effects CS4 supports Pixel Bender filters natively. The existing GPU-accelerated filters are Pixel Bender based and AE CS4 will also automatically load any 3rd party Pixel Bender filters that are put into a known directory.


• The 1.0 version of the Pixel Bender Toolkit will be available to be installed as part of the After Effects and Flash Authoring CS4 installs.

• In addition to Pixel Bender Kernels (pbk) and Pixel Bender Bytecode (pbj), there is another Pixel Bender format. Pixel Bender Graphs (pbg). Pixel Bender Graphs are supported directly in After Effects CS4 and the Photoshop Pixel Bender extension. Pixel Bender Graph is an xml-based format that allows you to combine a network of Pixel Bender Kernels into a single effect for WAAAY more powerful filters. We'll be posting a new version of the toolkit soon that supports the editing of these graphs and we'll also be posting a specification for them soon on Adobe labs.


• We're adding another tool to the Pixel Bender authoring arsenal. There have been a bunch of 3rd party developers who want to create tools that generate Pixel Bender. To make it easier for them, we'll be releasing a command-line utility that will allow you to compile a Pixel Bender Kernel file into a Pixel Bender Bytecode file. The work on this is complete. Look for this on labs very soon.

Got questions? Ask in the comments or on the Pixel Bender forum...

September 16, 2008

Pixel Bender video tutorial

Lee Brimelow has created two video Pixel Bender tutorials on his site gotoandlearn.com. A great intro on getting started with Pixel Bender and Flash. The site is a great resource in general with lots of other neato content as well.

September 11, 2008

Pixel Bender on twitter

(There will be a bunch of quick posts over the next few days. I'm returning from a super long vacation and stuff has piled up)

I created a PixelBender account on twitter for those who like to tweet. Follow @pixelbender. This account will be only Pixel Bender related stuff, saving those who don't want to hear about wiffle ball tournaments.

July 16, 2008

New Pixel Bender Toolkit release on Adobe Labs

Today we posted a new build of the Pixel Bender Toolkit, it is Preview Release #4.

No major changes in this one, lots of bug fixes and UI tweaks (many suggested by the users of the previous builds, thank you!).

Here is the link for downloading.

I'm always interested in hearing any problems that you encounter or features that you would like. Please post those in the forum or as comments on this post.

Also, if you are looking for cool filters to play with or you are looking for a place to share your Pixel Bender Kernels, don't forget the Pixel Bender Exchange.

June 25, 2008

MAX NA Experience launched, cool Pixel Bender sessions!

The MAX Experience website has launched, you can now get the list of sessions and schedule.

I was really excited to see a couple of Pixel Bender sessions being hosted by folks in the community...

More…

June 1, 2008

Talking to the Seattle Flash User's Group this month

On Tuesday, June 17th at 7pm, I'll be talking to the Flash User's Group in Seattle about Pixel Bender. The details are on their site.

May 23, 2008

Using Pixel Bender in Flash Player 10

Lee Brimelow has posted a code snippet to show you how to load Pixel Bender filters in your Flash projects at runtime. I expect a zillion fun Flash-based image and video processing apps to spring from this...

Tinic Uro has posted a really nice backgrounder on how Pixel Bender works in the Flash Player and how it is different from running in the Pixel Bender Toolkit. A must read if you want to understand the nitty gritty.

If you've got the Astro beta, you should check out the new version of the TubeView filter that Petri Leskinen wrote. This has been one of my favorite filters to demo with because of all the fun James Bond-ish possibilities...

May 21, 2008

the formatter in action!

I forgot to post the example! Here is the Fade To History kernel I posted yesterday, run through the formatter:

Pixel Bender HTML formatter

While the PRE tag is ok for displaying code, it always bores me. I like my code to look more interesting. When I was posting the code yesterday, I was asking around the company for formatting tips and I realized that it might be fun to do my own formatter, so I fired up Flex, read up on the String class and created the tool below. Usage is pretty simple, paste your code into the upper text box, hit the "Convert To HTML" button, see the text appear in the second text box and then hit the "Copy HTML to clipboard" button and paste into your blog or web page as raw (and very ugly) HTML code.

(sorry for the size, narrow blog templates)

May 20, 2008

the return of the Fade into history

<languageVersion: 1.0;>

kernel FadeToHistory
<   namespace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade
    < minValue: 0.0;
      maxValue: 2.0;
      defaultValue: 0.0;
    >;
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    
    input image4 src;
    output pixel4 dst;
    
    void
    evaluatePixel()
    {
        dst = sampleNearest(src,outCoord());
        float luminance = dot(dst.rgb, lumMult);
        float3 sepia = float3( dst.r * 0.4 +
                               dst.g * 0.769 +
                               dst.b * 0.189,
                               dst.r * 0.349 +
                               dst.g * 0.686 +
                               dst.b * 0.168,
                               dst.r * 0.272 +
                               dst.g * 0.534 +
                               dst.b * 0.131 );
        
        float3 startMix = dst.rgb;
        float3 endMix = float3(luminance);
        float mixValue = crossfade;
        if ( crossfade > 1.0 )
        {
            // normalize mix value to the range of 0-1
            mixValue -= 1.0;
            startMix = float3(luminance);
            endMix = sepia;
        }
        dst.rgb = mix(startMix, endMix, mixValue);
    }
}

May 16, 2008

The Pixel Bender Exchange is now live!

Waay easier to post on than the Pixel Bender gallery wiki page...

We've primed the pump with some of the filters from the install of the toolkit and we'll add some more from the team soon.

http://www.adobe.com/cfusion/exchange/index.cfm?event=productHome&exc=26

or

http://www.adobe.com/go/pixelbender/

May 14, 2008

Astro Beta and new Pixel Bender Toolkit now available!

Big news tonight!

Finally, you don't have to just hear about Pixel Bender Filters and play with them in the toolkit, now you can actually try some of them out! The beta of Astro is now available. Check out the demos to see Pixel Bender Shaders running on images, vectors and video in real time!

Also, today we released a new Pixel Bender Toolkit on our new Pixel Bender wiki pages. What is new in this release? The new name, obviously, but also some bug fixes and some new features (check out the preferences dialog box!)

More info soon!

May 5, 2008

Hydra is now Pixel Bender!

Hydra is an awesome name for a language like the one we created. At the very beginning, Jonathan Shekter came up with it as a code name for this cool language that could run on different kinds of hardware efficiently. The problem is that it's a great name for any kind of technology that does multiple things, so it is pretty popular. We didn't want to confuse folks, so we worked with the Adobe branding team to come up with a new name that we could use moving forward. That name is Pixel Bender (tm).

So from now on, if you hear me say "hydra" in a 1:1 conversation (sorry, conference talks are out), you can call me on it and I'll give you a cool limited-edition hydra pin (while supplies last).

Along with the name change of the language, we're also changing the name of the "AIF Toolkit" to be the "Pixel Bender Toolkit." The file extensions will also change. We should be posting a new version of the toolkit soon with these changes.

very obscure reference/fact
It is only partially true that the name Pixel Bender won out over another internal favorite "Shektran" because I was hoping to finagle a visit to Rough Draft Studios or The Curiosity Company (joke).

April 11, 2008

AIF Toolkit Preview Release #2 available now!

I've been promising it for a while, but it is finally available for download. The second release of the AIF Toolkit with a lot of changes under the hood (many requested by you!)

A partial list:

• Supports both GPU and CPU execution of Hydra filters. A filter will automatically run on the CPU if your graphics card is not supported by the application.


• Supports user preferences and their persistence across application launches.
  
  
  
  • Flash Warnings and Errors - with this preference checked, Hydra code will be checked for validity against the subset of Hydra that Flash Player supports.
  
  
  • Render on Idle - this setting determines if the toolkit should try to render as often as possible. You may choose to turn off this setting to conserve power when running using battery power on a laptop.
  
  


• Supports exporting a Hydra filter as Hydra Byte Code for future use with Astro.


• There have been some changes and additions to the Hydra language.
  
  
  
  • Conditionals are supported in Flash-compatible Hydra - you are now able to use 'if' statements in your Hydra for Flash programs.
  
  
  • A languageVersion statement is now compulsory in Hydra files
  
  
  • Images are now parameters - images are now available by name in the kernel scope instead of the evaluatePixel scope. That means that they can be used more easily in the per-frame functions like evaluateDependents and the region reasoning functions. In addition, the function signatures for the needed and changed functions have been simplified. The input_index has been replaced by an imageRef and the domain of definition array is no longer passed in. There is a new built-in function – "dod" – that returns the domain of definition of an image.
  
  
  • There are two new built-in functions in Hydra to support pixel aspect ratio (PAR): float2 pixelSize(image) float pixelAspectRatio(image) You can use these functions anywhere inside the kernel, as long as the image parameters to both functions are the declared global input or output image variables.
  
  

  There are a ton more changes big and small. This is still a preview release. There are more changes to come will be in a following preview release. Please try it out and let us know if you find any problems or have more suggestions.
  

  3:11 PM | Permalink | No Comments | No Trackbacks

February 22, 2008

New Adobe Hydra Blog and update

Brian Ronan, a developer on the AIF team has just started a new blog specifically on developing Hydra kernels. Hopefully, he'll make up some of my ample posting slack...

Some news, the long-promised technology preview 2 of the AIF Toolkit is nearing completion. Just ironing out the last few minor bugs and testing the installer now. This release will feature the final syntax of the Hydra 1.0 kernel language. Since my last post we've added a couple things to the language to make it more capable for advanced video and image application usage. I'll post more details and some tips and tricks when the new version is posted.

Stay tuned because we still have some really large surprises around Hydra to announce between now and the official release...

January 15, 2008

What if...

Conditionals were back in for Hydra in Astro?

Well... They Are!

You'll now be able to use if statements in your Hydra for Flash programs. The new toolkit will have that removed as an error. Thank Tinic and Bob Archer for stepping up to get this done for CS4.

This is one of the reasons why I've delayed releasing the Toolkit... I may have another surprise before the second preview release appears on labs.

Watch this space for more details...

January 10, 2008

Hey, where is the new AIF Toolkit?

So, I was hoping to get out a new version of the toolkit before the end of last year. One that would support all platforms. The good news is that it is pretty much done. I decided to wait on a couple features that the team was finishing up and that pushed it out a bit. I'm doing the final cleanup and the team is fixing a couple bugs that we found and then I'll get it out ASAP. So not too much longer now... Sorry for the delay.

December 11, 2007

Hydra course at FITC Amsterdam

I thought I'd give a quick plug to Joa Ebert who is teaching a course entitled Chop The Hydra at FITC Amsterdam in February. If you're attending FITC, you should definitely check it out.

November 15, 2007

Some hydra changes for the next AIF Toolkit release

I've been working on the next version of the toolkit and I'm hoping it will make a bunch of you hydra developers happy. I've rolled in a lot of feature requests from the forums and the big news is that it should work on ALL supported platforms (OS 10.4.10+, XP, Vista) REGARDLESS OF YOUR VIDEO CARD. We're still tuning CPU performance, so if you don't have a supported graphics card, the performance won't be stellar, but it will let you start playing with Hydra. I'm hoping to have it out to you all some time next month.

We have made a language change to Hydra that will be in this next release. It is small, but it will break any existing filters. Sorry about that, but at the moment, this is a technology preview after all. I'll include a script we wrote that will fix up your hydra files automagically. The change is that images are now parameters. This is a small change, but it has a big effect. Images will now be available by name in kernel scope instead of EvaluatePixel scope and that means that they can be used in per-frame functions like EvaluateDependents and the region reasoning functions more simply.

Here's an example of the old and new syntax:

kernel OldSyntax

{

    parameter float k;


    void evaluatePixel(

        in image4 foreground,

        in image4 background,

        out pixel4 result)

    {

        pixel4 fp = sample( foreground, outCoord() );

        pixel4 bp = sample( background, outCoord() );

        

        result = mix( fp, bp, k );

    }

    region needed(

        region output_region,

        int input_index,

        region input_DOD[])

    {

        if (input_index == imageIndex( background )) 

            return input_DOD[ imageIndex( background ) ];

        else 

            return output_region;

    }

}

	

kernel NewSyntax

{

    parameter float k;

    input image4 foreground;

    input image4 background;

    output pixel4 result;

    void evaluatePixel()

    {

        pixel4 fp = sample( foreground, tCoord() );

        pixel4 bp = sample( background, tCoord() );

        result = mix( fp, bp, k );

    }

    region needed(

        region output_region,

        imageRef input_image)

    {

        if( input_image == background ) )

            return dod( background );

        else 

            return output_region;

    }

}

As always, we always want to hear what you think. Either reply here or on the forum.

[Updated 11/19/07 to fix a typo made under the influence of too much caffeine]

October 31, 2007

Cool hydra filters

So, I've been meaning to post about some of the cool hydra filters that folks have been posting to their blogs and the Hydra Filter Gallery. I've been pretty busy working on the next release of the AIF Toolkit, but the whole team has been really digging some of the cool filters that everyone has been posting. Here are some of my favorites so far (in no particular order)... This is just a subset of what has been posted to the gallery...

In a future post, I'll link to some of the stuff that the hydra developers have been posting on their blogs.

What are your favorite filters? Also, what kinds of things would you like me to cover in future posts? Let me know in the comments or on the forum.

October 23, 2007

The FadeToHistory Hydra Filter in action

I figured that the static pictures were boring, so I decided to show what the FadeToHistory might look like in action.

yes. this is a movie, not some secret feature in Flash Player 9. I left in the rewind Flash to make it explicit.

Writing a FadeToHistory Hydra Filter, part 3

Previous parts to this tutorial:
Steps 1-3
Steps 4-6

At this point, we have a hydra filter which does an interactive cross-disolve from a color image to a luminance based black and white image: Download Hydra Filter

The current version of the filter is:

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade;
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float luminance = dot(dst.rgb, lumMult);
        dst.rgb = mix(dst.rgb, float3(luminance), crossfade);
    }
}
    

Step 7: Adding the Sepia Conversion

We're about halfway to being done. We are fading from color to black and white. Time to add the sepia toning so that we can finish this sucker off. There are a bunch of different methods for doing sepia conversion, but essentially the general idea is to mix the red, green and blue channels in different amounts to preserve the original image while giving it that old-time photographic feel. Unlike computing the luminance, we'll need to produce all three color channels, so the calculation is a bit more cumbersome:

        float3 sepia = float3( dst.r * 0.4 +
                               dst.g * 0.769 +
                               dst.b * 0.189,
                               dst.r * 0.349 +
                               dst.g * 0.686 +
                               dst.b * 0.168,
                               dst.r * 0.272 +
                               dst.g * 0.534 +
                               dst.b * 0.131 );
    

I've modified the filter above so that we can test the sepia toning.

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade;
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float luminance = dot(dst.rgb, lumMult);
        float3 sepia = float3( dst.r * 0.4 +
                               dst.g * 0.769 +
                               dst.b * 0.189,
                               dst.r * 0.349 +
                               dst.g * 0.686 +
                               dst.b * 0.168,
                               dst.r * 0.272 +
                               dst.g * 0.534 +
                               dst.b * 0.131 );
        dst.rgb = mix(dst.rgb, sepia, crossfade);
    }
}
    

So now we'll run the filter and made sure that it compiles without error and does something like what we want. If you set the crossfade parameter to 1.0, you should see something like this:

step 8: doing the full fade

We have black and white, we have sepia, now we want to fade from color to black and white and then to sepia. First, I'll show how I would do this with Flash compatibility mode off and then I'll show how I would do it with Flash compatibility on.

With Flash warnings and errors off, I would write the code like this:

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade<minValue:0.0; maxValue:2.0; defaultValue:0.0;>;
    
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float luminance = dot(dst.rgb, lumMult);
        float3 sepia = float3( dst.r * 0.4 +
                               dst.g * 0.769 +
                               dst.b * 0.189,
                               dst.r * 0.349 +
                               dst.g * 0.686 +
                               dst.b * 0.168,
                               dst.r * 0.272 +
                               dst.g * 0.534 +
                               dst.b * 0.131 );
        
        float3 startMix = dst.rgb;
        float3 endMix = float3(luminance);
        float mixValue = crossfade;
        if ( crossfade > 1.0 )
        {
            // normalize mix value to the range of 0-1
            mixValue -= 1.0;
            startMix = float3(luminance);
            endMix = sepia;
        }
        dst.rgb = mix(startMix, endMix, mixValue);
    }
}

If you want to try this, be sure to uncheck the "Turn on Flash warnings and Errors" in the build menu, or you'll get the error "ERROR: (line 30): If statements not supported in Hydra byte code"

A few things to note:

• I've added metadata to the crossfade parameter to specify that its range is now from 0.0 to 2.0 instead of the default 0.0 to 1.0
• I've added temporary variables for startMix, endMix and mixValue to simplify the code. By default, they are the original color, the luminance color and the crossfade value.
• If crossfade is larger than 1.0
  • I overwrite the temporary variable mixValue to be mapped into the range of 0.0 to 1.0 so that we can use that value in the mix function
  • I overwrite the startMix value with luminance color so that is where we mix from instead of the original color
  • I overwrite the endMix value with the sepia color

Now, about that error we get if we try to compile with Flash warnings and errors on. In order to support older graphics cards, we will not support conditional branching (if statements) in Astro. That makes things a bit harder, but in no way impossible. This is how I would rewrite the above filter to get around that limitation:

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade<minValue:0.0; maxValue:2.0; defaultValue:0.0;>;
    
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float3 luminance = float3( dot(dst.rgb, lumMult) );
        float3 sepia = float3( dst.r * 0.4 +
                               dst.g * 0.769 +
                               dst.b * 0.189,
                               dst.r * 0.349 +
                               dst.g * 0.686 +
                               dst.b * 0.168,
                               dst.r * 0.272 +
                               dst.g * 0.534 +
                               dst.b * 0.131 );
        
        float3 startMix = dst.rgb;
        startMix = (crossfade <= 1.0) ? startMix : luminance;
        float3 endMix = (crossfade <= 1.0) ? luminance : sepia;
        float mixMinusOne = crossfade - 1.0;
        float mixValue = (crossfade <= 1.0) ? crossfade : mixMinusOne;
        dst.rgb = mix(startMix, endMix, mixValue);
    }
}
    

Instead of doing conditional branching in Flash, I can do conditional assignments of pre-computed values. What this means is that I have to do a little more computation, but I can still get the results I want. Instead of only doing computation in an if or else statement, I do all the computation in the filter and then only assign the result to the final variables if the condition is met. Of course you can do conditional assignments in regular Hydra too to be clear.

The big changes in this version of the filter is that I have a couple of extra statements and an extra variable, but the output is exactly the same.

All we have left to do is a bit more cleanup and then we're done!

Step 9: final cleanup

That sepia calculation is annoying. There is a lot of math there and it takes up a lot of space. Doing these kinds of calculations can be pretty common in image and video processing and there is a simpler way to do them in Hydra. Doing this kind of operation where we combine each channel of a color with a blend of all of the colors added together is exactly the same thing we do when we multiply a vector by a matrix. Matricies are built-in to Hydra and you can use them like any other type. This is how that looks:

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade<minValue:0.0; maxValue:2.0; defaultValue:0.0;>;
    
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    const float3x3 sepiaMatrix = float3x3(0.400, 0.769, 0.189,
                                          0.349, 0.686, 0.168,
                                          0.272, 0.534, 0.131);
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float3 luminance = float3( dot(dst.rgb, lumMult) );
        float3 sepia = dst.rgb * sepiaMatrix;
        
        float3 startMix = dst.rgb;
        startMix = (crossfade <= 1.0) ? startMix : luminance;
        float3 endMix = (crossfade <= 1.0) ? luminance : sepia;
        float mixMinusOne = crossfade - 1.0;
        float mixValue = (crossfade <= 1.0) ? crossfade : mixMinusOne;
        
        dst.rgb = mix(startMix, endMix, mixValue);
    }
}
    

Once again, I've used a const for sepiaMatrix because it stays the same from pixel to pixel. Now everything looks cleaner and simpler. It is easier to understand and more importantly, it is easier to mess around with and make changes. Download the final version and try it for yourself!

Thanks for following this tutorial and let me know in the comments if you have questions or would like other tutorials.

October 18, 2007

Writing a FadeToHistory Hydra Filter, part 2

When we left off in part 1, we had a filter that took a color image and then created a black and white image by using the red channel from original image for the red, green and blue channels of the output image. Going from this:

to this:

And here is the hydra filter that we left off with:

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord()).rrra;
    }
}

If you would like to download what we have so far, a link to the above hydra filter is here: FadeToHistory2.hydra

Now, onward and upward!

Part Four: Making it interactive

We now have a filter that takes a color image and does a simple conversion to black and white. This is nice, but we could do it in Photoshop and use that instead. Where Hydra gets cool is by allowing you to interact with the filters, automate and animate them. Let's add a parameter to the filter that will allow us to animate the image from color to black and white. The new filter looks like this:

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade;
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float3 bw = dst.rrr;
        dst.rgb = ((1.0 - crossfade) * dst.rgb) + (crossfade * bw);
    }
}

There are a few changes here:

1. parameter float crossfade; This line tells the compiler that you want a parameter to your filter that will be exposed to the application hosting your filter. In this case, a float parameter. If you run the filter above, you will see a slider in the area to the right of your image. Moving this slider changes your parameter value.
2. dst = sampleNearest(src,outCoord()); We've reverted this back to what it used to be when we did new filter. We're going to use the dst variable to store the color version of the image.
3. float3 bw = dst.rrr; I've created a temporary variable to store the black-and-white version of the rgb channels from the original image. Float3 is an array of three float values. In Hydra, pixel values are stored as floats.
4. dst.rgb = ((1.0 - crossfade) * dst.rgb) + (crossfade * bw); This is a simple crossfade calculation using the crossfade variable to determine the amount. When the crossfade parameter is 0, you get the color image. When the crossfade parameter is 1, you get the black and white image. We overwrite the red, green, and blue colors from the original image with the mix of color and black and white values based on the crossfade parameter.

Part Five: Better Black and White conversion

If you've tried this Hydra kernel on any image without a lot of red content in it, the results are pretty unsatisfying. Time to do a better conversion.

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade;
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float luminance = dst.r * 0.3 + dst.g * 0.59 + dst.b * 0.11;
        dst.rgb = ((1.0 - crossfade) * dst.rgb) + (crossfade * float3(luminance));
    }
}

There are a couple of important things here:
float luminance = dst.r * 0.3 + dst.g * 0.59 + dst.b * 0.11;
This line computes a single float value that is a combination of the red, green and blue colors from the original image. This formula is pretty similar to how color TV images are converted for old black and white TV screens. The values are based on how your eyes perceive color. If you used 0.33 for each of the multipliers, you'd actually have something that looked kind of wrong because your eyes don't perceive blue and red in the same way you see green. For more info, check out this wikipedia article.

float3(luminance)
Here we create a new float3 variable in-line where each of the three indicies are set to the value of luminance.

Part Six: A Bit of Clean Up

For this last part of this post, I'm going to make a couple changes to clean up the code. These will simplify some things as well as show some more built in functions for Hydra.

kernel FadeToHistory
<   nameSpace : "Kevin's Tutorial Filters";
    vendor : "Kevin's Filters, Inc";
    version : 1;
    description : "Fade from color to B&W to sepia";
>
{
    parameter float crossfade;
    const float3 lumMult = float3(0.3, 0.59, 0.11);
    
    void
    evaluatePixel(in image4 src, out pixel4 dst)
    {
        dst = sampleNearest(src,outCoord());
        float luminance = dot(dst.rgb, lumMult);
        dst.rgb = mix(dst.rgb, float3(luminance), crossfade);
    }
}

With this iteration, I've created a const variable:
const float3 lumMult = float3(0.3, 0.59, 0.11);
This variable contains the numbers I multiply red, green, and blue by to get the luminance value. Since these numbers do not change ever, I can put them in a const variable. This simplifies the code and allows the compiler to do some optimization because I'm telling the compiler that this value will not change.

float luminance = dot(dst.rgb, lumMult);
I've changed from multiplying the red, green and blue components of the input image by the luminance calculation and then adding them together to using the built-in dot function which does exactly that. Dot specifies the dot product of two vectors, which is computed by multiplying the components of the vectors and then adding them up (which is exactly what we were doing to compute the luminance value). Fore more info on dot products, this wikipedia article is extensive.

dst.rgb = mix(dst.rgb, float3(luminance), crossfade);
Since operations like the crossfade that I was doing before are pretty common in image processing, we've have a built-in function to do linear interpolation between two values. This is the mix function, and I use it here rather than doing the computation myself.

At the end of part 6, we have a kernel with a parameter that lets us smoothly animate from a color image to a luminance-based black and white image. To play with the filter, you can download the Hydra file: Download the Hydra file

October 17, 2007

Writing a FadeToHistory Hydra Filter, part 1

So, I promised a walk through tutorial of the code I wrote in our MAX talk. It's taken me a while to get some time, but here I go.

So the filter I'm going to write works like this: I want to take a color image and then fade it to black and while and then continue to fade into a sepia-tone. Sort of a Ken Burns-ish thing. I'm going to walk through how I would do this iteratively, starting from the default new filter you get by choosing "New Filter" and then expand on it until I get my final goal.

To Illustrate

we'll create a filter that goes smoothly from this:

To This:

And finally to this:

More…

October 8, 2007

AIF Toolkit and Hydra slides from MAX posted

Hey all, I'm still getting out from under the mountains of mail I received during the MAX conference, so I'll be posting more on MAX and Hydra (tons of blog postings in the community!) soon. In the meantime, I wanted to let you know that we've posted our slides from our talk at MAX as a PDF. Download it here.

I'll be posting the code that I wrote live on the stage to the labs wiki soon.

October 1, 2007

Joa Ebert becomes the first 3rd party hydra developer

Joa Ebert has published the first non-Adobe hydra filter, while we're still updating the Wiki! It's simple, but it's pretty cool considering that the toolkit has only been available for a couple of hours!

What can you make between now and Wednesday?

Our talk again...
Wednesday, October 3rd @ 3pm
Room W-183B

It's Alive! (Hydra and Astro)

This morning, at the close of the MAX keynote, we formally announced that the AIF Team's image processing language, code-named "Hydra", will be part of Astro AKA Flash Player 10. Emmy and Justin from the Flash team demoed an application called the Adobe Image Foundation Toolkit Technology Preview. That tool is available from Adobe Labs NOW.

We are super-excited to get this technology into your hands to see what you can do with it!

The application that we have made available will let you build hydra filters and see them run on your images. The AIF team wants your feedback on the language and the tools so that we can continue to improve them before their final release.

While this technology is new to Flash, it isn't new to Adobe products, we first shipped this technology in After Effects CS3. That said, there has been a ton of effort to get this release ready for you to play with and it feels great to see it on the big screen in front of a crowd of enthusiastic developers!

Now that the cat is out of the bag, so to speak, I'll be posting more on this blog about Hydra and Astro so stay tuned. You can also post questions and comments on the labs forum.

Don't forget our Talk about the AIF Toolkit and Hydra! Wednesday, 3pm in Room
Room W-183B

September 21, 2007

See you at MAX!

The AIF team is having a coming out party at MAX and all of you are invited! Bob Archer and I will be speaking on Wednesday on from 3pm to 4pm.

Image and Video Processing using Adobe Image Foundation's Toolkit for Flash

Kind of a boring title, but we're engineers, and I promise the talk will be pretty exciting. Especially after you see the Keynote presentations...

"Discover a new language for image and video processing (developed by Adobe Image Foundation) that will soon be available on Adobe Labs. In this session, we will demonstrate tools for writing and testing the language, explain how to write efficient algorithms, and share examples of the possibilities for development. We'll also show how these tools were used to create some of the video processing effects shipping in Adobe After Effects CS3."

I want to talk more, but I don't want to ruin the surprises that we have in store. If you are at MAX, come by and say "Hi!" I'll be around all 3 days, there is plenty of stuff for me to see too! Also, I grew up in Chicago, so I can tell you where to get the good pizza and char dogs.

June 18, 2007

Hey, My Team is Hiring!

So, I've been quiet on this blog for a while because my team has been working on some new technologies and I didn't want to say anything until we went public. As I mentioned in my previous post, the first version of our project is shipping in After Effects 8. We're doing an advanced hardware-accelerated image and video processing library called the Adobe Image Foundation.

Now, we are looking for someone who wants to write filters for the world's best imaging company. For this position, we are looking for a very senior person, PhD-level or extensive industry experience. If this describes you or someone you know, check out the link below:
http://cooljobs.adobe.com/frameset.html?goto=er-viewjob&erjob=60804&eresc=ER

Now that the cat is out of the bag, so to speak, I hope to resume posting a little bit more.