Recently in Flash Category

Last week, we posted a new revision of the Pixel Bender Toolkit (1.5.1) into the downloads area on Adobe labs. This version fixes a couple bugs from the previous release in exporting Pixel Bender Bytecode for Flash. Booleans should now work correctly.

A request we've had a few times on the Pixel Bender forum and over on twitter has been that we post the language reference and developer's guide separate from the Toolkit download itself. Those documents are now available as well on the downloads page.

I've been doing some playing around with processing audio using Pixel Bender in Flash and I realized that it was hard to find some working code to get started with. So i wrote up this sample. I tried to do a minimal app that actually did something interesting and would be a start for someone else. To that end, this AIR app sample loads an MP3 file and then the embedded Pixel Bender kernel lets you change the level of the individual channels separately.

The MXML code is below:

All the action is in the ProcessAudio function, that pulls samples from the input file and executes a ShaderJob across them. There is something important to reference here:

effectShader.data["source"].width = BUFFER_SIZE / 1024;
effectShader.data["source"].height = 512;
effectShader.data["source"].input = shaderBuffer;
effectShader.data["volume"].value = [ leftSlider.value, righttSlider.value ];
				
var effectJob:ShaderJob = new ShaderJob( effectShader, event.data, BUFFER_SIZE / 1024, 512 );	
effectJob.start(true);

I pass the buffer into the shader job as a 2D buffer instead of a buffer with a height of one. This may make less sense logically, but the Flash player breaks the data up by rows for multi-threading, so this should make that perform faster.

Here is the kernel:

One thing to notice here is that rather than using an image2 as input and a pixel2 as output (which may make more sense logically again), I instead just use the buffer layout and process 2 stereo samples at the same time. This should also give you better performance for filters that can do this.

Here are the files:

• PixelBenderAudioDemo.mxml


• Volume1Channel.pbk

For more info, the following references might be helpful

• Tinic Uro's original post on audio processing with Pixel Bender


• A post on the mediabox forum which has a working version of Tinic's code


• A post from Саша Рудан that goes into more detail on sound production


• Some Pixel Bender experiments from Mihai Pricope, including an audio mixing sample


• Flash Magazine wrote up a great tutorial on using Pixel Bender for number crunching which got me thinking about doing this sample


• Ryan Taylor also has a great post on doing number crunching and audio processing with Pixel Bender

We tried to be semantically agnostic in the original design of the Pixel Bender language. We'd seen other languages go down rabbit holes of over-specification around what parameters really meant, locking them into archaic and insufficient implementations or clumsy hierarchies of meanings. We didn't want to limit Pixel Bender developers into what we could conceive at the time of the invention of the language.

We were being a bit too idealistic :). It is completely true that the community of Pixel Bender developers continues to blow our minds at what they accomplish with the language and we never would have anticipated half the stuff that you guys are using Pixel Bender for. However, having some generally useful semantic meanings for Pixel Bender parameters would definitely help those who design general user interfaces for Pixel Bender filters.

One smart thing we did (if I do say myself) was to allow parameter and kernel metadata to be extensible. It provided developers like After Effects and Picnik a way to add custom metadata to Pixel Bender that specified semantics or actual UI controls for Pixel Bender kernels and graphs. The picnik metadata and the After Effects metadata are different though. I started to get concerned that by not having something around this that we could end up with many different Pixel Bender semantic metadata mechanisms floating around. To that end, I created the proposal below and started floating it around Adobe and some of the sites that are heavy users of Pixel Bender.

In this design, I tried to follow some general rules:

• Pixel Bender is not only a way to write image and video filters for Adobe applications, but also a way for you to host 3rd party filters in your Flash-based applications. Any guidelines we create need to be implemented by us, but also by any independent Flash developers creating apps that use Pixel Bender kernels. To that end, I tried to keep the design relatively simple so that it wouldn't be too difficult to implement in Flash.


• This proposal adds semantic metadata, but avoids specifying specific user interface controls. Rather than specifying a slider as an editor for a parameter, I think it makes more sense to say that the parameter is a percentage and allow someone designing a UI to make a good percentage editor. We are definitely thinking about how to specify custom editor UIs for Pixel Bender filters, but this proposal does not approach that.


• The proposal is not a complete answer for all Pixel Bender filters. I'm trying to get the most universal semantics represented. The most generally useful, as opposed to trying to give the complete solution. If you have an application that is uses Pixel Bender kernels for something that makes sense to augment the metadata you can still choose to do so.


• The proposal represents guidelines, not requirements. As a developer consuming Pixel Bender kernels, you can choose to enumerate the metadata as described in the proposal or not. The suggested metadata is metadata. It is not required. The goal is that if you wish to take advantage of it when it is present, that you can provide a more compelling user interface to your users because you understand the intent of a parameter, not just its type.
  

There are a couple open questions in the proposal that I'd like feedback on. They are called out pretty clearly. I'd really like your feedback on this proposal. I hope to issue the final version soon. Reply to this post or in the Pixel Bender forums on Adobe Labs. If you decide to post a reply on your own blog, please post a link to your post here or in the forum. Right now trackbacks are off so I won't know about it otherwise. You can also tweet a reply to the Pixel Bender twitter account (if you can fit it in 140 characters :) ).

Pixel Bender Metadata Hinting for User Interface Guidelines public proposal.pdf

I've often been asked if Adobe was going to do an eclipse plug-in for Pixel Bender. It's been something that we definitely wanted to do, but I wasn't sure when we would get to it. Joa Ebert decided to take advantage of the PBUtil command-line application that we ship with the Pixel Bender Toolkit SDK to create his own eclipse plug-in, PBDT. He even came up with a way to get Pixel Bender development working with Linux (another often requested feature).

I've been playing with it a bit (it integrates with Flex and FDT pretty well) and it is a cool alternative way to do Pixel Bender development.

Thanks to Joa for his contribution to the community!

The newest version of the Pixel Bender Toolkit has been released on Adobe Labs. This is pre-release 6, but it is also version 1.5. This new version includes the ability to edit, compile and run Pixel Bender Graphs (supported in Photoshop and After Effects). It also has a number of bug fixes, specifically in areas around PBJ generation.

You can get it from the Pixel Bender Technology area on Adobe Labs, and please let us know about any issues or suggestions for the next release in our forums, our Adobe Group or on twitter.

Since the first posting of the Pixel Bender toolkit, developers have been pushing the boundaries, moving beyond image and video processing filters. Some really cool things have been posted, so I thought I would call attention to some of the things I've seen. There is a ton more out there to check out, this is just some of the things I've seen posted lately.

David Lenaerts has created a smoke/fluids simulation in Pixel Bender utilizing 49 Pixel Bender kernels per frame! Also, he has posted the source, which is always cool.

"the chuck" has created a sawtooth synthesizer in Flash using Pixel Bender to generate the waveforms.

Joe Cutting has created a wind tunnel simulator using Pixel Bender.

Mike Welsh created a cool ray tracer in Pixel Bender and was cool enough to post it in the Pixel Bender exchange.

Tom Beddard has posted a really pretty Mandelbrot generator in the Pixel Bender Exchange.

There has been literally a flood of cool posts on people's blogs about Pixel Bender, in the future, I'll try and collect more links to show you here. We're pretty busy working these days on the next gen of cool stuff for you to use... :)

For those of who couldn't make it, I'm happy to post up the exercises and slides from our "Creating Effects With Pixel Bender" Lab that was presented at MAX 2008 in San Francisco and Milan. The feedback from the lab was great, thanks to all who attended!

The goal of this lab is not to make you a Pixel Bender whiz, but to get you started. It is assumed that you know your way around Flash Authoring a bit and that you are at least familiar with ActionScript. No Pixel Bender or image processing knowledge is assumed (that is what the lab is for!). A lot of people who took the lab had been a bit intimidated by Pixel Bender, but realized that it is very easy and fun to play with.

In order to go through this lab, you need to have Flash Professional CS4 installed. If you don't have it on your computer, you can download a trial here. You also need the Pixel Bender Toolkit. If you purchased Flash, it should already be installed. If you don't have the Pixel Bender Toolkit, you can install it from here.

You can download the presentation slides from this link: Download file. Without us talking, they may not be that useful, but I'm posting them anyway.

You can download the exercises from this link: Download file.

A big thanks to Lee Brimelow from whom I liberally stole when writing the exercises and to Bob Archer and Brian Ronan, who helped me write the slides and who co-taught the class with me in San Francisco.