Posts in Category "Flash"
Pixel Bender has been living on Adobe labs since it was first introduced at MAX 2007. Since CS4 and Flash Player 10 though it has joined the ranks of “real” Adobe technologies.
We’ve finally created “official” areas for Pixel Bender in the Adobe Developer Connection and Adobe Cookbook sites. We’ve also moved the Pixel Bender forum out of the labs forums area and into the main Adobe forums site. These join the Pixel Bender exchange as great resources for Pixel Bender developers.
This is now the main page for the Pixel Bender technology. Currently it features information on learning Pixel Bender and links to tutorial videos, references, the latest official downloads and more.
Looking for a place to post useful Actionscript or Pixel Bender code, or find code from other folks? This is the new place. You can also request recipes from the community.
The Pixel Bender Exchange has also been around for a while and is THE place to post your Pixel Bender filters or browse filters created by other developers.
The Pixel Bender area on labs isn’t gone. Watch that space for future public betas or pre-releases.
Last week, we posted a new revision of the Pixel Bender Toolkit (1.5.1) into the downloads area on Adobe labs. This version fixes a couple bugs from the previous release in exporting Pixel Bender Bytecode for Flash. Booleans should now work correctly.
A request we’ve had a few times on the Pixel Bender forum and over on twitter has been that we post the language reference and developer’s guide separate from the Toolkit download itself. Those documents are now available as well on the downloads page.
I’ve been doing some playing around with processing audio using Pixel Bender in Flash and I realized that it was hard to find some working code to get started with. So i wrote up this sample. I tried to do a minimal app that actually did something interesting and would be a start for someone else. To that end, this AIR app sample loads an MP3 file and then the embedded Pixel Bender kernel lets you change the level of the individual channels separately.
The MXML code is below:
All the action is in the ProcessAudio function, that pulls samples from the input file and executes a ShaderJob across them. There is something important to reference here:
effectShader.data["source"].width = BUFFER_SIZE / 1024; effectShader.data["source"].height = 512; effectShader.data["source"].input = shaderBuffer; effectShader.data["volume"].value = [ leftSlider.value, righttSlider.value ]; var effectJob:ShaderJob = new ShaderJob( effectShader, event.data, BUFFER_SIZE / 1024, 512 ); effectJob.start(true);
I pass the buffer into the shader job as a 2D buffer instead of a buffer with a height of one. This may make less sense logically, but the Flash player breaks the data up by rows for multi-threading, so this should make that perform faster.
Here is the kernel:
One thing to notice here is that rather than using an image2 as input and a pixel2 as output (which may make more sense logically again), I instead just use the buffer layout and process 2 stereo samples at the same time. This should also give you better performance for filters that can do this.
Here are the files:
For more info, the following references might be helpful
- Tinic Uro’s original post on audio processing with Pixel Bender
- A post on the mediabox forum which has a working version of Tinic’s code
- A post from Саша Рудан that goes into more detail on sound production
- Some Pixel Bender experiments from Mihai Pricope, including an audio mixing sample
- Flash Magazine wrote up a great tutorial on using Pixel Bender for number crunching which got me thinking about doing this sample
- Ryan Taylor also has a great post on doing number crunching and audio processing with Pixel Bender
We tried to be semantically agnostic in the original design of the Pixel Bender language. We’d seen other languages go down rabbit holes of over-specification around what parameters really meant, locking them into archaic and insufficient implementations or clumsy hierarchies of meanings. We didn’t want to limit Pixel Bender developers into what we could conceive at the time of the invention of the language.
We were being a bit too idealistic :). It is completely true that the community of Pixel Bender developers continues to blow our minds at what they accomplish with the language and we never would have anticipated half the stuff that you guys are using Pixel Bender for. However, having some generally useful semantic meanings for Pixel Bender parameters would definitely help those who design general user interfaces for Pixel Bender filters.
One smart thing we did (if I do say myself) was to allow parameter and kernel metadata to be extensible. It provided developers like After Effects and Picnik a way to add custom metadata to Pixel Bender that specified semantics or actual UI controls for Pixel Bender kernels and graphs. The picnik metadata and the After Effects metadata are different though. I started to get concerned that by not having something around this that we could end up with many different Pixel Bender semantic metadata mechanisms floating around. To that end, I created the proposal below and started floating it around Adobe and some of the sites that are heavy users of Pixel Bender.
In this design, I tried to follow some general rules:
- Pixel Bender is not only a way to write image and video filters for Adobe applications, but also a way for you to host 3rd party filters in your Flash-based applications. Any guidelines we create need to be implemented by us, but also by any independent Flash developers creating apps that use Pixel Bender kernels. To that end, I tried to keep the design relatively simple so that it wouldn’t be too difficult to implement in Flash.
- This proposal adds semantic metadata, but avoids specifying specific user interface controls. Rather than specifying a slider as an editor for a parameter, I think it makes more sense to say that the parameter is a percentage and allow someone designing a UI to make a good percentage editor. We are definitely thinking about how to specify custom editor UIs for Pixel Bender filters, but this proposal does not approach that.
- The proposal is not a complete answer for all Pixel Bender filters. I’m trying to get the most universal semantics represented. The most generally useful, as opposed to trying to give the complete solution. If you have an application that is uses Pixel Bender kernels for something that makes sense to augment the metadata you can still choose to do so.
- The proposal represents guidelines, not requirements. As a developer consuming Pixel Bender kernels, you can choose to enumerate the metadata as described in the proposal or not. The suggested metadata is metadata. It is not required. The goal is that if you wish to take advantage of it when it is present, that you can provide a more compelling user interface to your users because you understand the intent of a parameter, not just its type.
There are a couple open questions in the proposal that I’d like feedback on. They are called out pretty clearly. I’d really like your feedback on this proposal. I hope to issue the final version soon. Reply to this post or in the Pixel Bender forums on Adobe Labs. If you decide to post a reply on your own blog, please post a link to your post here or in the forum. Right now trackbacks are off so I won’t know about it otherwise. You can also tweet a reply to the Pixel Bender twitter account (if you can fit it in 140 characters 🙂 ).
I’ve often been asked if Adobe was going to do an eclipse plug-in for Pixel Bender. It’s been something that we definitely wanted to do, but I wasn’t sure when we would get to it. Joa Ebert decided to take advantage of the PBUtil command-line application that we ship with the Pixel Bender Toolkit SDK to create his own eclipse plug-in, PBDT. He even came up with a way to get Pixel Bender development working with Linux (another often requested feature).
I’ve been playing with it a bit (it integrates with Flex and FDT pretty well) and it is a cool alternative way to do Pixel Bender development.
Thanks to Joa for his contribution to the community!
The newest version of the Pixel Bender Toolkit has been released on Adobe Labs. This is pre-release 6, but it is also version 1.5. This new version includes the ability to edit, compile and run Pixel Bender Graphs (supported in Photoshop and After Effects). It also has a number of bug fixes, specifically in areas around PBJ generation.
Since the first posting of the Pixel Bender toolkit, developers have been pushing the boundaries, moving beyond image and video processing filters. Some really cool things have been posted, so I thought I would call attention to some of the things I’ve seen. There is a ton more out there to check out, this is just some of the things I’ve seen posted lately.
“the chuck” has created a sawtooth synthesizer in Flash using Pixel Bender to generate the waveforms.
Joe Cutting has created a wind tunnel simulator using Pixel Bender.
Mike Welsh created a cool ray tracer in Pixel Bender and was cool enough to post it in the Pixel Bender exchange.
Tom Beddard has posted a really pretty Mandelbrot generator in the Pixel Bender Exchange.
There has been literally a flood of cool posts on people’s blogs about Pixel Bender, in the future, I’ll try and collect more links to show you here. We’re pretty busy working these days on the next gen of cool stuff for you to use… 🙂
For those of who couldn’t make it, I’m happy to post up the exercises and slides from our “Creating Effects With Pixel Bender” Lab that was presented at MAX 2008 in San Francisco and Milan. The feedback from the lab was great, thanks to all who attended!
The goal of this lab is not to make you a Pixel Bender whiz, but to get you started. It is assumed that you know your way around Flash Authoring a bit and that you are at least familiar with ActionScript. No Pixel Bender or image processing knowledge is assumed (that is what the lab is for!). A lot of people who took the lab had been a bit intimidated by Pixel Bender, but realized that it is very easy and fun to play with.
In order to go through this lab, you need to have Flash Professional CS4 installed. If you don’t have it on your computer, you can download a trial here. You also need the Pixel Bender Toolkit. If you purchased Flash, it should already be installed. If you don’t have the Pixel Bender Toolkit, you can install it from here.
You can download the presentation slides from this link: Download file. Without us talking, they may not be that useful, but I’m posting them anyway.
You can download the exercises from this link: Download file.
A big thanks to Lee Brimelow from whom I liberally stole when writing the exercises and to Bob Archer and Brian Ronan, who helped me write the slides and who co-taught the class with me in San Francisco.
There has been a lot of confusion around Pixel Bender and GPUs in CS4. Admittedly, some of it was caused by me :). I wanted to do a clarifying post about GPU, Pixel Bender, and multi-core and how apps in CS4 do different things.
One thing I wanted to correct is the assumption that GPU = FASTER. I’ve seen this misconception a lot, and I think it is confusing some people. The chips on graphics cards (GPUs) are extremely efficient processors capable of doing lots of math in parallel and have the benefits of fast local memory with a super fast connection to the processor. This makes them ideal for the kinds of things that Pixel Bender does. However, this super-efficient processor is connected to the main computer processor by a not-so-fast connection, the bus. Moving data on and off of the GPU is expensive relative to doing things on the GPU directly. What this means is that if you want to do something on the CPU with some data then do something on the GPU and then use the output of the GPU on the CPU again, it might be more expensive than having just done the whole thing on the CPU in the first place. The overhead of the bus transfers can overwhelm the benefits of the fast GPU computation. The busses are getting faster, and when things will work better in one place vs. another is very different from machine to machine. There are a ton of other details I’m glossing over. I’m just trying to make a central point here: that the GPU is not always faster than the CPU.
Pixel Bender is designed to run very efficiently on the GPU, but that design also allows it to execute extremely efficiently on a multi-core CPU. In Flash Player 10, Pixel Bender does not run on the GPU, it does run multi-threaded and executes really fast, especially on multi-core and multi-processor chips (see Tinic’s post for more info). The Flash team really has done an outstanding job with their JITter and their multi-threading and Pixel Bender runs pretty darn fast on every machine I’ve tried (from a lowly single core based laptop to an 8-core Penryn MacPro).
In After Effects CS4, all the OpenGL effects including the new Cartoon Effect, Turbulent Noise, and Bilateral Blur effects are written in Pixel Bender and can run on the GPU or CPU. When don’t they run on the GPU? When you have a non-GPU effect following them in the effects chain on the layer. In those cases, it isn’t clear if you would have a performance gain by running on the GPU. Cartoon is the exception. The algorithm is complex enough that AE assumes it is always faster on the GPU. All 3rd party Pixel Bender filters run multi-threaded on the CPU. This was an architectural decision.
In the Photoshop plug-in, Pixel Bender filters always run on the GPU if you have a graphics card that is supported by CS4. In other cases, the filters run multi-core. The new canvas rotate-pan-and-zoom and the gigantor image support are all done using the GPU. John Nack has lots of details on his blog. One thing I wanted to correct about Photoshop CS4: it is not using CUDA. Not sure how this rumour got out there, but it isn’t true. Not that we aren’t fans of CUDA, we just aren’t shipping anything that uses it in CS4.
There are other apps in CS4 with GPU support, but I wanted to keep this post to the ones that support Pixel Bender, just to clear up the confusion.