The subject of using Pixel Bender for audio processing on the Flash platform comes up a lot. Audio processing is very processor-intensive and math-heavy, so it would seem natural to leverage Pixel Bender to improve performance of audio within a SWF. At some point, last year, I was talking to Justin Everett-Church about doing a demo for Flash Player 10.1 multi-touch features. A synth seemed like a good idea, and it would let me kill two birds with one stone. So I coded up a synth (with a lame Flex UI) and Justin took that and made it pretty and added multi-touch support. That 2nd part of the demo never worked out the way we meant it too (missed the MAX 2009 keynote by this much). After that, I planned on cleaning up the code and posting it, but I got busy shipping CS5 and well, 7 months later, I finally got around to posting it. Right now it is just the playable demo with the crummy Flex UI.
It is a total processor hog, on purpose. I basically wanted to use it to push the limit of what could be done in the player, so I kept adding more filters and processors to it until the audio started to break up on my Core Duo 2 Mac Book Pro and then stepped back just a little. It turns out that you can actually do a ton of audio processing interactively in the player leveraging Pixel Bender. This was also designed to run as an AIR app, which means that if you really want to play with it, close all your other tabs. Really.
Pixel Bender Synth Experiment
I’ll be posting all the code on the page eventually.
Some techy details:
Each Pixel Bender shader is encapsulated within an Actionscript class and launched via a ShaderJob.
Each Pixel Bender filter processes two samples at once to double throughput.
The synth uses three Pixel Bender Shaders as does the Reverb, the other filters use one.
The delay-based filters (reverb/delay) are using simple bit-bucket style delay in actionscript, but the delayed signals are mixed back with the main signal via Pixel Bender shaders.
All signal paths are stereo. When mixing two stereo inputs, a pixel4 representing the 2 sets of stereo channels is used. When processing/generating a single stereo signal, a pixel4 representing two sequential stereo samples is used.
The filters are ordered as such:
Will post more info soon. Ask any questions in the comments. Thanks!
Pixel Bender’s heritage is from the 3D Pixel Shading languages, but we carefully removed most of that to make it specialized for signal processing. The lineage is there though, right below the surface, and Pixel Bender’s utility for doing a lot of math fast makes it really attractive for doing 3D. I’ve written about 3D experiments from the community in the posts Some cool experiments with Pixel Bender and More Cool Pixel Bender Experiments. There has been a lot of really cool work done in the community since then, and I wanted to highlight that:
I’ve been doing some playing around with processing audio using Pixel Bender in Flash and I realized that it was hard to find some working code to get started with. So i wrote up this sample. I tried to do a minimal app that actually did something interesting and would be a start for someone else. To that end, this AIR app sample loads an MP3 file and then the embedded Pixel Bender kernel lets you change the level of the individual channels separately.
The MXML code is below:
All the action is in the ProcessAudio function, that pulls samples from the input file and executes a ShaderJob across them. There is something important to reference here:
I pass the buffer into the shader job as a 2D buffer instead of a buffer with a height of one. This may make less sense logically, but the Flash player breaks the data up by rows for multi-threading, so this should make that perform faster.
Here is the kernel:
One thing to notice here is that rather than using an image2 as input and a pixel2 as output (which may make more sense logically again), I instead just use the buffer layout and process 2 stereo samples at the same time. This should also give you better performance for filters that can do this.