by Kevin Goldsmith
The subject of using Pixel Bender for audio processing on the Flash platform comes up a lot. Audio processing is very processor-intensive and math-heavy, so it would seem natural to leverage Pixel Bender to improve performance of audio within a SWF. At some point, last year, I was talking to Justin Everett-Church about doing a demo for Flash Player 10.1 multi-touch features. A synth seemed like a good idea, and it would let me kill two birds with one stone. So I coded up a synth (with a lame Flex UI) and Justin took that and made it pretty and added multi-touch support. That 2nd part of the demo never worked out the way we meant it too (missed the MAX 2009 keynote by this much). After that, I planned on cleaning up the code and posting it, but I got busy shipping CS5 and well, 7 months later, I finally got around to posting it. Right now it is just the playable demo with the crummy Flex UI.
It is a total processor hog, on purpose. I basically wanted to use it to push the limit of what could be done in the player, so I kept adding more filters and processors to it until the audio started to break up on my Core Duo 2 Mac Book Pro and then stepped back just a little. It turns out that you can actually do a ton of audio processing interactively in the player leveraging Pixel Bender. This was also designed to run as an AIR app, which means that if you really want to play with it, close all your other tabs. Really.
Pixel Bender Synth Experiment
I’ll be posting all the code on the page eventually.
Some techy details:
- Each Pixel Bender shader is encapsulated within an Actionscript class and launched via a ShaderJob.
- Each Pixel Bender filter processes two samples at once to double throughput.
- The synth uses three Pixel Bender Shaders as does the Reverb, the other filters use one.
- The delay-based filters (reverb/delay) are using simple bit-bucket style delay in actionscript, but the delayed signals are mixed back with the main signal via Pixel Bender shaders.
- All signal paths are stereo. When mixing two stereo inputs, a pixel4 representing the 2 sets of stereo channels is used. When processing/generating a single stereo signal, a pixel4 representing two sequential stereo samples is used.
- The filters are ordered as such:
Will post more info soon. Ask any questions in the comments. Thanks!