August 21, 2010
Sambaverse by Ben Gomez Farrell
One post that got people excited this week was the announcement of the Sambaverse alpha from Ben Gomez Farrell. This is Ben’s description of what he is trying to accomplish: “I needed a tool that was smart enough to load an MP3, take a good stab at automatically detecting beats, break, loud sections, and different sections of the song – like verses and choruses.” And Pixel Bender? “I used Pixel Bender to get a huge speed increase when loading the sound initially. That was a lot of fun once I figured out how to get it working.”
Color keying with Flash Builder 4 by Luis Estevez
Luis Estevez created a sample of using Pixel Bender to do color keying in the player. Here is his post in Spanish and English.
Flex 4 Fun by Chet Haase
Chet Haase has written a book called “Flex 4 Fun” that includes at least a chapter on Pixel Bender.
"Pixel Bender: A Gradient… and Some Math" from David Van Brink
Here is an older link, but one I just came across this week, from David Van Brink’s Omino Pixel Blog: “Pixel Bender: A Gradient… and Some Math.” I really like how he digs into the math to explain how it works. A great blog, and a nice filter too!
It is pretty amazing that every week I have new links to post from Pixel Bender developers. I’m going to try and work in some of the old links that I’ve got too as I go too. More links next week!
April 23, 2010
The subject of using Pixel Bender for audio processing on the Flash platform comes up a lot. Audio processing is very processor-intensive and math-heavy, so it would seem natural to leverage Pixel Bender to improve performance of audio within a SWF. At some point, last year, I was talking to Justin Everett-Church about doing a demo for Flash Player 10.1 multi-touch features. A synth seemed like a good idea, and it would let me kill two birds with one stone. So I coded up a synth (with a lame Flex UI) and Justin took that and made it pretty and added multi-touch support. That 2nd part of the demo never worked out the way we meant it too (missed the MAX 2009 keynote by this much). After that, I planned on cleaning up the code and posting it, but I got busy shipping CS5 and well, 7 months later, I finally got around to posting it. Right now it is just the playable demo with the crummy Flex UI.
It is a total processor hog, on purpose. I basically wanted to use it to push the limit of what could be done in the player, so I kept adding more filters and processors to it until the audio started to break up on my Core Duo 2 Mac Book Pro and then stepped back just a little. It turns out that you can actually do a ton of audio processing interactively in the player leveraging Pixel Bender. This was also designed to run as an AIR app, which means that if you really want to play with it, close all your other tabs. Really.
Pixel Bender Synth Experiment
I’ll be posting all the code on the page eventually.
Some techy details:
- Each Pixel Bender shader is encapsulated within an Actionscript class and launched via a ShaderJob.
- Each Pixel Bender filter processes two samples at once to double throughput.
- The synth uses three Pixel Bender Shaders as does the Reverb, the other filters use one.
- The delay-based filters (reverb/delay) are using simple bit-bucket style delay in actionscript, but the delayed signals are mixed back with the main signal via Pixel Bender shaders.
- All signal paths are stereo. When mixing two stereo inputs, a pixel4 representing the 2 sets of stereo channels is used. When processing/generating a single stereo signal, a pixel4 representing two sequential stereo samples is used.
- The filters are ordered as such:
Will post more info soon. Ask any questions in the comments. Thanks!
August 19, 2009
I’ve been doing some playing around with processing audio using Pixel Bender in Flash and I realized that it was hard to find some working code to get started with. So i wrote up this sample. I tried to do a minimal app that actually did something interesting and would be a start for someone else. To that end, this AIR app sample loads an MP3 file and then the embedded Pixel Bender kernel lets you change the level of the individual channels separately.
The MXML code is below:
All the action is in the ProcessAudio function, that pulls samples from the input file and executes a ShaderJob across them. There is something important to reference here:
effectShader.data["source"].width = BUFFER_SIZE / 1024;
effectShader.data["source"].height = 512;
effectShader.data["source"].input = shaderBuffer;
effectShader.data["volume"].value = [ leftSlider.value, righttSlider.value ];
var effectJob:ShaderJob = new ShaderJob( effectShader, event.data, BUFFER_SIZE / 1024, 512 );
I pass the buffer into the shader job as a 2D buffer instead of a buffer with a height of one. This may make less sense logically, but the Flash player breaks the data up by rows for multi-threading, so this should make that perform faster.
Here is the kernel:
One thing to notice here is that rather than using an image2 as input and a pixel2 as output (which may make more sense logically again), I instead just use the buffer layout and process 2 stereo samples at the same time. This should also give you better performance for filters that can do this.
Here are the files:
For more info, the following references might be helpful