August 21, 2009

## Matrices and Pixel Bender

This is a guest post from Bob Archer, a Development Lead in the Adobe Image Foundation team. I ofter see people doing a lot of extra math in their kernels when they could have simply done a matrix multiplication. This post won’t explain linear algebra to you, but if you’ve been confused by Pixel Bender’s column major order or aren’t sure how it works, it should hopefully give you some pointers… [End of intro]

I thought this would be of interest to everyone – I finally have the definitive guide on how matrix math works within Pixel Bender.

Pixel Bender matrixes are in column-major order. This means that this line:

`float3x3 m = float3x3( a, b, c, d, e, f, g, h, i );`

Sets up a matrix that looks like this:

```a  d  g
b  e  h
c  f  i```

If I access a matrix using a single [] operator I get these results:

```m[ 0 ] == float3( a, b, c );
m[ 1 ] == float3( d, e, f );
m[ 2 ] == float3( g, h, i );```

If I set up a vector like this:

`float3 v = float3( X, Y, Z );`

and do some vector / matrix or matrix / vector multiplications I get these results:

```v * m == float3( Xa+Yb+Zc, Xd+Ye+Zf, Xg+Yh+Zi )
m * v == float3( Xa+Yd+Zg, Xb+Ye+Zh, Xc+Yf+Zi )```

i.e. v * m does this calculation:

```( X  Y  Z  )  *  a  d  g
b  e  h
c  f  i```

while m * v does this calculation:

```a  d  g     X
b  e  h  *  Y
c  f  i     Z```

If we are multiplying two matrices together:

```float3x3 m1 = float3x3( a, c, b, d, e, f, g, h, i );
float3x3 m2 = float3x3( J, K, L, M, N, O, P, Q, R );```

m1 * m2 does this calculation:

```a  d  g     J  M  P      aJ+dK+gL  aM+dN+gO  aP+dQ+gR
b  e  h  *  K  N  Q  ==  bJ+eK+hL  bM+eN+hO  bP+eQ+hR
c  f  i     L  O  R      cJ+fK+iL  cM+fN+iO  cP+fQ+iR```
August 19, 2009

## Pixel Bender audio processing sample code

I’ve been doing some playing around with processing audio using Pixel Bender in Flash and I realized that it was hard to find some working code to get started with. So i wrote up this sample. I tried to do a minimal app that actually did something interesting and would be a start for someone else. To that end, this AIR app sample loads an MP3 file and then the embedded Pixel Bender kernel lets you change the level of the individual channels separately.

The MXML code is below:

All the action is in the ProcessAudio function, that pulls samples from the input file and executes a ShaderJob across them. There is something important to reference here:

```effectShader.data["source"].width = BUFFER_SIZE / 1024;
effectShader.data["volume"].value = [ leftSlider.value, righttSlider.value ];

effectJob.start(true);
```

I pass the buffer into the shader job as a 2D buffer instead of a buffer with a height of one. This may make less sense logically, but the Flash player breaks the data up by rows for multi-threading, so this should make that perform faster.

Here is the kernel:

One thing to notice here is that rather than using an image2 as input and a pixel2 as output (which may make more sense logically again), I instead just use the buffer layout and process 2 stereo samples at the same time. This should also give you better performance for filters that can do this.

Here are the files:

August 7, 2009

## A proposal on semantic hinting in Pixel Bender metadata

We tried to be semantically agnostic in the original design of the Pixel Bender language. We’d seen other languages go down rabbit holes of over-specification around what parameters really meant, locking them into archaic and insufficient implementations or clumsy hierarchies of meanings. We didn’t want to limit Pixel Bender developers into what we could conceive at the time of the invention of the language.

We were being a bit too idealistic :). It is completely true that the community of Pixel Bender developers continues to blow our minds at what they accomplish with the language and we never would have anticipated half the stuff that you guys are using Pixel Bender for. However, having some generally useful semantic meanings for Pixel Bender parameters would definitely help those who design general user interfaces for Pixel Bender filters.

One smart thing we did (if I do say myself) was to allow parameter and kernel metadata to be extensible. It provided developers like After Effects and Picnik a way to add custom metadata to Pixel Bender that specified semantics or actual UI controls for Pixel Bender kernels and graphs. The picnik metadata and the After Effects metadata are different though. I started to get concerned that by not having something around this that we could end up with many different Pixel Bender semantic metadata mechanisms floating around. To that end, I created the proposal below and started floating it around Adobe and some of the sites that are heavy users of Pixel Bender.

In this design, I tried to follow some general rules:

• Pixel Bender is not only a way to write image and video filters for Adobe applications, but also a way for you to host 3rd party filters in your Flash-based applications. Any guidelines we create need to be implemented by us, but also by any independent Flash developers creating apps that use Pixel Bender kernels. To that end, I tried to keep the design relatively simple so that it wouldn’t be too difficult to implement in Flash.
• This proposal adds semantic metadata, but avoids specifying specific user interface controls. Rather than specifying a slider as an editor for a parameter, I think it makes more sense to say that the parameter is a percentage and allow someone designing a UI to make a good percentage editor. We are definitely thinking about how to specify custom editor UIs for Pixel Bender filters, but this proposal does not approach that.
• The proposal is not a complete answer for all Pixel Bender filters. I’m trying to get the most universal semantics represented. The most generally useful, as opposed to trying to give the complete solution. If you have an application that is uses Pixel Bender kernels for something that makes sense to augment the metadata you can still choose to do so.
• The proposal represents guidelines, not requirements. As a developer consuming Pixel Bender kernels, you can choose to enumerate the metadata as described in the proposal or not. The suggested metadata is metadata. It is not required. The goal is that if you wish to take advantage of it when it is present, that you can provide a more compelling user interface to your users because you understand the intent of a parameter, not just its type.