Requirements

We had to meet the following requirements :

• the system needs to be wireless, worn by the dancer and should not embarrass him in any way
• the system needs to be placed on each shoe (no cable running on the dancer’s leg, or belt case)
• the system should remain efficient at a 6-7 meter range from the speakers
• a very low latency is critical (max 30ms)
• the whole system should handle up to 6 shoes simultaneously
• no fake: the sound should be genuinely generated by the dancers’ steps

Hardware and assembly

The concept is rather straightforward: I needed two force sensors for each shoe (one for the heel and one for the toe), and a wireless transmitter per shoe to capture the pressure applied on each of the sensors.

I used the Force Sensitive Resistor from Interlink [http://www.sparkfun.com/products/9376]. They have the right square size, are super-thin, and accurate enough to sense the pressure of a dancer’s foot on the ground.

As for the transmitter, I just had a size constraint: I wanted something really small, capable of handling two analogical inputs: one for the heel and one for the toe.

Only thin components such as the sensors can be placed under the internal sole of the shoe.

The first transmitter prototype I built was using a BlueTooth chip (class 3). Although the result was satisfactory at half-a-meter distance from the receiver, the latency increased with the distance. At 7 meters, the latency went as much as 1/8th second.

Then I gave XBee technology a try: these wireless modules have very narrow settings, a low power consumption, and I was confident I could reach a minimum latency at a decent distance.

Once I had solved the range / latency issue, one of the major challenges was to make the XBee module so small as to be wearable on a shoe. As we didn’t have the budget to industrialize the process, all of the assembly was made by hand

Below the two final prototypes before and after building optimization :

On each shoe, the module included:

• A 6-pin dock to connect the sensors and charge the battery
• A status LED
• A switch
• A 110 mah liPo battery
• A voltage regulator
• A XBee chip
• Some resistors

Finally, a receiver is connected to the computer to gather the signals from all the XBee chips.

The XBee chip (the emitter) can be programmed so that it emits on its own at a certain frequency both analogical inputs to another XBee chip (the receiver).

The receiver is a XBee chip plugged on a USB dongle. At the beginning, I had a single Xbee chip to handle the signal of the 6 emitters. I became quickly aware of  performance and latency issues, due to colliding packets. To solve this, I put three XBee chips as receivers. Each chip was in charge of handling two emitters and the collision/latency issue was gone.

The software

As for the software, two separate programs had to be developed:

• a server processing the receivers’ inputs (Processing)
• an AIR application that interprets the signals, chooses and play the sounds.

The communication between the processing server and the air application is achieved by a socket server hosted by the processing server.

The signal analysis was not an easy task. My intention was to assess when the heel or the toe was hit. The major difficulty is that when the dancer dances or even stands, there can be significant shifts of balance from one sensor to another, without an actual impact having really happened.

I created a rather simple algorithm, which you’ll find the basics below:

• evaluate the average pressure applied on the sensor during the last second
• evaluate the slope between the current average pressure and the average pressure applied ‘n’ milliseconds ago
• if the slope matches a specific percentage, assume a positive impact
• lock the sensor during ‘n’ milliseconds to avoid any duplicates

The application offered the following options:

• loop a sound after one single impact
• the ability to stop the sound of sensor X when sensor Y was triggered
• mute the whole shoe when two sensors were hit simultaneously (for a short duration).

To design the frontend of the AIR application, I used the Minimal Components of Keith Peters.

Photos

Conclusion

I did face loads of technical and hardware problems during the production and tests. Generating live music requires very low latency, and creating a shoe-wearable wireless system was a real challenge.

The dancers that operated on the video clip did train a lot to succeed in synchronizing to play the actual song.

Aside from the technical standpoint, it has been for me a great and fulfilling experience to discover the dance environment.

Credits

Conception : SID LEE Paris

Film : YAK Films

Dancers &  Singer : Lamine, Laurent & Larry (Twins), Mounir, La Fouine.

Thanks to Keith Petters Minimal Components [http://www.minimalcomps.com/]

Thanks to my brothers Fabien & Fabrice for their help.

If you have any questions on programming XBees, on the processing server or anything else, share them with me in the comments: I am no electronics expert, but I did learn a few things on that project

Links

Sparkfun shopping list  : http://www.sparkfun.com/wish_lists/13870

Making-of full res photos pack : http://sidlee.didierbrun.com/adidas/megalizer_photos.zip

Emitter electronic scheme PDF  : http://sidlee.didierbrun.com/adidas/megalizer_electronic_scheme.pdf

Processing server sources : http://sidlee.didierbrun.com/adidas/megalizer_processing_sources.zip