On the beach… (part 4)
For previous Flash on the Beach posts in this series, see On the Beach (part 1), On the Beach (part 2)., and On the Beach (part 3)
The second afternoon session had Jared Ficklin showing us how one can visualize sound in nature and code. Jared began the session with demonstrating how sound physically manifests itself to give us an idea of the sort of properties of a sound wave you could apply to code. Jared used some braniac style practical examples to help us visualize this; for instance he blasted out some music into some artificial smoke to demonstrate sound waves moving through the air.
Rubens Tube
Jared then took us through a Rubens Tube ‘fire’ example. This experiment essentially involves a tin foil lines PVC pipe, with a hundred or so holes drilled in and a speaker at one end matching the diameter of the pipe. At the other end of the pipe is a connection to a propane tank, feeding propane gas into the tube. After lighting it up, a mild and steady oscillation of flame appears through the holes in the pipe, whilst blasting some music or sound through the other end causes the vibrations in the tube to allow more gas through in some holes on the pipe, and restricting it in others, forming sound wave int he flame. Since the experiment involves open flame he was not able to demonstrate a live version, so he played this video:
Needless to say it was a lot of fun to watch, and served as quite a nice visual way to visualize the amplitude of different frequencies in sound.
Visualization in Code
Jared used flash and AS3 code as a bridge to demonstrate how some of the properties inherent in a sound wave can be translated into code and applied to various visual properties in flash. He showed us some of his own visualization ideas, and also some quite impressive and rather original ideas from Annika Hamann. Here Annika uses properties of the sound such as amplitude and frequency to affect the animation and timing cycles of various flash animated cartoon characters. Jared demonstrated how you could easily affect flash properties such as movieclip position, scale, alpha, tint etc with influence from sound data.
AS3 has made it more simple than ever to extract various sound data from a sound wave using the SoundMixer class. You can see a first experiment I have attempted myself after being inspired by Jared’s fire example on my blog post here. It combines some particle system experiments and affecting the movement of the particles with the beat of the audio. You can easily extract the data from a sound wave into a byte array in AS3 like this:
var PLOT_HEIGHT:Number = 200; //size of area to animate in
var snd:Sound = new Sound();
var req:URLRequest = new URLRequest("your track url");
var channel:SoundChannel;
var bytes:ByteArray = new ByteArray();
snd.load(req);
this.addEventListener(Event.ENTER_FRAME, enterFrameEvent);
channel = snd.play(0,3);
function enterFrameEvent(event:Event):void {
SoundMixer.computeSpectrum(bytes, true,0);
// left channel
for (var i:Number = 0; i < 256; i++) {
val = (bytes.readFloat() * PLOT_HEIGHT);
}
}
The sound mixer’s computeSpectrum function stores the audio data as normalized floating point values between -1 and 1 for a default range of 512 into the byte array on each on enter frame event. The above example code will do the first 256 values for the left channel. You can then cycle through this array and pull out the values as you see fit. If the second parameter of the computeSpectrum function is set to true than the function will perform a Fast Fourier Transform to convert it into its frequency domain before storing the values. Anyway I don’t want to fill this with details on computeSpectrum but for more details there is an excellent tutorial here.
Stephen Hawking’s Universe
Jared showed us an experiment he had been working on at the end of the session, where he streamed an audio recording of one of Stephen Hawking’s Cambridge University lectures into one of his flash experiments. The idea of the algorithm is that as the audio recording progresses a line is traced zig-zagging upwards across the screen with the x and y changing based on the variances in Stephens voice, and after each sentence it detects the pause and draws a star, with the brightness being dictated by the length of the sentence. Jared then showed us how he had created a (Stephen Hawking’s) ‘Universe’ of stars against a black space backdrop. It was very impressive an another interesting original way of visualizing sound, in a manner far removed from the usual Windows Media Player and iTunes standard visualizations. Jared then impressed everyone again when he demonstrated how fellow Flash on the Beach speaker Mario Klingemann inspired him to make it interactive, in that you can click on a specific star and it’s associated sentence plays out!
Anyway, Jared has posted a nice video of the process on YouTube:
