Audio reactive visuals

There are several ways to create audio-reactive visuals in Punctual: using oscillators whose frequency is synchronized with the music tempo, using the intensity of predefined audio frequencies bands, or using the Fast Fourier Transform of the captured audio.

Time and tempo

Punctual don’t use the common tempo mesure of BPM (beats per minute) but a specific measurement called CPS: cycles per second. A cycle is similar to a measure in music.

If you are making visuals to go with music and the music follows a 4/4 time signature, a Punctual cycle would be equivalent to four beats.

To translate from cycles per second to beats per minute, this formula can be used, assuming a 4/4 time signature:

BPM = CPS*60*4

And from BPM to CPS:

CPS = BPM/60/4

The standalone version of Punctual has a fixed tempo of 0.5 CPS (120 BPM in a 4/4 time signature).

In Estuary, you can see the current tempo with !showtempo and change the tempo by using the terminal command !setcps (for example, !setcps 0.542 for 130 BPM), or by taping the tempo (see below).

These functions can be used to get information about the time passed, and the current tempo:

cps

The current tempo in cycles per second. Usually used as oscillator’s frequencies in order to synchronize them with the music beats.

In this simple example, we use two cells from Estuary. One of them has the Punctual code:

fit 1 $ circle 0 (0.6-0.5*saw (cps*4)) >> add;

And the other the MiniTidal code:

s "bd*4"

Note how the sound and visuals are synchronized.

time

Returns how much time in seconds has passed since “beat 0” of the tempo. In the standalone version of Punctual beat 0 is when you load the web page; in Estuary beat 0 can be anytime in history, but is usually the time at which a collaborative ensemble was created.

In Estuary, you can use the command !resettempo to set cps to 0.5 and time to 0.

beat

How many cycles have passed since “beat 0” of the tempo. Note that despite its name, beat refers to cycles, not beats.

Let’s try to visualize this. Next example is meant for Estuary:

hline (time % 1) 0.01 >> red;
hline (beat % 1) 0.01 >> green;
hline ((beat*4)%1) 0.01 >> blue;

The red line counts seconds. Each time it returns to the center, one second has passed.

The green line counts cycles. As the default tempo is 0.5 CPS, the green line resets every two seconds.

The blue line counts beats in a 4/4 time signature.

To test it with sound, you can take another cell, select MiniTidal as language in the drop-down menu, write and evaluate s "bd".

Now, a bass drum will sound at the beginning of each cycle, at the same time the green line resets.

If you write s "bd*4", there will be a bass drum sound at each beat, synchronized with the blue line.

Now, if you change the tempo with !setcps 0.2, note how the sound, and the blue and green lines slow, but the red line keeps the same pace.

etime

Similar to time, but it returns how much time in seconds has passed since code was last evaluated.

ebeat

Similar to beat, but it returns how much time has passed since code was last evaluated, expressed in cycles.

hline (etime % 1) 0.01 >> red;
hline (ebeat % 1) 0.01 >> green;
hline ((ebeat*4)%1) 0.01 >> blue;

This is the same as before, but using etime and ebeat. Note that now the visuals aren’t synchronized to the bass drum, and every time you run the code, all lines are reseted.

Continuous drawing

As seen in the oscillators section, these functions are related to the oscillators phase, and many times it’s easier to just use an oscillator. However, I found them quite useful to draw geometrical patterns in the style of mandalas.

The trick here is set feedback to one, and use a point or a little circle as a pen. Thanks to rtxy, we can think in terms of radius and angle. The angle simply moves forward. It’s in the angle where we can apply many variations to create different drawings:

fit 1 $ circle (0.8*rtxy [(sin' $ 0.5*pi*etime), etime]) 0.01 >> add;
fb fxy >> add;

fit 1 $ point (0.8*rtxy [0.1+(sin' $ pi*etime), etime]) >> add;
fb fxy >> add;

fit 1 $ point (rtxy [0.2 ~~ 0.8 $ (sin' $ pi*1.3*etime), etime]) >> add;
fb fxy >> add;

Replicating the pattern with tile and using polar coordinates we can create amazing patterns:

fit 1 $ setfxy [fr,ft*pi] $ tile [2*pi,pi] $ circle (0.8*rtxy [(sin' $ 0.5*pi*etime), etime]) 0.04 * [1,0,1] >> add;
fb fxy >> add;

Note the use of pi inside tile to force the pattern to fit when curving it later with setfxy.

By playing with the feedback instead of keeping it static, our patterns gain dynamism, like in this example that resembles see waves:

(fit 1 $ setfxy [fr,ft*5] $ tile [2*pi,pi] $ circle (0.8*rtxy [(sin' $ 0.5*pi*etime), etime]) 0.08) * [0.3,0.3,1] >> add;
move [-0.003,0] $ fb fxy >> add;

There are some more examples using beat and cps in the colors section.

Tap tempo in Estuary

In addition to set the tempo by the command !setcps, it’s also possible to set it by taping a button. This is a hidden feature and you have to manually change the view layout in order to make this button visible.

Use the command !localview to change the layout.

For example:

!localview $ grid 1 1 [[taptempo,label 1,code 2 0 []]]

This sets a single cell and shows the tap tempo button.

!localview $ grid 2 1 [[taptempo,label 1,code 2 0 []], [label 3,code 4 0 []]]

Same than before, but with two cells, in two columns.

!localview $ grid 2 3 [[label 1,code 2 0 []],[label 3,code 4 0 []],[label 5,code 6 0 []],[label 7,code 8 0 [],taptempo],[label 9,code 10 0 []],[label 11,code 12 0 []]]

The default view, but adding the tap tempo button under the forth cell.

Use !presetview def to reset the view layout to the default.

Tap tempo is very useful when performing visuals live with external sound, as an easy way to synchronize the internal Estuary tempo to the outside music without the need of any additional setup.

To tap the tempo you have to click the button 9 times in a row, starting on a downbeat (first beat of the bar), 4 beats per cycle (so you will “hit” three downbeats in performing the tapping).

Frequency analysis

lo, mid, hi, ilo, imid, ihi

There are six functions that analyze a particular frequency band of the sound and returns a number between 0 and 1.

For internal sound (produced by Punctual or by any of the supported languages inside Estuary): lo, mid and hi, corresponding to low, middle and high frequencies respectively. For external sound (captured by a microphone or an audio interface): ilo, imid and ihi.

l << spin (saw 0.02) $ tile [16,32] $ (vline 0 0.1 +: hline 0 0.1);
setfx [fx+fy*imid, fx-fy*imid] $ move [-1,0] $ setfxy [fr*2, ft*2] $ l * 0.25*[fx*ilo,fy*imid,ihi] >> add;
0.8 * fb fxy >> add;

fft, ifft

These two functions can be used to obtain a complete frequency analysis of the internal (fft) or external (ifft) audio using the Fast Fourier Transform.

Both functions need a graph as an argument. Audible audio frequencies are mapped to the range 0 - 1. For each fragment, the argument value is computed, and the intensity of the frequency corresponding to this value is returned.

ifft 0.5 >> add;

Here, all fragments will show the intensity of a middle frequency.

ifft (abs fx) >> add;

All the frequencies spectrum mirrored on the vertical axis.

o << osc [0.04,0.041];
x << o*[-1,-0.9..1];
l << mono $ setfy [fy,(-1)*fy] $ iline [0,-1] [x,1] 0.001;
l * [ifft $ abs fx,0,ifft $ fit 1 $ fr] >> add;

Here, we create some vertical lines using the Haskell shortcut for creating lists ([0.1,0.17..0.8]). This vertical lines are then modified several times. First, for each fragment, we change its x-coordinate according to an audio frequency that depends on the absolute value of the y-coordinate.

The linear rescale and other values are used to adjust the amount of deformation, mono is used to keep all the image white even though it has two channels, and the [0.3,-0.3] part doubles the transformation creating a left-right symmetry (this is the bit that creates two channels):

mono $ setfx [fx+[0.3,-0.3]*(ifft $ linlin [0,1] [0.1,0.5] (abs fy))] $ vline [0.1,0.17..0.8] $ px*0.5 >> add;

After that we apply a second transformation that simply rescale the y-axis. This is to better distribute frequencies on the next step (try to remove the setfy (fy/pi) part to see the effect). Next, we apply another transformation that interprets x,y Cartesian coordinates as r,t in polar coordinates, converting the vertical lines into circles. fit is used to avoid getting ovals and not circles due to the aspect ratio. Finally, a bit of feedback is added to enhance the result:

fit 1 $ setfxy frt $ setfy (fy/pi) $ mono $ setfx [fx+[0.3,-0.3]*(ifft $ linlin [0,1] [0.1,0.5] (abs fy))] $ vline [0.1,0.17..0.8] $ px*0.5 >> add;
0.8 * fb fxy >> add;

So far, all our examples have utilized a continuous form of the Fast Fourier Transform (FFT). However, we can also discretize it, which means we only consider a finite subset of the frequency intensities instead of all of them.

In this example, we generate a series of horizontal segments where the length is determined by the intensity of its associated frequency.

The y coordinates are uniformly distributed along the range [-1, 1]. This is achieved by creating a list ranging from 0 to 32, dividing each element by 32, which results in the list 0, 1/32, 2/32, and so on, and then transforming the range [0, 1] to [-1, 1] using the bipolar function.

Similarly, the x coordinates are constructed using the list 0, 1/32, 2/32, etc., which is then passed through the ifft function to obtain the intensities of these 33 uniformly distributed frequencies. The resulting values are then multiplied by -1 to get the negative counterparts, allowing the segments to be symmetrical across the y-axis.

The final segments are created using the linep function and stored in l. To add color, we manipulate the distance of each fragment from the y-axis. This determines the shading, transitioning from green in the central parts to red as the fragments move further away.

y << bipolar $ [0,1..32]/32;
x1 << (ifft $ [0,1..32]/32);
x2 << (-1)*x1;
l << mono $ linep (zip x1 y) (zip x2 y) 0.008;
l*[abs fx, 0.5 - abs fx, 0] >> add;

What if you want your visuals to react only to a certain frequency? My tests indicate that this can be achieved by dividing the desired frequency by 24000.

So, if for example, you want to flash your screen every time an A in the forth octave is heart (A4), and knowing that this pitch has a frequency of 440, you could do something like:

gatep 0.8 $ ifft (440/24000) >> add;