Thursday, May 22, 2014

How to keep an installation running

Writing a program that can make it through the length of a single 20 minute performance can sometimes be challenging, but installation code needs to run for a full day or sometimes weeks and keep going. The first step, of course, is to make your code as bug-free as possible. However, even in this case, your code will eventually crash, though no wrong doing of your own. Therefore, the thing to do is to recover gracefully.

The latest versions of SuperCollider are actually three programs running at once - the IDE; sclang, the language; and scserver, the audio server. Any one of these things can crash.

SkipJack

SkipJack is an object that is like a Task, except that it survives cmd-period. It takes several arguments, the first of which is a function and the second of which is a number indication the wait time in between executions of the function.

SkipJack({"SkipJack".postln}, 2)

This code would print out SkipJack once every two seconds. The advantage of using SkipJack in this case is not that it will keep going after a comand-periiod, but rather that it's on a different thread than the rest of your program. If your main execution loop gets stuck some place and effectively dies, SkipJack will likely carry on. Therefore, we can use SkipJack to check on your main loop and try to revive it.

Sclang

How can we tell your main thread is still running without also stopping if it stops? One way to check is by looking at a shared variable. Let's have a number. Every time we go through the loop, we can set it to 3. Meanwhile, SkipJack could be running with a duration roughly equivalent to how long it should take to get through our loop. It could subtract one from our number. If our main execution loop stops, that number will count down towards zero and then go negative.


var alive, click, dur, task;

dur = 2;
click = { alive = 3 };

task = Task({
inf.do({

"still allive".postln;
click.value;
dur.wait;
})
}).play;

SkipJack({
"are we alive?".postln;
(alive <=0).if({
task.resume;
task.play;
"play!".postln;
(alive <= -2).if({
1.exit;
})
});
alive = alive -1;
}, dur);

If alive gets to zero, first we try to get the task running again. This sometimes works. If it fails, however, we call 1.exit, which causes all of sclang to exit. If we can't recover inside sclang, we can recover outside it.

The Server

We'll need a separate loop to check on the server.


SkipJack({
Server.default.boot(onFailure:{});
Server.default.doWhenBooted({}, onFailure:{1.exit});

}, dur);

This may look odd because it changes the 'onFailure' argument, but the effect of it is that if the server is not already booted, it will take a moment and may struggle to boot. If it fails, all of SuperCollider exits.

Keeping a Synth Alive

If your loop is firing off new Synths, you don't need to worry about whether each individual synth keeps going, but if you're just changing parameters on an individual synth that keeps running, you also need to watch out for it perishing. there are a few ways to do this. Maybe you want to check if it has gone silent?


(
var syn, lastrms, max_silence;

SynthDef(\stereoListenForSilence, {|in=0, out=0|
var input;
input = In.ar(in, Server.default.options.numOutputBusChannels);
SendPeakRMS.kr(input, 1, 3, '/loudnessMonitoring'); // send the RMS once per second
ReplaceOut.ar(0, LeakDC.ar(input).tanh); // Optional line to get rid of offset and peaking
}).add;

/* ... */


Synth(\stereoListenForSilence, nil, RootNode(s), \addToTail);

max_silence = 10; // 10 seconds


lastrms=Array.fill(max_silence, {1});

osc_listener = OSCFunc({ |msg|
var rms;
rms = msg[4].asFloat.max(msg[6].asFloat);
lastrms.removeAt(0);
lastrms.add(rms);
(lastrms.sum <= 0.0001).if ({
"too quiet".postln;
// retsart your synths
s.freeAll;
Synth(\myAmazingSynthDef);
Synth(\stereoListenForSilence, nil, RootNode(s), \addToTail);
});
}, '/loudnessMonitoring');


You can put a monitoring Synthdef on the server's root node and use SendPeakRMS to send OSC messages with the overall amplitude of all running synthdefs. Then, set up an OSCFunc to check if the peak amplitude has been near zero for too long. If it has, free everything and put up new synths. This will not tell you if your server freezes or if your monitoring synth stops sending OSC messages.

Or if you just want to check if an individual Synth is still running, you can use OSCFuncs and SkipJack together.


(

var syn_alive, dur;

dur =1;
syn_alive = 3;

SynthDef(\myAmazingSynthDef, {

var sin, trig;
SendTrig.kr(Impulse.kr(dur.reciprocal));
sin = SinOsc.ar;
Out.ar(0, sin);

}).add;


/* ... */

Synth(\myAmazingSynthDef);

OSCFunc({ arg msg, time;
syn_alive = 3;
},'/tr', s.addr);


SkipJack({
(syn_alive <=0).if({

s.freeAll;
Synth(\myAmazingSynthDef);

(syn_alive <= -2).if({
1.exit;
})
});
syn_alive = syn_alive -1;
}, dur);


SkipJack({
Server.default.boot(onFailure:{});
Server.default.doWhenBooted({}, onFailure:{1.exit});

}, dur);
)

Try quitting the Server via the gui and everything gets back to where it was in under 3 seconds.

No IDE

Previously in this document, we've intentionally made sclang crash, which, if you're running the IDE, is no fun. However, we will not be running sclang through the IDE. Instead, we'll run it from a BASH script. On your computer (if you have a mac or unix), when you open a terminal, what's running in it is a Bash shell. You can write scripts for this shell, which you edit in a plain text editor.


#!/bin/bash


while true
do

/path/to/sclang installation.scd

sleep 1

killall scsynth

sleep 1

done


Ok, so first things first, open a terminal and type:

which bash

It may be /bin/bash or /usr/bin/bash. It's what you want in the first line of the bash script. So if you get back /usr/bin/bash, change the first line to #!/usr/bin/bash.

To find the path to sclang, you can try typing 'which sclang', but if you're on a mac, this is not going to work. Instead, you will need to find the SuperCollider application on you hard drive. Right click on it to examine package contents. If you poke around in there in folders called things like Resources or MacOs, you will eventually find sclang. Drag that file to your SuperCollider IDE to find out the path for it. Put that path into your bash script in the place of '/path/to/sclang'.

Save the script as installation.sh and save your supercollider file as installation.scd. Put them together in the same folder or directory. In your terminal window, cd to that directory and type:

chmod +x installation.sh

Or, alternately, if you'd prefer to use a GUI, get information about the installation.sh file and click a checkbox to make it executable.

What this script does is loop forever. It runs your program. When your program exists, it kills whatever server might still be running and then restarts your program. If your program crashes out entirely, this script will restart it.

Helper Apps

If your installation relies on a helper application, like let's say you're using PD to manage HID communications, you'll want that in your loop also, but as both applications are running at the same time, you'll need to run the helper application in the background, but keep track of the PID so you can kill it when your program crashes.


#!/bin/bash


while true
do

/path/to/pd hid_helper.pd &
pid=$!
/path/to/sclang installation.scd

sleep 1

killall scsynth
kill $pid

sleep 1

done

Make sure your hid_helper is in the same folder as as your other files. Find the path to pd in the same way you got the path to sclang. The & makes it run in the background and the next line tracks the PID, so you can kill it later.

Obviously, you'll also want to keep track of your helper application, which you can do via OSC in the same way you keep track of your synthdefs. If your helper application quits, you'll need to do a 1.exit to force the bash script to restart everything.

Making it all go

This is an installation, so if you're using your own laptop, don't run it as yourself. Make a new user account and don't give that user administrative rights. Make all your files READABLE by that user (or everyone on the system), but don't give that user write access. Set the system so that when it boots up, it automatically logs in as that user.

Log in to the system as the new user. Go to the settings and say you want to autostart an application on login. The application you want to autostart is installation.sh

Try booting your computer. Does it start your installation? Once you've got that sorted out, leave it running in your living room for days and days until you think you're losing your mind. Every so often, use the process manager to kill the server or the helper application or wiggle a wire or otherwise create a bit of problems and see if your installation survives.

1 comment:

Anonymous said...

Very useful, thank You!
I always struggle with things crashing even in studio, a live performance would definitely need those tricks