autoradio

The autoradio service aims to provide a reliable, fault-tolerant Icecast streaming service for audio and video. It provides all the necessary components to ensure that the traffic from the source to the clients is uninterrupted, even in face of high load or server crashes. All this, if possible, without any operator intervention.

It is a full-stack service, meaning that it includes its own DNS and HTTP servers, for full control of the request flow.

Autoradio works by using etcd to coordinate the various nodes and store the global mount configuration. The intended target is a set of homogeneous servers (or virtual machines) dedicated to this purpose. Autoradio also needs a dedicated DNS domain (or a delegation for a subdomain).

Installation

The simplest installation method is probably to use the pre-built Debian packages (only available for amd64 at the moment), by placing this line in /etc/apt/sources.list.d/autoradio.list:

deb http://www.incal.net/ale/debian autoradio/

And then running:

$ sudo apt-key adv --recv-key 0xC0EAC2F9CE9ED9B0
$ sudo apt-get update
$ sudo apt-get install etcd autoradio

This will install and start the necessary jobs (which will initially fail due to the missing configuration).

Edit /etc/default/autoradio and set, at least, the DOMAIN variable to what you've assigned to the cluster. The jobs will automatically start as soon as the configuration is saved.

Full cluster install procedure

Note: this procedure requires etcd 2.0 or later.

This assumes that you have an existing domain name (here example.com) that you control, and that you will run the cluster under a sub-domain (radio.example.com). The procedure will install an etcd server on each node, so it will work best for a small, odd number of machines.

Having said that, follow these steps to bootstrap a new streaming cluster:

1. Make sure that, on each of your servers, the output of hostname -f is the fully-qualified hostname of the machine, and that it resolves to its public IP (possibly using /etc/hosts).

2. On every server, run the above-mentioned steps to set up the APT repository and install (do not configure) the etcd and autoradio packages.

3. Pick one of your servers and add a delegation for radio.example.com to it. For instance, with bind:

    radio  IN   NS  3600  machine1.example.com.

4. On machine1, edit /etc/default/etcd with the following contents:

    START=1
    BOOTSTRAP=1

   Once you save the file, the etcd daemon will start and initialize an empty database.

5. On machine1, edit /etc/default/autoradio and set DOMAIN=radio.example.com. This will start the radiod and redirectord daemons, and you will be able to serve DNS records for the radio.example.com zone. Check with:

    $ ping -c1 radio.example.com

   This should send a ping to machine1.

6. Set up the remaining machines. It is a two-step process: first, run the following command on the first machine:

    $ etcdctl member add machine2 http://machine2:2380

   (remember not to include a final slash on the node URL).

   This will print out some environment variables. You should copy the ETCD_INITIAL_CLUSTER line into /etc/default/etcd on machine2, resulting in something like:

    START=1
    ETCD_INITIAL_CLUSTER=machine1=http://...,machine2=http://...

   Finally, set DOMAIN=radio.example.com in /etc/default/autoradio, and the daemons will start automatically.

   Note that you will need to wait for etcd on the new machine to start successfully before you can run etcdctl member add for the next one. For further instructions on how to change the etcd cluster configuration at runtime, see the etcd documentation.

Building from source

To build autoradio from source, you should have a Go environment set up properly on your machine. Autoradio uses godep to manage its dependencies, so make sure you have that installed as well. Building autoradio should then be as simple as running, from the top-level source directory:

$ godep go install ./...

This should install the radiod, redirectord and radioctl executables in $GOPATH/bin.

Operation

In order to create a new stream (mount, in the Icecast terminology), assuming you are running autoradio on the example.com domain:

1. On any node, run:

    $ radioctl create-mount /path/to/mount.ogg

   this will output the username and password used to authenticate the source. Take note of them.

   The cluster will be automatically reconfigured with the new mount in a few seconds at most.

2. Configure the source, using the username/password provided in the previous step, and point it at the following URL:

    http://stream.example.com/path/to/mount.ogg

3. Tell your users to listen to the stream at:

    http://stream.example.com/path/to/mount.ogg.m3u

Note: some sources are unable to handle HTTP redirects: in that case, you might want to enable proxying on autoradio, and tell the client to use the direct-path URL:

http://stream.example.com/_stream/path/to/mount.ogg

DNS zone delegation

Since we can't modify the DNS glue records for the zone delegation in real-time, we have to restrict slightly the assumptions on the availability of nodes in the cluster: you have to assume that at least N of your nodes will be partially available at any one time (i.e. at least one of a chosen subset of N servers will be reachable). The number N should be fairly low, say 3. Then, you can use those 3 servers as the nameservers for zone delegation, and the other nodes are free to have dynamic membership.

Proxy

The autoradio HTTP server can operate in one of two modes:

• clients connect directly to Icecast

  When a client connects to the service on port 80, it is sent a redirect to an Icecast server on port 8000. Unfortunately some older clients (especially sources) don't handle redirects too well.

• connections to Icecast are proxied by autoradio

  Clients talk to the autoradio HTTP server, which proxies connections to the back-end Icecast servers. This way the clients only need to talk to port 80, which not only avoids using redirects but might simplify access for people behind corporate proxies and such.

  This behavior is controlled by the --enable-icecast-proxy command-line flag to redirectord.

Firewalls

The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and 8000/tcp (unless proxying is enabled) on all nodes. Nodes should be able to reach 2379/tcp and 2380/tcp (the etcd ports) on each other; these two ports can be public if you've set up X509-based authentication for etcd.

Securing etcd

In a production cluster, you will want to limit access to the etcd daemons so that only the other nodes can connect to it. While it is possible to do this with firewall rules, the dynamic membership of the cluster may make this difficult. We suggest using instead etcd's support for X509 client authentication, together with a tool to manage an online CA (such as autoca. This way, enrolling a new machine in the cluster only requires generating a new client certificate, and no other configuration.

Install the CA certificate in /etc/autoradio/etcd_ca.pem, the client certificate in /etc/autoradio/etcd_client.pem and its private key in /etc/autoradio/etcd_client.key, and the clients will connect to etcd using SSL authentication.

Instrumentation

The radiod and redirectord daemons can send runtime metrics to a statsd server (by default on localhost:8125).

Transcoding

It is possible to set up a mount to relay an upstream mount re-encoded with different parameters, using the radioctl create-transcoding-mount command. In this case, autoradio will automatically start up a process (a liquidsoap instance) to perform the re-encoding, which will connect as the mount source. A master-election protocol is used to ensure that only one such process per mount is started in the whole cluster.

Testing

There's a Vagrant environment in the vagrant-test subdirectory that will set up a test three-node cluster (with Debian Wheezy as the base system) using pre-packaged binaries. To run it:

$ cd vagrant-test
$ vagrant up

It will take a while to download the base image the first time, then it will turn up three nodes called node1, node2 and node3. Use vagrant ssh to inspect them.

If you want to test a locally-built package, copy the autoradio and etcd Debian packages in the vagrant-test directory and set the LOCAL environment variable to a non-empty string when invoking vagrant:

$ LOCAL=1 vagrant up