9.56 KB
Newer Older
ale's avatar
ale committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

The *autoradio* service aims to provide a reliable, fault-tolerant
Icecast streaming service for audio and video. It provides all the
necessary components to ensure that the traffic from the source to the
clients is uninterrupted, even in face of high load or server crashes.
All this, if possible, without any operator intervention.

It is a full-stack service, meaning that it includes its own DNS and
HTTP servers, for full control of the request flow.

Autoradio works by using [etcd]( to
coordinate the various nodes and store the global mount configuration.
The intended target is a set of homogeneous servers (or virtual
machines) dedicated to this purpose. Autoradio also needs a dedicated
DNS domain (or a delegation for a subdomain).

# Installation

The simplest installation method is probably to use the pre-built
Debian packages (only available for amd64 at the moment), by placing
this line in `/etc/apt/sources.list.d/autoradio.list`:

    deb autoradio/

And then running:

ale's avatar
ale committed
    $ sudo apt-key adv --recv-key 0xC0EAC2F9CE9ED9B0
ale's avatar
ale committed
    $ sudo apt-get update
    $ sudo apt-get install etcd autoradio-server
ale's avatar
ale committed
33 34 35 36

## Full cluster install procedure

37 38
Note: this procedure assumes a Debian distribution, it should work
either with Wheezy (oldstable) or Jessie (stable).
ale's avatar
ale committed
39 40 41 42 43 44 45 46 47 48

This assumes that you have an existing domain name (here
**) that you control, and that you will run the cluster
under a sub-domain (**). The procedure will install
an etcd server on each node, so it will work best for a small, odd
number of machines.

Having said that, follow these steps to bootstrap a new streaming

49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
1. Make sure that, on each of your servers, the output of `hostname
   -f` is the fully-qualified hostname of the machine, and that it
   resolves to its public IP (possibly using `/etc/hosts`). This way
   autoradio can detect the IP address that peers should use when
   communicating with each host. Also, for simplicity, we're going to
   assume that each host can resolve the IP address of each other just
   by using its short name.

2. Pick one of your servers, say *host1*, and add a delegation for
   ** to it. For instance, in a `bind`-formatted zone

        radio  IN   NS  3600

3. On *host1*, edit `/etc/default/etcd` with the following contents:
ale's avatar
ale committed

65 66 67 68 69 70
ale's avatar
ale committed

72 73
    Once you save the file, restart the *etcd* daemon: this will
    initialize an empty database:
ale's avatar
ale committed

        $ service etcd restart
ale's avatar
ale committed

4. On *host1*, edit `/etc/default/autoradio` and add:
ale's avatar
ale committed

ale's avatar
ale committed

81 82 83
5. Run the steps in the _Installation_ section above to set up the APT
   repository and install the `etcd` and `autoradio` packages using
   the configuration you just wrote.
ale's avatar
ale committed

85 86 87 88
   This will start the *radiod* and *redirectord* daemons, and you
   will be able to serve DNS records for the ** zone.

   Check that the service is healthy with:
ale's avatar
ale committed
89 90 91

        $ ping -c1

92 93 94 95
    This should send a ping to *host1*.

Now that the first node is up and running, set up the remaining
machines. For every host:
ale's avatar
ale committed

97 98
1. Set up etcd. First, run the following command on the first machine
ale's avatar
ale committed

        $ etcdctl member add host2 http://host2:2380
ale's avatar
ale committed

   (remember not to include a final slash on the node URL).
ale's avatar
ale committed

104 105 106 107 108
   This will print out some environment variables. You should copy
   the `ETCD_INITIAL_CLUSTER` line into `/etc/default/etcd` on
   the new host. The other lines of that file should be identical to
   what shown in step 4 of the previous checklist, replacing the host
   name where necessary.
ale's avatar
ale committed

110 111 112 113 114
   Note that you will need to wait for etcd on the new machine to
   start successfully before you can run `etcdctl member add` for the
   next one. For further instructions on how to change the etcd
   cluster configuration at runtime, see
   [the etcd documentation](
ale's avatar
ale committed

116 117 118 119 120 121
2. Set `DOMAIN` in `/etc/default/autoradio` (as shown in step 5 of
   the previous checklist above), and the daemons will start

3. Install the autoradio packages, see _Installation_ section above.
   The daemons should start automatically with the new configuration.
ale's avatar
ale committed
122 123 124 125 126 127 128 129 130 131 132 133

## Building from source

To build autoradio from source, you should have a Go environment set
up properly on your machine. Autoradio uses
[godep]( to manage its dependencies, so
make sure you have that installed as well. Building autoradio should
then be as simple as running, from the top-level source directory:

    $ godep go install ./...

ale's avatar
ale committed
This should install the *radiod*, *redirectord* and *radioctl*
ale's avatar
ale committed
135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155
executables in `$GOPATH/bin`.

# Operation

In order to create a new stream (*mount*, in the Icecast terminology),
assuming you are running autoradio on the `` domain:

1. On any node, run:

        $ radioctl create-mount /path/to/mount.ogg

   this will output the username and password used to authenticate the
   source. Take note of them.

   The cluster will be automatically reconfigured with the new mount in
   a few seconds at most.

2. Configure the source, using the username/password provided in the
   previous step, and point it at the following URL:

ale's avatar
ale committed
157 158 159

3. Tell your users to listen to the stream at:

ale's avatar
ale committed
161 162 163 164 165

Note: some sources are unable to handle HTTP redirects: in that case,
you might want to enable proxying on autoradio, and tell the client to
use the direct-path URL:

ale's avatar
ale committed
167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199

## DNS zone delegation

Since we can't modify the DNS glue records for the zone delegation in
real-time, we have to restrict slightly the assumptions on the
availability of nodes in the cluster: you have to assume that at least
N of your nodes will be partially available at any one time (i.e. at
least one of a chosen subset of N servers will be reachable). The
number N should be fairly low, say 3. Then, you can use those 3
servers as the nameservers for zone delegation, and the other nodes
are free to have dynamic membership.

## Proxy

The autoradio HTTP server can operate in one of two modes:

* clients connect directly to Icecast

    When a client connects to the service on port 80, it is sent a
    redirect to an Icecast server on port 8000. Unfortunately some
    older clients (especially sources) don't handle redirects too

* connections to Icecast are proxied by autoradio

    Clients talk to the autoradio HTTP server, which proxies
    connections to the back-end Icecast servers. This way the clients
    only need to talk to port 80, which not only avoids using
    redirects but might simplify access for people behind corporate
    proxies and such.

200 201
This behavior is controlled by the `--enable-icecast-proxy`
command-line flag to *redirectord*. It is set to true by default.
ale's avatar
ale committed
202 203 204 205 206

## Firewalls

The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and
207 208 209
8000/tcp (the latter only if proxying is disabled) on all nodes. Nodes
should be able to reach 2379/tcp and 2380/tcp (the etcd ports) on each
other; these two ports can be public if you've set up X509-based
ale's avatar
ale committed
authentication for etcd.
ale's avatar
ale committed
211 212 213 214

## Securing etcd

ale's avatar
ale committed
In a production cluster, you will want to limit access to the etcd
ale's avatar
ale committed
216 217
daemons so that only the other nodes can connect to it. While it is
possible to do this with firewall rules, the dynamic membership of the
ale's avatar
ale committed
cluster may make this difficult. We suggest using instead etcd's
ale's avatar
ale committed
219 220 221 222 223 224 225 226
support for X509 client authentication, together with a tool to manage
an online CA (such as [autoca](
This way, enrolling a new machine in the cluster only requires
generating a new client certificate, and no other configuration.

Install the CA certificate in `/etc/autoradio/etcd_ca.pem`, the client
certificate in `/etc/autoradio/etcd_client.pem` and its private key in
`/etc/autoradio/etcd_client.key`, and the clients will connect to
ale's avatar
ale committed
etcd using SSL authentication.
ale's avatar
ale committed
228 229 230 231 232 233 234 235 236 237 238 239 240 241

## Instrumentation

The *radiod* and *redirectord* daemons can send runtime metrics to
a *statsd* server (by default on localhost:8125).

## Transcoding

It is possible to set up a mount to relay an upstream mount re-encoded
with different parameters, using the `radioctl
create-transcoding-mount` command. In this case, autoradio will
automatically start up a process (a
[liquidsoap]( instance) to perform the
ale's avatar
ale committed
243 244 245 246 247 248 249 250 251
re-encoding, which will connect as the mount source. A master-election
protocol is used to ensure that only one such process per mount is
started in the whole cluster.

# Testing

There's a [Vagrant]( environment in the
`vagrant-test` subdirectory that will set up a test three-node cluster
(with Debian Jessie as the base system) using pre-packaged binaries.
ale's avatar
ale committed
253 254 255 256 257 258 259 260 261 262
To run it:

    $ cd vagrant-test
    $ vagrant up

It will take a while to download the base image the first time, then
it will turn up three nodes called **node1**, **node2** and **node3**.
Use `vagrant ssh` to inspect them.

If you want to test a locally-built package, copy the `autoradio` and
263 264
`etcd` Debian packages in the `vagrant-test` directory and, in that
same directory, run
ale's avatar
ale committed

    $ dpkg-scanpackages -m . >Packages
ale's avatar
ale committed

268 269
the provisioning process will automatically use the local packages if
they are available.