README.md 9.56 KB
Newer Older
ale's avatar
ale committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

autoradio
=========

The *autoradio* service aims to provide a reliable, fault-tolerant
Icecast streaming service for audio and video. It provides all the
necessary components to ensure that the traffic from the source to the
clients is uninterrupted, even in face of high load or server crashes.
All this, if possible, without any operator intervention.

It is a full-stack service, meaning that it includes its own DNS and
HTTP servers, for full control of the request flow.

Autoradio works by using [etcd](https://github.com/coreos/etcd) to
coordinate the various nodes and store the global mount configuration.
The intended target is a set of homogeneous servers (or virtual
machines) dedicated to this purpose. Autoradio also needs a dedicated
DNS domain (or a delegation for a subdomain).


# Installation

The simplest installation method is probably to use the pre-built
Debian packages (only available for amd64 at the moment), by placing
this line in `/etc/apt/sources.list.d/autoradio.list`:

    deb http://www.incal.net/ale/debian autoradio/

And then running:

ale's avatar
ale committed
31
    $ sudo apt-key adv --recv-key 0xC0EAC2F9CE9ED9B0
ale's avatar
ale committed
32
    $ sudo apt-get update
33
    $ sudo apt-get install etcd autoradio-server
ale's avatar
ale committed
34 35 36 37


## Full cluster install procedure

38 39
Note: this procedure assumes a Debian distribution, it should work
either with Wheezy (oldstable) or Jessie (stable).
ale's avatar
ale committed
40 41 42 43 44 45 46 47 48 49

This assumes that you have an existing domain name (here
*example.com*) that you control, and that you will run the cluster
under a sub-domain (*radio.example.com*). The procedure will install
an etcd server on each node, so it will work best for a small, odd
number of machines.

Having said that, follow these steps to bootstrap a new streaming
cluster:

50 51 52 53 54 55 56 57 58 59 60 61 62 63 64
1. Make sure that, on each of your servers, the output of `hostname
   -f` is the fully-qualified hostname of the machine, and that it
   resolves to its public IP (possibly using `/etc/hosts`). This way
   autoradio can detect the IP address that peers should use when
   communicating with each host. Also, for simplicity, we're going to
   assume that each host can resolve the IP address of each other just
   by using its short name.

2. Pick one of your servers, say *host1*, and add a delegation for
   *radio.example.com* to it. For instance, in a `bind`-formatted zone
   file:

        radio  IN   NS  3600  host1.example.com.

3. On *host1*, edit `/etc/default/etcd` with the following contents:
ale's avatar
ale committed
65

66 67 68 69 70 71
        ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
        ETCD_INITIAL_ADVERTISE_PEER_URLS=http://host1:2380
        ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
        ETCD_ADVERTISE_CLIENT_URLS=http://host1:2379
        ETCD_INITIAL_CLUSTER_STATE=new
        ETCD_INITIAL_CLUSTER=node1=http://host1:2380
ale's avatar
ale committed
72

73 74
    Once you save the file, restart the *etcd* daemon: this will
    initialize an empty database:
ale's avatar
ale committed
75

76
        $ service etcd restart
ale's avatar
ale committed
77

78
4. On *host1*, edit `/etc/default/autoradio` and add:
ale's avatar
ale committed
79

80
        DOMAIN="--domain=radio.example.com"
ale's avatar
ale committed
81

82 83 84
5. Run the steps in the _Installation_ section above to set up the APT
   repository and install the `etcd` and `autoradio` packages using
   the configuration you just wrote.
ale's avatar
ale committed
85

86 87 88 89
   This will start the *radiod* and *redirectord* daemons, and you
   will be able to serve DNS records for the *radio.example.com* zone.

   Check that the service is healthy with:
ale's avatar
ale committed
90 91 92

        $ ping -c1 radio.example.com

93 94 95 96
    This should send a ping to *host1*.

Now that the first node is up and running, set up the remaining
machines. For every host:
ale's avatar
ale committed
97

98 99
1. Set up etcd. First, run the following command on the first machine
   (host1):
ale's avatar
ale committed
100

101
        $ etcdctl member add host2 http://host2:2380
ale's avatar
ale committed
102

103
   (remember not to include a final slash on the node URL).
ale's avatar
ale committed
104

105 106 107 108 109
   This will print out some environment variables. You should copy
   the `ETCD_INITIAL_CLUSTER` line into `/etc/default/etcd` on
   the new host. The other lines of that file should be identical to
   what shown in step 4 of the previous checklist, replacing the host
   name where necessary.
ale's avatar
ale committed
110

111 112 113 114 115
   Note that you will need to wait for etcd on the new machine to
   start successfully before you can run `etcdctl member add` for the
   next one. For further instructions on how to change the etcd
   cluster configuration at runtime, see
   [the etcd documentation](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md).
ale's avatar
ale committed
116

117 118 119 120 121 122
2. Set `DOMAIN` in `/etc/default/autoradio` (as shown in step 5 of
   the previous checklist above), and the daemons will start
   automatically.

3. Install the autoradio packages, see _Installation_ section above.
   The daemons should start automatically with the new configuration.
ale's avatar
ale committed
123 124 125 126 127 128 129 130 131 132 133 134


## Building from source

To build autoradio from source, you should have a Go environment set
up properly on your machine. Autoradio uses
[godep](https://github.com/tools/godep) to manage its dependencies, so
make sure you have that installed as well. Building autoradio should
then be as simple as running, from the top-level source directory:

    $ godep go install ./...

ale's avatar
ale committed
135
This should install the *radiod*, *redirectord* and *radioctl*
ale's avatar
ale committed
136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156
executables in `$GOPATH/bin`.


# Operation

In order to create a new stream (*mount*, in the Icecast terminology),
assuming you are running autoradio on the `example.com` domain:

1. On any node, run:

        $ radioctl create-mount /path/to/mount.ogg

   this will output the username and password used to authenticate the
   source. Take note of them.

   The cluster will be automatically reconfigured with the new mount in
   a few seconds at most.

2. Configure the source, using the username/password provided in the
   previous step, and point it at the following URL:

157
        http://example.com/path/to/mount.ogg
ale's avatar
ale committed
158 159 160

3. Tell your users to listen to the stream at:

161
        http://example.com/path/to/mount.ogg.m3u
ale's avatar
ale committed
162 163 164 165 166

Note: some sources are unable to handle HTTP redirects: in that case,
you might want to enable proxying on autoradio, and tell the client to
use the direct-path URL:

167
    http://example.com/_stream/path/to/mount.ogg
ale's avatar
ale committed
168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200


## DNS zone delegation

Since we can't modify the DNS glue records for the zone delegation in
real-time, we have to restrict slightly the assumptions on the
availability of nodes in the cluster: you have to assume that at least
N of your nodes will be partially available at any one time (i.e. at
least one of a chosen subset of N servers will be reachable). The
number N should be fairly low, say 3. Then, you can use those 3
servers as the nameservers for zone delegation, and the other nodes
are free to have dynamic membership.


## Proxy

The autoradio HTTP server can operate in one of two modes:

* clients connect directly to Icecast

    When a client connects to the service on port 80, it is sent a
    redirect to an Icecast server on port 8000. Unfortunately some
    older clients (especially sources) don't handle redirects too
    well.

* connections to Icecast are proxied by autoradio

    Clients talk to the autoradio HTTP server, which proxies
    connections to the back-end Icecast servers. This way the clients
    only need to talk to port 80, which not only avoids using
    redirects but might simplify access for people behind corporate
    proxies and such.

201 202
This behavior is controlled by the `--enable-icecast-proxy`
command-line flag to *redirectord*. It is set to true by default.
ale's avatar
ale committed
203 204 205 206 207


## Firewalls

The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and
208 209 210
8000/tcp (the latter only if proxying is disabled) on all nodes. Nodes
should be able to reach 2379/tcp and 2380/tcp (the etcd ports) on each
other; these two ports can be public if you've set up X509-based
ale's avatar
ale committed
211
authentication for etcd.
ale's avatar
ale committed
212 213 214 215


## Securing etcd

ale's avatar
ale committed
216
In a production cluster, you will want to limit access to the etcd
ale's avatar
ale committed
217 218
daemons so that only the other nodes can connect to it. While it is
possible to do this with firewall rules, the dynamic membership of the
ale's avatar
ale committed
219
cluster may make this difficult. We suggest using instead etcd's
ale's avatar
ale committed
220 221 222 223 224 225 226 227
support for X509 client authentication, together with a tool to manage
an online CA (such as [autoca](https://git.autistici.org/ai/autoca).
This way, enrolling a new machine in the cluster only requires
generating a new client certificate, and no other configuration.

Install the CA certificate in `/etc/autoradio/etcd_ca.pem`, the client
certificate in `/etc/autoradio/etcd_client.pem` and its private key in
`/etc/autoradio/etcd_client.key`, and the clients will connect to
ale's avatar
ale committed
228
etcd using SSL authentication.
ale's avatar
ale committed
229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252


## Instrumentation

The *radiod* and *redirectord* daemons can send runtime metrics to
a *statsd* server (by default on localhost:8125).


## Transcoding

It is possible to set up a mount to relay an upstream mount re-encoded
with different parameters, using the `radioctl
create-transcoding-mount` command. In this case, autoradio will
automatically start up a process (a
[liquidsoap](http://savonet.sourceforge.net/) instance) to perform the
re-encoding, which will connect as the mount source. A master-election
protocol is used to ensure that only one such process per mount is
started in the whole cluster.


# Testing

There's a [Vagrant](http://www.vagrantup.com/) environment in the
`vagrant-test` subdirectory that will set up a test three-node cluster
253
(with Debian Jessie as the base system) using pre-packaged binaries.
ale's avatar
ale committed
254 255 256 257 258 259 260 261 262 263
To run it:

    $ cd vagrant-test
    $ vagrant up

It will take a while to download the base image the first time, then
it will turn up three nodes called **node1**, **node2** and **node3**.
Use `vagrant ssh` to inspect them.

If you want to test a locally-built package, copy the `autoradio` and
264 265
`etcd` Debian packages in the `vagrant-test` directory and, in that
same directory, run
ale's avatar
ale committed
266

267
    $ dpkg-scanpackages -m . >Packages
ale's avatar
ale committed
268

269 270
the provisioning process will automatically use the local packages if
they are available.