diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..0e3b824672e3d91957f659aff3432d2ce82fe81c --- /dev/null +++ b/README.md @@ -0,0 +1,253 @@ + +autoradio +========= + +The *autoradio* service aims to provide a reliable, fault-tolerant +Icecast streaming service for audio and video. It provides all the +necessary components to ensure that the traffic from the source to the +clients is uninterrupted, even in face of high load or server crashes. +All this, if possible, without any operator intervention. + +It is a full-stack service, meaning that it includes its own DNS and +HTTP servers, for full control of the request flow. + +Autoradio works by using [etcd](https://github.com/coreos/etcd) to +coordinate the various nodes and store the global mount configuration. +The intended target is a set of homogeneous servers (or virtual +machines) dedicated to this purpose. Autoradio also needs a dedicated +DNS domain (or a delegation for a subdomain). + + +# Installation + +The simplest installation method is probably to use the pre-built +Debian packages (only available for amd64 at the moment), by placing +this line in `/etc/apt/sources.list.d/autoradio.list`: + + deb http://www.incal.net/ale/debian autoradio/ + +And then running: + + $ sudo apt-get update + $ sudo apt-get install etcd autoradio + +This will install and start the necessary jobs (which will initially +fail due to the missing configuration). + +Edit `/etc/default/autoradio` and set, at least, the `DOMAIN` +variable to what you've assigned to the cluster. The jobs will +automatically start as soon as the configuration is saved. + + +## Full cluster install procedure + +Note: this procedure requires etcd 2.0 or later. + +This assumes that you have an existing domain name (here +*example.com*) that you control, and that you will run the cluster +under a sub-domain (*radio.example.com*). The procedure will install +an etcd server on each node, so it will work best for a small, odd +number of machines. + +Having said that, follow these steps to bootstrap a new streaming +cluster: + +1. Make sure that, on each of your servers, the output of `hostname + -f` is the fully-qualified hostname of the machine, and that it + resolves to its public IP (possibly using `/etc/hosts`). + +2. On every server, run the above-mentioned steps to set up the APT + repository and install (do not configure) the `etcd` and + `autoradio` packages. + +3. Pick one of your servers and add a delegation for + *radio.example.com* to it. For instance, with `bind`: + + radio IN NS 3600 machine1.example.com. + +4. On *machine1*, edit `/etc/default/etcd` with the following + contents: + + START=1 + BOOTSTRAP=1 + + Once you save the file, the `etcd` daemon will start and + initialize an empty database. + +5. On *machine1*, edit `/etc/default/autoradio` and set + `DOMAIN=radio.example.com`. This will start the `radiod` and + `redirectord` daemons, and you will be able to serve DNS records + for the *radio.example.com* zone. Check with: + + $ ping -c1 radio.example.com + + This should send a ping to *machine1*. + +6. Set up the remaining machines. It is a two-step process: first, run + the following command on the first machine: + + $ etcdctl member add machine2 http://machine2:2380/ + + This will print out some environment variables. You should copy + the `ETCD_INITIAL_CLUSTER` line into `/etc/default/etcd` on + machine2, resulting in something like: + + START=1 + ETCD_INITIAL_CLUSTER=machine1=http://...,machine2=http://... + + Finally, set `DOMAIN=radio.example.com` in + `/etc/default/autoradio`, and the daemons will start + automatically. + + Note that you will need to wait for `etcd` on the new machine to + start successfully before you can run `etcdctl member add` for the + next one. For further instructions on how to change the etcd + cluster configuration at runtime, see + [the etcd documentation](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md). + + +## Building from source + +To build autoradio from source, you should have a Go environment set +up properly on your machine. Autoradio uses +[godep](https://github.com/tools/godep) to manage its dependencies, so +make sure you have that installed as well. Building autoradio should +then be as simple as running, from the top-level source directory: + + $ godep go install ./... + +This should install the `radiod`, `redirectord` and `radioctl` +executables in `$GOPATH/bin`. + + +# Operation + +In order to create a new stream (*mount*, in the Icecast terminology), +assuming you are running autoradio on the `example.com` domain: + +1. On any node, run: + + $ radioctl create-mount /path/to/mount.ogg + + this will output the username and password used to authenticate the + source. Take note of them. + + The cluster will be automatically reconfigured with the new mount in + a few seconds at most. + +2. Configure the source, using the username/password provided in the + previous step, and point it at the following URL: + + http://stream.example.com/path/to/mount.ogg + +3. Tell your users to listen to the stream at: + + http://stream.example.com/path/to/mount.ogg.m3u + +Note: some sources are unable to handle HTTP redirects: in that case, +you might want to enable proxying on autoradio, and tell the client to +use the direct-path URL: + + http://stream.example.com/_stream/path/to/mount.ogg + + +## DNS zone delegation + +Since we can't modify the DNS glue records for the zone delegation in +real-time, we have to restrict slightly the assumptions on the +availability of nodes in the cluster: you have to assume that at least +N of your nodes will be partially available at any one time (i.e. at +least one of a chosen subset of N servers will be reachable). The +number N should be fairly low, say 3. Then, you can use those 3 +servers as the nameservers for zone delegation, and the other nodes +are free to have dynamic membership. + + +## Proxy + +The autoradio HTTP server can operate in one of two modes: + +* clients connect directly to Icecast + + When a client connects to the service on port 80, it is sent a + redirect to an Icecast server on port 8000. Unfortunately some + older clients (especially sources) don't handle redirects too + well. + +* connections to Icecast are proxied by autoradio + + Clients talk to the autoradio HTTP server, which proxies + connections to the back-end Icecast servers. This way the clients + only need to talk to port 80, which not only avoids using + redirects but might simplify access for people behind corporate + proxies and such. + + This behavior is controlled by the `--enable-icecast-proxy` + command-line flag to *redirectord*. + + +## Firewalls + +The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and +8000/tcp on all nodes. Nodes should be able to reach 4001/tcp and +4002/tcp on each other; these two ports can be public if you've set up +X509-based authentication to etcd. + + +## Securing etcd + +In a production cluster, you will want to limit access to the *etcd* +daemons so that only the other nodes can connect to it. While it is +possible to do this with firewall rules, the dynamic membership of the +cluster may make this difficult. We suggest using instead *etcd*'s +support for X509 client authentication, together with a tool to manage +an online CA (such as [autoca](https://git.autistici.org/ai/autoca). +This way, enrolling a new machine in the cluster only requires +generating a new client certificate, and no other configuration. + +Install the CA certificate in `/etc/autoradio/etcd_ca.pem`, the client +certificate in `/etc/autoradio/etcd_client.pem` and its private key in +`/etc/autoradio/etcd_client.key`, and the clients will connect to +*etcd* using SSL authentication. + + +## Instrumentation + +The *radiod* and *redirectord* daemons can send runtime metrics to +a *statsd* server (by default on localhost:8125). + + +## Transcoding + +It is possible to set up a mount to relay an upstream mount re-encoded +with different parameters, using the `radioctl +create-transcoding-mount` command. In this case, autoradio will +automatically start up a process (a +[liquidsoap](http://savonet.sourceforge.net/) instance) to perform the +re-encoding, which will connect as the mount source. A master-election +protocol is used to ensure that only one such process per mount is +started in the whole cluster. + + +# Testing + +There's a [Vagrant](http://www.vagrantup.com/) environment in the +`vagrant-test` subdirectory that will set up a test three-node cluster +(with Debian Wheezy as the base system) using pre-packaged binaries. +To run it: + + $ cd vagrant-test + $ vagrant up + +It will take a while to download the base image the first time, then +it will turn up three nodes called **node1**, **node2** and **node3**. +Use `vagrant ssh` to inspect them. + +If you want to test a locally-built package, copy the `autoradio` and +`etcd` Debian packages in the `vagrant-test` directory and set the +`LOCAL` environment variable to a non-empty string when invoking +vagrant: + + $ LOCAL=1 vagrant up + + diff --git a/README.rst b/README.rst deleted file mode 100644 index 6da2a789a1328ae1bc4b739860519402b1d2b421..0000000000000000000000000000000000000000 --- a/README.rst +++ /dev/null @@ -1,202 +0,0 @@ - -========= -autoradio -========= - -The *autoradio* service aims to provide a reliable, fault-tolerant -Icecast streaming service for audio and video. It provides all the -necessary components to ensure that the traffic from the source to the -clients is uninterrupted, even in face of high load or server crashes. -All this, if possible, without any operator intervention. - -It is a full-stack service, meaning that it includes its own DNS and -HTTP servers, for full control of the request flow. - -Autoradio works by using etcd_ to coordinate the various nodes and store -the global mount configuration. The intended target is a set of -homogeneous servers (or virtual machines) dedicated to this purpose. -Autoradio also needs a dedicated DNS domain (or a delegation for a -subdomain). - - -Installation ------------- - -The simplest installation method is probably to use the pre-built -Debian packages (only available for amd64 at the moment), by placing -this line in ``/etc/apt/sources.list.d/autoradio.list``:: - - deb http://www.incal.net/ale/debian autoradio/ - -And then running:: - - $ sudo apt-get update - $ sudo apt-get install etcd autoradio - -This will install and start the necessary jobs (which will initially -fail due to the missing configuration). - -Edit ``/etc/default/autoradio`` and set, at least, the ``DOMAIN`` -variable to what you've assigned to the cluster. The jobs will -automatically start as soon as the configuration is saved. - - -Full cluster install procedure -++++++++++++++++++++++++++++++ - -This assumes that you have an existing domain name (here -*example.com*) that you control, and that you will run the cluster -under a sub-domain (*radio.example.com*). Follow these steps to -bootstrap a new streaming cluster: - -#. Make sure that, on each of your servers, the output of ``hostname - -f`` is the fully-qualified hostname of the machine, and that it - resolves to its public IP (possibly using ``/etc/hosts``). - -#. On every server, run the above-mentioned steps to set up the APT - repository and install (do not configure) the ``etcd`` and - ``autoradio`` packages. - -#. Pick one of your servers and add a delegation for - *radio.example.com* to it. For instance, with ``bind``:: - - radio IN NS 3600 machine1.example.com. - -#. On *machine1*, edit ``/etc/default/etcd`` and set - ``BOOTSTRAP=1``. Once you save the file, the ``etcd`` daemon will - start with an empty database. - -#. On *machine1*, edit ``/etc/default/autoradio`` and set - ``DOMAIN=radio.example.com``. This will start the ``radiod`` and - ``redirectord`` daemons, and you will be able to serve DNS records - for the *radio.example.com* zone. Check with:: - - $ ping -c1 radio.example.com - - This should send a ping to *machine1*. - -#. Set up all other machines, setting - ``ETCD_SERVER=etcd.radio.example.com`` in ``/etc/default/etcd`` and - ``DOMAIN=radio.example.com`` in ``/etc/default/autoradio``. - - -Securing etcd -+++++++++++++ - -In a production cluster, you will want to limit access to the *etcd* -daemons so that only the other nodes can connect to it. While it is -possible to do this with firewall rules, the dynamic membership of the -cluster may make this difficult. We suggest using instead *etcd*'s -support for X509 client authentication, together with a tool to manage -an online CA (such as autoca_). This way, enrolling a new machine in -the cluster only requires generating a new client certificate, and no -other configuration. - -Install the CA certificate in ``/etc/autoradio/etcd_ca.pem``, the client -certificate in ``/etc/autoradio/etcd_client.pem`` and its private key in -``/etc/autoradio/etcd_client.key``, and the clients will connect to -*etcd* using SSL authentication. - - -Building from source -++++++++++++++++++++ - -To build autoradio from source, you should have a Go environment set -up properly on your machine. Autoradio uses godep_ to manage its -dependencies, so make sure you have that installed as well. Building -autoradio should then be as simple as running, from the top-level -source directory:: - - $ godep go install ./... - -This should install the ``radiod``, ``redirectord`` and ``radioctl`` -executables in ``$GOPATH/bin``. - - -Operation ---------- - -In order to create a new stream (*mount*, in the Icecast terminology), -assuming you are running autoradio on the ``example.com`` domain: - -#. On any node, run:: - - $ radioctl create-mount /path/to/mount.ogg - - this will output the username and password used to authenticate the - source. Take note of them. - - The cluster will be automatically reconfigured with the new mount in - a few seconds at most. - -#. Configure the source, using the username/password provided in the - previous step, and point it at the following URL:: - - http://stream.example.com/path/to/mount.ogg - -#. Tell your users to listen to the stream at:: - - http://stream.example.com/path/to/mount.ogg.m3u - - -DNS zone delegation -+++++++++++++++++++ - -Since we can't modify the DNS glue records for the zone delegation in -real-time, we have to restrict slightly the assumptions on the -availability of nodes in the cluster: you have to assume that at least -N of your nodes will be partially available at any one time (i.e. at -least one of a chosen subset of N servers will be reachable). The -number N should be fairly low, say 3. Then, you can use those 3 -servers as the nameservers for zone delegation, and the other nodes -are free to have dynamic membership. - - -Firewalls -+++++++++ - -The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and -8000/tcp on all nodes. Nodes should be able to reach 4001/tcp and -4002/tcp on each other; these two ports can be public if you've set up -X509-based authentication to etcd. - - -Instrumentation -+++++++++++++++ - -The ``radiod`` and ``redirectord`` daemons can send runtime metrics to -a *statsd* server (by default on localhost:8125). - - -Transcoding -+++++++++++ - -It is possible to set up a mount to relay an upstream mount re-encoded -with different parameters, using the ``radioctl -create-transcoding-mount`` command. In this case, autoradio will -automatically start up a process (a liquidsoap_ instance) to perform -the re-encoding, which will connect as the mount source. A -master-election protocol is used to ensure that only one such process -per mount is started in the whole cluster. - - -Testing -------- - -There's a Vagrant_ configuration in the ``vagrant-test`` subdirectory -that will turn up a test three-node cluster (with Debian Wheezy as the -base system) using pre-packaged binaries. To run it:: - - $ cd vagrant-test - $ vagrant up - -It will take a while to download the base image the first time, then -it will turn up three nodes called **node1**, **node2** and **node3**. -Use ``vagrant ssh`` to inspect them. - - -.. _etcd: https://github.com/coreos/etcd -.. _autoca: https://git.autistici.org/ai/autoca -.. _Vagrant: http://www.vagrantup.com/ -.. _godep: https://github.com/tools/godep -.. _liquidsoap: http://savonet.sourceforge.net/ diff --git a/TUNING.rst b/TUNING.md similarity index 89% rename from TUNING.rst rename to TUNING.md index d488d0df0f6ea9503e70bce936b0a14f53b9af2e..5b38d1caafb11f3c10af4393cfa11bd4d7dfe765 100644 --- a/TUNING.rst +++ b/TUNING.md @@ -1,5 +1,4 @@ -====================== autoradio Tuning Guide ====================== @@ -10,8 +9,7 @@ box in testing environments, most real-world deployments will require some tuning. -Etcd ----- +# Etcd The default settings for etcd are tuned for a local (LAN) network environment. In the case of a geographically distributed cluster, @@ -23,7 +21,7 @@ in your cluster, while the election timeout should be at least 3 times the heartbeat interval. With our etcd package, you can set these values in -``/etc/default/etcd`` (values are milliseconds):: +`/etc/default/etcd` (values are milliseconds): DAEMON_OPTS="--peer-heartbeat-interval=1000 --peer-election-timeout=3000" @@ -33,17 +31,15 @@ node failure. It is advisable that the radiod master election ttl is set to a value greater than the etcd peer election timeout. -Radiod timeouts ---------------- +# Radiod timeouts Similar considerations, with respect to latency, apply to the presence and master-election protocols that are run by autoradio itself. These -are controlled by radiod's ``--heartbeat`` and -``--master-election-ttl`` command-line flags. For these time values, -though, there are further considerations to be made: +are controlled by radiod's `--heartbeat` and `--master-election-ttl` +command-line flags. For these time values, though, there are further +considerations to be made: -Presence -~~~~~~~~ +## Presence The node presence heartbeat sets the lower time bound for peers to discover that a node is down, and stop sending client requests to it. @@ -61,8 +57,7 @@ reloads of the full configuration, causing even more unnecessary network traffic and increasing the load on etcd. -Master Election -~~~~~~~~~~~~~~~ +## Master Election The node master election timeout determines how quickly a source (assuming it retries continuously on error) will be able to reconnect @@ -70,8 +65,7 @@ to the cluster if the node that is currently the master becomes unavailable. -Capacity --------- +# Capacity One of the nice properties of the autoradio traffic control logic is the ability to reject incoming traffic when the cluster reaches its @@ -85,12 +79,12 @@ Autoradio models capacity along two separate dimensions: bandwidth (outbound), and number of connected listeners. CPU/memory are not included due to their negligible incremental cost per-request. Limits can be set separately for each node in the cluster, by passing the -``--bwlimit`` and ``--max-clients`` command-line flags to ``radiod``. +`--bwlimit` and `--max-clients` command-line flags to *radiod*. The traffic control logic is then able to use utilization metrics to make decisions about where to send traffic. For details on how this is done, and how to control it, check the Go source documentation for the -``fe/lbv2`` package. +`fe/lbv2` package. The default traffic control policy only checks the number of listeners: this is because it usually makes the most sense to express