Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
A
autoradio
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ale
autoradio
Commits
86b39ed4
Commit
86b39ed4
authored
9 years ago
by
ale
Browse files
Options
Downloads
Patches
Plain Diff
convert documentation to Markdown
parent
220eb0c1
Branches
Branches containing commit
No related tags found
No related merge requests found
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
README.md
+253
-0
253 additions, 0 deletions
README.md
TUNING.md
+11
-17
11 additions, 17 deletions
TUNING.md
with
264 additions
and
17 deletions
README.
rst
→
README.
md
+
253
−
0
View file @
86b39ed4
=========
autoradio
=========
...
...
@@ -12,23 +11,22 @@ All this, if possible, without any operator intervention.
It is a full-stack service, meaning that it includes its own DNS and
HTTP servers, for full control of the request flow.
Autoradio works by using etcd
_ to coordinate the various nodes and store
the global mount configuration. The intended target is a set of
homogeneous servers (or virtual machines) dedicated to this purpose.
Autoradio also needs a dedicated DNS domain (or a delegation for a
subdomain).
Autoradio works by using
[
etcd
](
https://github.com/coreos/etcd
)
to
coordinate the various nodes and store the global mount configuration.
The intended target is a set of homogeneous servers (or virtual
machines) dedicated to this purpose. Autoradio also needs a dedicated
DNS domain (or a delegation for a
subdomain).
Installation
------------
# Installation
The simplest installation method is probably to use the pre-built
Debian packages (only available for amd64 at the moment), by placing
this line in
`
`/etc/apt/sources.list.d/autoradio.list`
`:
:
this line in
`/etc/apt/sources.list.d/autoradio.list`
:
deb http://www.incal.net/ale/debian autoradio/
And then running:
:
And then running:
$ sudo apt-get update
$ sudo apt-get install etcd autoradio
...
...
@@ -36,92 +34,100 @@ And then running::
This will install and start the necessary jobs (which will initially
fail due to the missing configuration).
Edit
`
`/etc/default/autoradio`
`
and set, at least, the
`
`DOMAIN`
`
Edit
`/etc/default/autoradio`
and set, at least, the
`DOMAIN`
variable to what you've assigned to the cluster. The jobs will
automatically start as soon as the configuration is saved.
Full cluster install procedure
++++++++++++++++++++++++++++++
## Full cluster install procedure
Note: this procedure requires etcd 2.0 or later.
This assumes that you have an existing domain name (here
*example.com*
) that you control, and that you will run the cluster
under a sub-domain (*radio.example.com*). Follow these steps to
bootstrap a new streaming cluster:
under a sub-domain (
*radio.example.com*
). The procedure will install
an etcd server on each node, so it will work best for a small, odd
number of machines.
#. Make sure that, on each of your servers, the output of ``hostname
-f`` is the fully-qualified hostname of the machine, and that it
resolves to its public IP (possibly using ``/etc/hosts``).
Having said that, follow these steps to bootstrap a new streaming
cluster:
#
.
On every server, run the above-mentioned steps to set up the APT
repository and install (do not configure) the ``etcd`` and
``autoradio`` packages
.
1
.
Make sure that, on each of your servers, the output of
`hostname
-f`
is the fully-qualified hostname of the machine, and that it
resolves to its public IP (possibly using
`/etc/hosts`
)
.
#. Pick one of your servers and add a delegation for
*radio.example.com* to it. For instance, with ``bind``::
2.
On every server, run the above-mentioned steps to set up the APT
repository and install (do not configure) the
`etcd`
and
`autoradio`
packages.
radio IN NS 3600 machine1.example.com.
3.
Pick one of your servers and add a delegation for
*radio.example.com*
to it. For instance, with
`bind`
:
#. On *machine1*, edit ``/etc/default/etcd`` and set
``BOOTSTRAP=1``. Once you save the file, the ``etcd`` daemon will
start with an empty database.
radio IN NS 3600 machine1.example.com.
#. On *machine1*, edit ``/etc/default/autoradio`` and set
``DOMAIN=radio.example.com``. This will start the ``radiod`` and
``redirectord`` daemons, and you will be able to serve DNS records
for the *radio.example.com* zone. Check with::
4.
On
*machine1*
, edit
`/etc/default/etcd`
with the following
contents:
$ ping -c1 radio.example.com
START=1
BOOTSTRAP=1
This should send a ping to *machine1*.
Once you save the file, the `etcd` daemon will start and
initialize an empty database.
#. Set up all other machines, setting
``ETCD_SERVER=etcd.radio.example.com`` in ``/etc/default/etcd`` and
``DOMAIN=radio.example.com`` in ``/etc/default/autoradio``.
5.
On
*machine1*
, edit
`/etc/default/autoradio`
and set
`DOMAIN=radio.example.com`
. This will start the
`radiod`
and
`redirectord`
daemons, and you will be able to serve DNS records
for the
*radio.example.com*
zone. Check with:
$ ping -c1 radio.example.com
Securing etcd
+++++++++++++
This should send a ping to *machine1*.
In a production cluster, you will want to limit access to the *etcd*
daemons so that only the other nodes can connect to it. While it is
possible to do this with firewall rules, the dynamic membership of the
cluster may make this difficult. We suggest using instead *etcd*'s
support for X509 client authentication, together with a tool to manage
an online CA (such as autoca_). This way, enrolling a new machine in
the cluster only requires generating a new client certificate, and no
other configuration.
6.
Set up the remaining machines. It is a two-step process: first, run
the following command on the first machine:
Install the CA certificate in ``/etc/autoradio/etcd_ca.pem``, the client
certificate in ``/etc/autoradio/etcd_client.pem`` and its private key in
``/etc/autoradio/etcd_client.key``, and the clients will connect to
*etcd* using SSL authentication.
$ etcdctl member add machine2 http://machine2:2380/
This will print out some environment variables. You should copy
the `ETCD_INITIAL_CLUSTER` line into `/etc/default/etcd` on
machine2, resulting in something like:
START=1
ETCD_INITIAL_CLUSTER=machine1=http://...,machine2=http://...
Finally, set `DOMAIN=radio.example.com` in
`/etc/default/autoradio`, and the daemons will start
automatically.
Building from source
++++++++++++++++++++
Note that you will need to wait for `etcd` on the new machine to
start successfully before you can run `etcdctl member add` for the
next one. For further instructions on how to change the etcd
cluster configuration at runtime, see
[the etcd documentation](https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md).
## Building from source
To build autoradio from source, you should have a Go environment set
up properly on your machine. Autoradio uses
godep_ to manage its
dependencies, so make sure you have that installed as well. Building
autoradio should then be as simp
le as
running, from the top-level
source directory:
:
up properly on your machine. Autoradio uses
[
godep
](
https://github.com/tools/godep
)
to manage its dependencies, so
make sure you have that instal
le
d
as
well. Building autoradio should
then be as simple as running, from the top-level
source directory:
$ godep go install ./...
This should install the
`
`radiod`
`
,
`
`redirectord`
`
and
`
`radioctl`
`
executables in
`
`$GOPATH/bin`
`
.
This should install the
`radiod`
,
`redirectord`
and
`radioctl`
executables in
`$GOPATH/bin`
.
Operation
---------
# Operation
In order to create a new stream (
*mount*
, in the Icecast terminology),
assuming you are running autoradio on the
`
`example.com`
`
domain:
assuming you are running autoradio on the
`example.com`
domain:
#
. On any node, run:
:
1
.
On any node, run:
$ radioctl create-mount /path/to/mount.ogg
$ radioctl create-mount /path/to/mount.ogg
this will output the username and password used to authenticate the
source. Take note of them.
...
...
@@ -129,18 +135,23 @@ assuming you are running autoradio on the ``example.com`` domain:
The cluster will be automatically reconfigured with the new mount in
a few seconds at most.
#. Configure the source, using the username/password provided in the
previous step, and point it at the following URL::
2.
Configure the source, using the username/password provided in the
previous step, and point it at the following URL:
http://stream.example.com/path/to/mount.ogg
3.
Tell your users to listen to the stream at:
http://stream.example.com/path/to/mount.ogg
http://stream.example.com/path/to/mount.ogg
.m3u
#. Tell your users to listen to the stream at::
Note: some sources are unable to handle HTTP redirects: in that case,
you might want to enable proxying on autoradio, and tell the client to
use the direct-path URL:
http://stream.example.com/path/to/mount.ogg
.m3u
http://stream.example.com/
_stream/
path/to/mount.ogg
DNS zone delegation
+++++++++++++++++++
## DNS zone delegation
Since we can't modify the DNS glue records for the zone delegation in
real-time, we have to restrict slightly the assumptions on the
...
...
@@ -152,8 +163,30 @@ servers as the nameservers for zone delegation, and the other nodes
are free to have dynamic membership.
Firewalls
+++++++++
## Proxy
The autoradio HTTP server can operate in one of two modes:
*
clients connect directly to Icecast
When a client connects to the service on port 80, it is sent a
redirect to an Icecast server on port 8000. Unfortunately some
older clients (especially sources) don't handle redirects too
well.
*
connections to Icecast are proxied by autoradio
Clients talk to the autoradio HTTP server, which proxies
connections to the back-end Icecast servers. This way the clients
only need to talk to port 80, which not only avoids using
redirects but might simplify access for people behind corporate
proxies and such.
This behavior is controlled by the `--enable-icecast-proxy`
command-line flag to *redirectord*.
## Firewalls
The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and
8000/tcp on all nodes. Nodes should be able to reach 4001/tcp and
...
...
@@ -161,42 +194,60 @@ The users should be able to reach ports 53/tcp, 53/udp, 80/tcp and
X509-based authentication to etcd.
Instrumentation
+++++++++++++++
## Securing etcd
In a production cluster, you will want to limit access to the
*etcd*
daemons so that only the other nodes can connect to it. While it is
possible to do this with firewall rules, the dynamic membership of the
cluster may make this difficult. We suggest using instead
*etcd*
's
support for X509 client authentication, together with a tool to manage
an online CA (such as
[
autoca
](
https://git.autistici.org/ai/autoca
)
.
This way, enrolling a new machine in the cluster only requires
generating a new client certificate, and no other configuration.
Install the CA certificate in
`/etc/autoradio/etcd_ca.pem`
, the client
certificate in
`/etc/autoradio/etcd_client.pem`
and its private key in
`/etc/autoradio/etcd_client.key`
, and the clients will connect to
*etcd*
using SSL authentication.
## Instrumentation
The
``
radiod
``
and
``
redirectord
``
daemons can send runtime metrics to
The
*
radiod
*
and
*
redirectord
*
daemons can send runtime metrics to
a
*statsd*
server (by default on localhost:8125).
Transcoding
+++++++++++
## Transcoding
It is possible to set up a mount to relay an upstream mount re-encoded
with different parameters, using the ``radioctl
create-transcoding-mount`` command. In this case, autoradio will
automatically start up a process (a liquidsoap_ instance) to perform
the re-encoding, which will connect as the mount source. A
master-election protocol is used to ensure that only one such process
per mount is started in the whole cluster.
with different parameters, using the
`radioctl
create-transcoding-mount`
command. In this case, autoradio will
automatically start up a process (a
[
liquidsoap
](
http://savonet.sourceforge.net/
)
instance) to perform the
re-encoding, which will connect as the mount source. A master-election
protocol is used to ensure that only one such process per mount is
started in the whole cluster.
Testing
-------
# Testing
There's a Vagrant_ configuration in the ``vagrant-test`` subdirectory
that will turn up a test three-node cluster (with Debian Wheezy as the
base system) using pre-packaged binaries. To run it::
There's a
[
Vagrant
](
http://www.vagrantup.com/
)
environment in the
`vagrant-test`
subdirectory that will set up a test three-node cluster
(with Debian Wheezy as the base system) using pre-packaged binaries.
To run it:
$ cd vagrant-test
$ vagrant up
It will take a while to download the base image the first time, then
it will turn up three nodes called
**node1**
,
**node2**
and
**node3**
.
Use ``vagrant ssh`` to inspect them.
Use
`vagrant ssh`
to inspect them.
If you want to test a locally-built package, copy the
`autoradio`
and
`etcd`
Debian packages in the
`vagrant-test`
directory and set the
`LOCAL`
environment variable to a non-empty string when invoking
vagrant:
$ LOCAL=1 vagrant up
.. _etcd: https://github.com/coreos/etcd
.. _autoca: https://git.autistici.org/ai/autoca
.. _Vagrant: http://www.vagrantup.com/
.. _godep: https://github.com/tools/godep
.. _liquidsoap: http://savonet.sourceforge.net/
This diff is collapsed.
Click to expand it.
TUNING.
rst
→
TUNING.
md
+
11
−
17
View file @
86b39ed4
======================
autoradio Tuning Guide
======================
...
...
@@ -10,8 +9,7 @@ box in testing environments, most real-world deployments will require
some tuning.
Etcd
----
# Etcd
The default settings for etcd are tuned for a local (LAN) network
environment. In the case of a geographically distributed cluster,
...
...
@@ -23,7 +21,7 @@ in your cluster, while the election timeout should be at least 3 times
the heartbeat interval.
With our etcd package, you can set these values in
`
`/etc/default/etcd`
`
(values are milliseconds):
:
`/etc/default/etcd`
(values are milliseconds):
DAEMON_OPTS="--peer-heartbeat-interval=1000 --peer-election-timeout=3000"
...
...
@@ -33,17 +31,15 @@ node failure. It is advisable that the radiod master election ttl is
set to a value greater than the etcd peer election timeout.
Radiod timeouts
---------------
# Radiod timeouts
Similar considerations, with respect to latency, apply to the presence
and master-election protocols that are run by autoradio itself. These
are controlled by radiod's
`
`--heartbeat`
`
and
``--master-election-ttl``
command-line flags. For these time values,
though, there are further
considerations to be made:
are controlled by radiod's
`--heartbeat`
and
`--master-election-ttl`
command-line flags. For these time values,
though, there are further
considerations to be made:
Presence
~~~~~~~~
## Presence
The node presence heartbeat sets the lower time bound for peers to
discover that a node is down, and stop sending client requests to it.
...
...
@@ -61,8 +57,7 @@ reloads of the full configuration, causing even more unnecessary
network traffic and increasing the load on etcd.
Master Election
~~~~~~~~~~~~~~~
## Master Election
The node master election timeout determines how quickly a source
(assuming it retries continuously on error) will be able to reconnect
...
...
@@ -70,8 +65,7 @@ to the cluster if the node that is currently the master becomes
unavailable.
Capacity
--------
# Capacity
One of the nice properties of the autoradio traffic control logic is
the ability to reject incoming traffic when the cluster reaches its
...
...
@@ -85,12 +79,12 @@ Autoradio models capacity along two separate dimensions: bandwidth
(outbound), and number of connected listeners. CPU/memory are not
included due to their negligible incremental cost per-request. Limits
can be set separately for each node in the cluster, by passing the
`
`--bwlimit`
`
and
`
`--max-clients`
`
command-line flags to
``
radiod
``
.
`--bwlimit`
and
`--max-clients`
command-line flags to
*
radiod
*
.
The traffic control logic is then able to use utilization metrics to
make decisions about where to send traffic. For details on how this is
done, and how to control it, check the Go source documentation for the
`
`fe/lbv2`
`
package.
`fe/lbv2`
package.
The default traffic control policy only checks the number of
listeners: this is because it usually makes the most sense to express
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment