Skip to content

GitLab

A
autoradio
Commit 098c74b3 authored by ale's avatar ale
Browse files
Options

Initial refactoring for v2 

Includes a merge of radiod and redirectord, updates to more modern Go
idioms, adoption of the higher level etcd concurrency primitives, and
a lot of other minor implementation changes.

The refactor is still incomplete: the daemons won't compile and the
code to start the RPC servers is missing.
parent e9d7e68e
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 236815 additions and 77118 deletions

Too many changes to show.

To preserve performance only 1000 of 1000+ files are displayed.
Plain diff Email patch
DESIGN.md 0 → 100644
View file @ 098c74b3
Architecture Overview
===
*autoradio* makes a number of decisions based on a bunch of assumptions:
* scaling up / down should be easy (eventually automated)
* individual nodes can be somewhat unreliable / unstable
* the bottleneck is bandwidth
* we can steer client traffic with redirects (either at the HTTP
level, or with the m3u -> stream link)
Internally, autoradio is split into a small number of components:
* the *node*, which controls a Icecast daemon on the same machine;
* the *frontend*, which responds to client requests and either
redirects them to a different frontend or proxies the stream to a
node;
* the *transcoder* which re-encodes streams by running liquidsoap.
These components coordinate with each other using *etcd*, which is
also the data store for the application-level configuration (streams
and users etc). Runtime information, which is primarily used by the
load balancing algorithm to determine utilization, is shared via a
separate gossip-like protocol to reduce the load on etcd.
Inter-process coordination is achieved using a small number of
coordination primitives:
* A *presence* primitive, which registers *endpoints* for specific
services (named ip:port pairs). It is then possible to retrieve the
list of all endpoints, or just one with a specific name.
* A *leader election* primitive, which picks a single leader out of
the participants, and can return at any time the leader endpoint. It
is possible to observe the state of the election without
participating in it.
We use leader election for tasks that should be unique in the entire
cluster, like selecting the Icecast master, or running the transcoder
for a specific stream.
In previous autoradio implementations, the node and the frontend were
split into separate binaries. But if we are trying to optimize
bandwidth usage, it doesn't make much sense for a frontend to proxy a
stream to a different host: why not send the client to that host in
the first place (except for sources)? So nodes and frontends would
always be co-located on the same host, because it was better to always
proxy frontend requests to the local Icecast, and using Icecast
relaying capabilities to cut the inter-node bandwidth.
Autoradio version 2 moves node and frontend in the same binary:
removing the implicit assumption of co-location simplifies the code
quite a bit, reducing the necessary coordination. On the other hand,
the transcoder is moved into a separate binary, allowing the
deployment of transcode-only nodes that do not participate in the
streaming cluster (useful because transcoding is heavily CPU-bound).
In this scheme:
* the nodes register presence for the following services:
* *status*, to transfer status information
* *icecast*, for the public Icecast address sent to clients
* and they run the following leader elections:
* *icecast*, to determine the Icecast master
The frontend then uses them depending on the incoming request type:
* DNS requests (HA-focused) will return all IPs in the *icecast*
presence set;
* HTTP client requests (LB-focused) will return a single IP picked by
the loadbalancing algorithm among the *icecast* presence set;
* HTTP source requests will be forwarded to the *icecast* leader.
The *status* presence set is used by the gossip algorithm to pick a
random node to send the next status update to.
Godeps/Godeps.json deleted 100644 → 0
View file @ e9d7e68e
{
"ImportPath": "git.autistici.org/ale/autoradio",
"GoVersion": "go1.5.1",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "github.com/aryann/difflib",
"Rev": "035af7c09b120b0909dd998c92745b82f61e0b1c"
},
{
"ImportPath": "github.com/cactus/go-statsd-client/statsd",
"Rev": "a99092dcd2d2f2a604a6f2502ba9ebba6a1165e5"
},
{
"ImportPath": "github.com/coreos/go-etcd/etcd",
"Comment": "v2.0.0-22-g9847b93",
"Rev": "9847b93751a5fbaf227b893d172cee0104ac6427"
},
{
"ImportPath": "github.com/gonuts/commander",
"Rev": "f8ba4e959ca914268227c3ebbd7f6bf0bb35541a"
},
{
"ImportPath": "github.com/gonuts/flag",
"Rev": "741a6cbd37a30dedc93f817e7de6aaf0ca38a493"
},
{
"ImportPath": "github.com/gorilla/handlers",
"Rev": "40694b40f4a928c062f56849989d3e9cd0570e5f"
},
{
"ImportPath": "github.com/jmcvetta/randutil",
"Rev": "53e1e064d354a2d879935e250c1445526d73b96c"
},
{
"ImportPath": "github.com/miekg/dns",
"Rev": "874ec871288a738d8d87fd5cec1dd71e88fdacb2"
},
{
"ImportPath": "github.com/ugorji/go/codec",
"Rev": "5abd4e96a45c386928ed2ca2a7ef63e2533e18ec"
}
]
}
Godeps/Readme deleted 100644 → 0
View file @ e9d7e68e
This directory tree is generated automatically by godep.
Please do not edit.
See https://github.com/tools/godep for more information.
Godeps/_workspace/src/github.com/aryann/difflib/README.md deleted 100644 → 0
View file @ e9d7e68e
difflib
=======
difflib is a simple library written in [Go](http://golang.org/) for
diffing two sequences of text.
Installing
----------
To install, issue:
go get github.com/aryann/difflib
Using
-----
To start using difflib, create a new file in your workspace and import
difflib:
import (
...
"fmt"
"github.com/aryann/difflib"
...
)
Then call either `difflib.Diff` or `difflib.HTMLDiff`:
fmt.Println(difflib.HTMLDiff([]string{"one", "two", "three"}, []string{"two", "four", "three"}))
If you'd like more control over the output, see how the function
`HTMLDiff` relies on `Diff` in difflib.go.
Running the Demo
----------------
There is a demo application in the difflib_demo directory. To run it,
navigate to your `$GOPATH` and run:
go run src/github.com/aryann/difflib/difflib_server/difflib_demo.go <file-1> <file-2>
Where `<file-1>` and `<file-2>` are two text files you'd like to
diff. The demo will launch a web server that will contain a table of
the diff results.
Godeps/_workspace/src/github.com/aryann/difflib/difflib.go deleted 100644 → 0
View file @ e9d7e68e
// Copyright 2012 Aryan Naraghi (aryan.naraghi@gmail.com)
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//