Skip to content

GitLab

Commit 098c74b3 authored by ale's avatar ale
Browse files

Initial refactoring for v2 

Includes a merge of radiod and redirectord, updates to more modern Go
idioms, adoption of the higher level etcd concurrency primitives, and
a lot of other minor implementation changes.

The refactor is still incomplete: the daemons won't compile and the
code to start the RPC servers is missing.
parent e9d7e68e
Hide whitespace changes
Inline Side-by-side
DESIGN.md 0 → 100644
View file @ 098c74b3
Architecture Overview
===
*autoradio* makes a number of decisions based on a bunch of assumptions:
* scaling up / down should be easy (eventually automated)
* individual nodes can be somewhat unreliable / unstable
* the bottleneck is bandwidth
* we can steer client traffic with redirects (either at the HTTP
level, or with the m3u -> stream link)
Internally, autoradio is split into a small number of components:
* the *node*, which controls a Icecast daemon on the same machine;
* the *frontend*, which responds to client requests and either
redirects them to a different frontend or proxies the stream to a
node;
* the *transcoder* which re-encodes streams by running liquidsoap.
These components coordinate with each other using *etcd*, which is
also the data store for the application-level configuration (streams
and users etc). Runtime information, which is primarily used by the
load balancing algorithm to determine utilization, is shared via a
separate gossip-like protocol to reduce the load on etcd.
Inter-process coordination is achieved using a small number of
coordination primitives:
* A *presence* primitive, which registers *endpoints* for specific
services (named ip:port pairs). It is then possible to retrieve the
list of all endpoints, or just one with a specific name.
* A *leader election* primitive, which picks a single leader out of
the participants, and can return at any time the leader endpoint. It
is possible to observe the state of the election without
participating in it.
We use leader election for tasks that should be unique in the entire
cluster, like selecting the Icecast master, or running the transcoder
for a specific stream.
In previous autoradio implementations, the node and the frontend were
split into separate binaries. But if we are trying to optimize
bandwidth usage, it doesn't make much sense for a frontend to proxy a
stream to a different host: why not send the client to that host in
the first place (except for sources)? So nodes and frontends would
always be co-located on the same host, because it was better to always
proxy frontend requests to the local Icecast, and using Icecast
relaying capabilities to cut the inter-node bandwidth.
Autoradio version 2 moves node and frontend in the same binary:
removing the implicit assumption of co-location simplifies the code
quite a bit, reducing the necessary coordination. On the other hand,
the transcoder is moved into a separate binary, allowing the
deployment of transcode-only nodes that do not participate in the
streaming cluster (useful because transcoding is heavily CPU-bound).
In this scheme:
* the nodes register presence for the following services:
* *status*, to transfer status information
* *icecast*, for the public Icecast address sent to clients
* and they run the following leader elections:
* *icecast*, to determine the Icecast master
The frontend then uses them depending on the incoming request type:
* DNS requests (HA-focused) will return all IPs in the *icecast*
presence set;
* HTTP client requests (LB-focused) will return a single IP picked by
the loadbalancing algorithm among the *icecast* presence set;
* HTTP source requests will be forwarded to the *icecast* leader.
The *status* presence set is used by the gossip algorithm to pick a
random node to send the next status update to.
Godeps/Godeps.json deleted 100644 → 0
View file @ e9d7e68e
{
"ImportPath": "git.autistici.org/ale/autoradio",
"GoVersion": "go1.5.1",
"Packages": [
"./..."
],
"Deps": [
{
"ImportPath": "github.com/aryann/difflib",
"Rev": "035af7c09b120b0909dd998c92745b82f61e0b1c"
},
{
"ImportPath": "github.com/cactus/go-statsd-client/statsd",
"Rev": "a99092dcd2d2f2a604a6f2502ba9ebba6a1165e5"
},
{
"ImportPath": "github.com/coreos/go-etcd/etcd",
"Comment": "v2.0.0-22-g9847b93",
"Rev": "9847b93751a5fbaf227b893d172cee0104ac6427"
},
{
"ImportPath": "github.com/gonuts/commander",
"Rev": "f8ba4e959ca914268227c3ebbd7f6bf0bb35541a"
},
{
"ImportPath": "github.com/gonuts/flag",
"Rev": "741a6cbd37a30dedc93f817e7de6aaf0ca38a493"
},
{
"ImportPath": "github.com/gorilla/handlers",
"Rev": "40694b40f4a928c062f56849989d3e9cd0570e5f"
},
{
"ImportPath": "github.com/jmcvetta/randutil",
"Rev": "53e1e064d354a2d879935e250c1445526d73b96c"
},
{
"ImportPath": "github.com/miekg/dns",
"Rev": "874ec871288a738d8d87fd5cec1dd71e88fdacb2"
},
{
"ImportPath": "github.com/ugorji/go/codec",
"Rev": "5abd4e96a45c386928ed2ca2a7ef63e2533e18ec"
}
]
}
Godeps/Readme deleted 100644 → 0
View file @ e9d7e68e
This directory tree is generated automatically by godep.
Please do not edit.
See https://github.com/tools/godep for more information.
Godeps/_workspace/src/github.com/aryann/difflib/README.md deleted 100644 → 0
View file @ e9d7e68e
difflib
=======
difflib is a simple library written in [Go](http://golang.org/) for
diffing two sequences of text.
Installing
----------
To install, issue:
go get github.com/aryann/difflib
Using
-----
To start using difflib, create a new file in your workspace and import
difflib:
import (
...
"fmt"
"github.com/aryann/difflib"
...
)
Then call either `difflib.Diff` or `difflib.HTMLDiff`:
fmt.Println(difflib.HTMLDiff([]string{"one", "two", "three"}, []string{"two", "four", "three"}))
If you'd like more control over the output, see how the function
`HTMLDiff` relies on `Diff` in difflib.go.
Running the Demo
----------------
There is a demo application in the difflib_demo directory. To run it,
navigate to your `$GOPATH` and run:
go run src/github.com/aryann/difflib/difflib_server/difflib_demo.go <file-1> <file-2>
Where `<file-1>` and `<file-2>` are two text files you'd like to
diff. The demo will launch a web server that will contain a table of
the diff results.
Godeps/_workspace/src/github.com/aryann/difflib/difflib.go deleted 100644 → 0
View file @ e9d7e68e
// Copyright 2012 Aryan Naraghi (aryan.naraghi@gmail.com)
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package difflib provides functionality for computing the difference
// between two sequences of strings.
package difflib
import (
"bytes"
"fmt"
"math"
)
// DeltaType describes the relationship of elements in two
// sequences. The following table provides a summary:
//
// Constant Code Meaning
// ---------- ------ ---------------------------------------
// Common " " The element occurs in both sequences.
// LeftOnly "-" The element is unique to sequence 1.
// RightOnly "+" The element is unique to sequence 2.
type DeltaType int
const (
Common DeltaType = iota
LeftOnly
RightOnly
)
// String returns a string representation for DeltaType.
func (t DeltaType) String() string {
switch t {
case Common:
return " "
case LeftOnly:
return "-"
case RightOnly:
return "+"
}
return "?"
}
type DiffRecord struct {
Payload string
Delta DeltaType
}
// String returns a string representation of d. The string is a
// concatenation of the delta type and the payload.
func (d DiffRecord) String() string {
return fmt.Sprintf("%s %s", d.Delta, d.Payload)
}
// Diff returns the result of diffing the seq1 and seq2.
func Diff(seq1, seq2 []string) (diff []DiffRecord) {
// Trims any common elements at the heads and tails of the
// sequences before running the diff algorithm. This is an
// optimization.
start, end := numEqualStartAndEndElements(seq1, seq2)
for _, content := range seq1[:start] {
diff = append(diff, DiffRecord{content, Common})
}
diffRes := compute(seq1[start:len(seq1)-end], seq2[start:len(seq2)-end])
diff = append(diff, diffRes...)
for _, content := range seq1[len(seq1)-end:] {
diff = append(diff, DiffRecord{content, Common})
}
return
}
// HTMLDiff returns the results of diffing seq1 and seq2 as an HTML
// string. The resulting HTML is a table without the opening and
// closing table tags. Each table row represents a DiffRecord. The
// first and last columns contain the "line numbers" for seq1 and
// seq2, respectively (the function assumes that seq1 and seq2
// represent the lines in a file). The second and third columns
// contain the actual file contents.
//
// The cells that contain line numbers are decorated with the class
// "line-num". The cells that contain deleted elements are decorated
// with "deleted" and the cells that contain added elements are
// decorated with "added".
func HTMLDiff(seq1, seq2 []string) string {
buf := bytes.NewBufferString("")
i, j := 0, 0
for _, d := range Diff(seq1, seq2) {
buf.WriteString(`<tr><td class="line-num">`)
if d.Delta == Common || d.Delta == LeftOnly {
i++
fmt.Fprintf(buf, "%d</td><td", i)
if d.Delta == LeftOnly {
fmt.Fprint(buf, ` class="deleted"`)
}
fmt.Fprintf(buf, "><pre>%s</pre>", d.Payload)
} else {
buf.WriteString("</td><td>")
}
buf.WriteString("</td><td")
if d.Delta == Common || d.Delta == RightOnly {
j++
if d.Delta == RightOnly {
fmt.Fprint(buf, ` class="added"`)
}
fmt.Fprintf(buf, `><pre>%s</pre></td><td class="line-num">%d`, d.Payload, j)
} else {
buf.WriteString("></td><td>")
}
buf.WriteString("</td></tr>\n")
}
return buf.String()
}
// numEqualStartAndEndElements returns the number of elements a and b
// have in common from the beginning and from the end. If a and b are
// equal, start will equal len(a) == len(b) and end will be zero.
func numEqualStartAndEndElements(seq1, seq2 []string) (start, end int) {
for start < len(seq1) && start < len(seq2) && seq1[start] == seq2[start] {
start++
}
i, j := len(seq1)-1, len(seq2)-1
for i > start && j > start && seq1[i] == seq2[j] {
i--
j--
end++
}
return
}
// intMatrix returns a 2-dimensional slice of ints with the given
// number of rows and columns.
func intMatrix(rows, cols int) [][]int {
matrix := make([][]int, rows)
for i := 0; i < rows; i++ {
matrix[i] = make([]int, cols)
}
return matrix
}
// longestCommonSubsequenceMatrix returns the table that results from
// applying the dynamic programming approach for finding the longest
// common subsequence of seq1 and seq2.
func longestCommonSubsequenceMatrix(seq1, seq2 []string) [][]int {
matrix := intMatrix(len(seq1)+1, len(seq2)+1)
for i := 1; i < len(matrix); i++ {
for j := 1; j < len(matrix[i]); j++ {
if seq1[len(seq1)-i] == seq2[len(seq2)-j] {
matrix[i][j] = matrix[i-1][j-1] + 1
} else {
matrix[i][j] = int(math.Max(float64(matrix[i-1][j]),
float64(matrix[i][j-1])))
}
}
}
return matrix
}
// compute is the unexported helper for Diff that returns the results of
// diffing left and right.
func compute(seq1, seq2 []string) (diff []DiffRecord) {
matrix := longestCommonSubsequenceMatrix(seq1, seq2)
i, j := len(seq1), len(seq2)
for i > 0 || j > 0 {
if i > 0 && matrix[i][j] == matrix[i-1][j] {
diff = append(diff, DiffRecord{seq1[len(seq1)-i], LeftOnly})
i--
} else if j > 0 && matrix[i][j] == matrix[i][j-1] {
diff = append(diff, DiffRecord{seq2[len(seq2)-j], RightOnly})
j--
} else if i > 0 && j > 0 {
diff = append(diff, DiffRecord{seq1[len(seq1)-i], Common})
i--
j--
}
}
return
}
Godeps/_workspace/src/github.com/aryann/difflib/difflib_server/difflib_demo.go deleted 100644 → 0
View file @ e9d7e68e
// Copyright 2012 Aryan Naraghi (aryan.naraghi@gmail.com)
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// A demo for difflib. This program accepts the paths to two files and
// launches a web server at port 8080 that serves the diff results.
package main
import (
"fmt"
"git.autistici.org/ale/autoradio/Godeps/_workspace/src/github.com/aryann/difflib"
"html"
"html/template"
"io/ioutil"
"net/http"
"os"
"strings"
)
var hostPort = "localhost:8080"
var templateString = `
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>difflib results</title>
<style type="text/css">
table {
background-color: lightgrey;
border-spacing: 1px;
}
tr {
background-color: white;
border-bottom: 1px solid black;
}
.line-num {
width: 50px;
}
.added {
background-color: green;
}
.deleted {
background-color: red;
}
</style>
</head>
<body>
<table>
<tr>
<th></th>
<th><em>{{.Filename1}}</em></th>
<th><em>{{.Filename2}}</em></th>
<th></th>
</tr>
{{.Diff}}
</table>
</body>
</html>
`
func main() {
if len(os.Args) != 3 {
fmt.Fprintf(os.Stderr, "USAGE: %s <file-1> <file-2>\n", os.Args[0])
os.Exit(1)
}
http.HandleFunc("/", diffHandler(os.Args[1], os.Args[2]))
fmt.Printf("Starting server at %s.\n", hostPort)
err := http.ListenAndServe(hostPort, nil)
if err != nil {
panic(err)
}
}
// diffHandler returns an http.HandlerFunc that serves the diff of the
// two given files.
func diffHandler(filename1, filename2 string) http.HandlerFunc {
diff := difflib.HTMLDiff(fileToLines(filename1), fileToLines(filename2))
tmpl, _ := template.New("diffTemplate").Parse(templateString)
return func(w http.ResponseWriter, r *http.Request) {
err := tmpl.Execute(w, map[string]interface{}{
"Diff": template.HTML(diff),
"Filename1": filename1,
"Filename2": filename2,
})
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
}
}
// filesToLines takes a path to a file and returns a string array of
// the lines in the file. Any HTML in the file is escaped.
func fileToLines(filename string) []string {
contents, _ := ioutil.ReadFile(filename)
return strings.Split(html.EscapeString(string(contents)), "\n")
}
Godeps/_workspace/src/github.com/aryann/difflib/difflib_test.go deleted 100644 → 0
View file @ e9d7e68e
// Copyright 2012 Aryan Naraghi (aryan.naraghi@gmail.com)
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package difflib
import (
"reflect"
"strings"
"testing"
)
var lcsTests = []struct {
seq1 string
seq2 string
lcs int
}{
{"", "", 0},
{"abc", "abc", 3},
{"mzjawxu", "xmjyauz", 4},
{"human", "chimpanzee", 4},
{"Hello, world!", "Hello, world!", 13},
{"Hello, world!", "H e l l o , w o r l d !", 13},
}
func TestLongestCommonSubsequenceMatrix(t *testing.T) {
for i, test := range lcsTests {
seq1 := strings.Split(test.seq1, "")
seq2 := strings.Split(test.seq2, "")
matrix := longestCommonSubsequenceMatrix(seq1, seq2)
lcs := matrix[len(matrix)-1][len(matrix[0])-1] // Grabs the lower, right value.
if lcs != test.lcs {
t.Errorf("%d. longestCommonSubsequence(%v, %v)[last][last] => %d, expected %d",
i, seq1, seq2, lcs, test.lcs)
}
}
}
var numEqualStartAndEndElementsTests = []struct {
seq1 string
seq2 string
start int
end int
}{
{"", "", 0, 0},
{"abc", "", 0, 0},
{"", "abc", 0, 0},
{"abc", "abc", 3, 0},
{"abhelloc", "abbyec", 2, 1},
{"abchello", "abcbye", 3, 0},
{"helloabc", "byeabc", 0, 3},
}
func TestNumEqualStartAndEndElements(t *testing.T) {
for i, test := range numEqualStartAndEndElementsTests {
seq1 := strings.Split(test.seq1, "")
seq2 := strings.Split(test.seq2, "")
start, end := numEqualStartAndEndElements(seq1, seq2)
if start != test.start || end != test.end {
t.Errorf("%d. numEqualStartAndEndElements(%v, %v) => (%d, %d), expected (%d, %d)",
i, seq1, seq2, start, end, test.start, test.end)
}
}
}
var diffTests = []struct {
Seq1 string
Seq2 string
Diff []DiffRecord
HtmlDiff string
}{
{
"",
"",
[]DiffRecord{
{"", Common},
},
`<tr><td class="line-num">1</td><td><pre></pre></td><td><pre></pre></td><td class="line-num">1</td></tr>
`,
},
{
"same",
"same",
[]DiffRecord{
{"same", Common},
},
`<tr><td class="line-num">1</td><td><pre>same</pre></td><td><pre>same</pre></td><td class="line-num">1</td></tr>
`,
},
{
`one
two
three
`,
`one
two
three
`,
[]DiffRecord{
{"one", Common},
{"two", Common},
{"three", Common},
{"", Common},
},