Commit 2be2e08d authored by ale's avatar ale

Add SQL support

Introduces the "sql" backend, and refactors backends behind a nicer
interface (which looks a lot like the one in id/auth). Users can
define arbitrary SQL queries to fetch the key data, to adapt to their
database schema.

Include vendored dependencies for sqlite3/mysql/postgres.
parent 079c7ff5
Pipeline #2657 passed with stages
in 6 minutes and 4 seconds
......@@ -57,8 +57,8 @@ Forget the key for a given user.
The final consumer for user encryption keys is the Dovecot
service. The *dovecot-keylookupd* daemon can read the user public and
private keys from LDAP, and serve the *unencrypted* keys to Dovecot
using its [dict proxy
private keys from the database, and serve the *unencrypted* keys to
Dovecot using its [dict proxy
protocol](https://wiki2.dovecot.org/AuthDatabase/Dict).
*NOTE* that passdb lookups using *dovecot-keylookupd* contain the
......@@ -79,21 +79,10 @@ following attributes:
* `sso_public_key_file`: path to the SSO Ed25519 public key
* `sso_service`: SSO service for this application
* `sso_domain`: SSO domain
* `ldap`: LDAP backend configuration
* `uri`: LDAP server URI
* `bind_dn`: bind DN (for simple bind, SASL is not supported)
* `bind_pw`: bind password
* `bind_pw_file`: bind password (load from this file), in
alternative to *bind_pw*
* `query`: Parameters for the LDAP search query
* `search_base`: base DN for the search
* `search_filter`: search filter. The filter string may contain a
literal `%s` token somewhere, that will be replaced with the
(escaped) username.
* `scope`: search scope, one of *sub* (default), *one* or *base*
* `public_key_attr`: attribute that contains the user's public key
* `private_key_attr`: attribute that contains the user's encrypted
key(s)
* `backend`: backend configuration
* `type`: backend type, one of *ldap* or *sql*
* `params`: backend parameters, type-specific (see *Backend
configuration*, below)
* `http_server`: HTTP server configuration
* `tls`: contains the server-side TLS configuration:
* `cert`: path to the server certificate
......@@ -110,7 +99,10 @@ following attributes:
The *dovecot-keylookupd* daemon uses a similar configuration, read by
default from */etc/keystore/dovecot.yml*:
* `ldap`: LDAP backend configuration, see above
* `backend`: backend configuration
* `type`: backend type, one of *ldap* or *sql*
* `params`: backend parameters, type-specific (see *Backend
configuration*, below)
* `keystore`: configures the connection to the keystore service
* `url`: URL for the keystore service
* `sharded`: if true, requests to the keystore service will be
......@@ -121,3 +113,45 @@ default from */etc/keystore/dovecot.yml*:
* `ca`: path to the CA used to validate the server
* `shard`: shard identifier for the local host. Must be set if
keystore.sharded is true.
## Backend configuration
The keystore servers can talk to a LDAP or a SQL database. In both
cases it is possible to adapt to the database schema by defining the
exact queries to use. All we need to do is to retrieve the public and
private parts of the user encryption key.
The *ldap* database backend understands the following configuration
parameters:
* `uri`: LDAP server URI
* `bind_dn`: bind DN (for simple bind, SASL is not supported)
* `bind_pw`: bind password
* `bind_pw_file`: bind password (load from this file), in
alternative to *bind_pw*
* `query`: Parameters for the LDAP search query
* `search_base`: base DN for the search
* `search_filter`: search filter. The filter string may contain a
literal `%s` token somewhere, that will be replaced with the
(escaped) username.
* `scope`: search scope, one of *sub* (default), *one* or *base*
* `public_key_attr`: attribute that contains the user's public key
* `private_key_attr`: attribute that contains the user's encrypted
key(s)
The *sql* database backend requires the following parameters:
* `driver`: SQL driver, one of *sqlite3*, *mysql* or *postgres*
* `db_uri`: database URI (a.k.a. DSN), whose exact syntax will depend
on the chosen driver. Check out the documentation for the
database/sql [sqlite](https://github.com/mattn/go-sqlite3),
[mysql](https://github.com/go-sql-driver/mysql) and
[postgres](https://godoc.org/github.com/lib/pq) drivers.
* `queries`: map with the known queries. All SQL queries take one
parameter (the user name), and return one or more rows with a single
column. Use the `?` placeholder for the parameter. Known queries:
* `get_user_public_key`: must return a single row with the public
key
* `get_user_private_keys`: must return one or more rows with the
user's private keys (copies of the same key encrypted with
different passwords).
package backend
import (
"context"
"gopkg.in/yaml.v2"
)
// Database represents the interface to the underlying backend for
// encrypted user keys.
type Database interface {
GetPublicKey(context.Context, string) ([]byte, error)
GetPrivateKeys(context.Context, string) ([][]byte, error)
}
// Config is how users configure a database backend.
type Config struct {
Type string `yaml:"type"`
Params yaml.MapSlice `yaml:"params"`
}
// UnmarshalMapSlice re-unmarshals a partially-parsed yaml.MapSlice.
func UnmarshalMapSlice(raw yaml.MapSlice, obj interface{}) error {
b, err := yaml.Marshal(raw)
if err != nil {
return err
}
return yaml.Unmarshal(b, obj)
}
......@@ -10,9 +10,12 @@ import (
ldaputil "git.autistici.org/ai3/go-common/ldap"
ct "git.autistici.org/ai3/go-common/ldap/compositetypes"
"gopkg.in/ldap.v3"
"gopkg.in/yaml.v2"
"git.autistici.org/id/keystore/backend"
)
type LDAPQueryConfig struct {
type ldapQuery struct {
// SearchBase, SearchFilter and Scope define parameters for
// the LDAP search. The search should return a single object.
// SearchBase and SearchFilter can contain the string "%s",
......@@ -29,7 +32,7 @@ type LDAPQueryConfig struct {
}
// Valid returns an error if the configuration is invalid.
func (c *LDAPQueryConfig) Valid() error {
func (c *ldapQuery) Valid() error {
if c.SearchBase == "" {
return errors.New("empty search_base")
}
......@@ -58,7 +61,7 @@ func (c *LDAPQueryConfig) Valid() error {
return nil
}
func (c *LDAPQueryConfig) searchRequest(username string, attrs ...string) *ldap.SearchRequest {
func (c *ldapQuery) searchRequest(username string, attrs ...string) *ldap.SearchRequest {
u := ldap.EscapeFilter(username)
base := strings.Replace(c.SearchBase, "%s", u, -1)
filter := strings.Replace(c.SearchFilter, "%s", u, -1)
......@@ -75,17 +78,17 @@ func (c *LDAPQueryConfig) searchRequest(username string, attrs ...string) *ldap.
)
}
// LDAPConfig holds the global configuration for the LDAP user backend.
type LDAPConfig struct {
URI string `yaml:"uri"`
BindDN string `yaml:"bind_dn"`
BindPw string `yaml:"bind_pw"`
BindPwFile string `yaml:"bind_pw_file"`
Query *LDAPQueryConfig `yaml:"query"`
// ldapConfig holds the global configuration for the LDAP user backend.
type ldapConfig struct {
URI string `yaml:"uri"`
BindDN string `yaml:"bind_dn"`
BindPw string `yaml:"bind_pw"`
BindPwFile string `yaml:"bind_pw_file"`
Query *ldapQuery `yaml:"query"`
}
// Valid returns an error if the configuration is invalid.
func (c *LDAPConfig) Valid() error {
func (c *ldapConfig) Valid() error {
if c.URI == "" {
return errors.New("empty uri")
}
......@@ -102,11 +105,17 @@ func (c *LDAPConfig) Valid() error {
}
type ldapBackend struct {
config *LDAPConfig
config *ldapConfig
pool *ldaputil.ConnectionPool
}
func NewLDAPBackend(config *LDAPConfig) (*ldapBackend, error) {
// New returns a new LDAP backend.
func New(params yaml.MapSlice) (backend.Database, error) {
var config ldapConfig
if err := backend.UnmarshalMapSlice(params, &config); err != nil {
return nil, err
}
// Validate configuration.
if err := config.Valid(); err != nil {
return nil, err
......@@ -129,7 +138,7 @@ func NewLDAPBackend(config *LDAPConfig) (*ldapBackend, error) {
}
return &ldapBackend{
config: config,
config: &config,
pool: pool,
}, nil
}
......
package sql
import (
"context"
"database/sql"
"errors"
_ "github.com/go-sql-driver/mysql"
_ "github.com/lib/pq"
_ "github.com/mattn/go-sqlite3"
"gopkg.in/yaml.v2"
"git.autistici.org/id/keystore/backend"
)
// Names for known SQL queries.
const (
sqlQueryGetPublicKey = "get_public_key"
sqlQueryGetPrivateKeys = "get_private_keys"
)
type sqlConfig struct {
Driver string `yaml:"driver"`
URI string `yaml:"db_uri"`
Queries map[string]string `yaml:"queries"`
}
func (c *sqlConfig) Valid() error {
if c.Driver == "" {
return errors.New("empty driver")
}
if c.URI == "" {
return errors.New("empty uri")
}
return nil
}
type sqlBackend struct {
db *sql.DB
pubStmt *sql.Stmt
privStmt *sql.Stmt
}
// New returns a new SQL backend.
func New(params yaml.MapSlice) (backend.Database, error) {
var config sqlConfig
if err := backend.UnmarshalMapSlice(params, &config); err != nil {
return nil, err
}
if err := config.Valid(); err != nil {
return nil, err
}
db, err := sql.Open(config.Driver, config.URI)
if err != nil {
return nil, err
}
be := sqlBackend{db: db}
be.pubStmt, err = db.Prepare(config.Queries[sqlQueryGetPublicKey])
if err != nil {
db.Close()
return nil, err
}
be.privStmt, err = db.Prepare(config.Queries[sqlQueryGetPrivateKeys])
if err != nil {
db.Close()
return nil, err
}
return &be, nil
}
func (b *sqlBackend) GetPrivateKeys(ctx context.Context, username string) ([][]byte, error) {
tx, err := b.db.BeginTx(ctx, nil)
if err != nil {
return nil, err
}
defer tx.Rollback() // nolint
rows, err := tx.Stmt(b.privStmt).Query(username)
if err != nil {
return nil, err
}
var out [][]byte
for rows.Next() {
var key []byte
if err := rows.Scan(&key); err != nil {
return nil, err
}
}
return out, nil
}
func (b *sqlBackend) GetPublicKey(ctx context.Context, username string) ([]byte, error) {
tx, err := b.db.BeginTx(ctx, nil)
if err != nil {
return nil, err
}
defer tx.Rollback() // nolint
row := tx.Stmt(b.privStmt).QueryRow(username)
var key []byte
if err := row.Scan(&key); err != nil {
return nil, err
}
return key, nil
}
......@@ -11,35 +11,30 @@ import (
"git.autistici.org/ai3/go-common/userenckey"
"git.autistici.org/id/keystore/backend"
ldapBE "git.autistici.org/id/keystore/backend/ldap"
sqlBE "git.autistici.org/id/keystore/backend/sql"
"git.autistici.org/id/keystore/client"
)
// Config for the dovecot-keystore daemon.
type Config struct {
Shard string `yaml:"shard"`
LDAPConfig *backend.LDAPConfig `yaml:"ldap"`
Keystore *clientutil.BackendConfig `yaml:"keystore"`
Shard string `yaml:"shard"`
Backend *backend.Config `yaml:"backend"`
Keystore *clientutil.BackendConfig `yaml:"keystore"`
// Set this to true if the keys obtained from the backend need
// to be base64-encoded before being sent to Dovecot.
Base64Encode bool `yaml:"base64_encode_results"`
}
// Database represents the interface to the underlying backend for
// encrypted user keys.
type Database interface {
GetPublicKey(context.Context, string) ([]byte, error)
GetPrivateKeys(context.Context, string) ([][]byte, error)
}
func (c *Config) check() error {
if c.Keystore == nil {
return errors.New("missing keystore config")
}
if c.LDAPConfig == nil {
if c.Backend == nil {
return errors.New("missing backend config")
}
return c.LDAPConfig.Valid()
return nil
}
// The response returned to userdb lookups. It contains the user's
......@@ -86,7 +81,7 @@ var passwordSep = "/"
type KeyLookupProxy struct {
config *Config
keystore client.Client
db Database
db backend.Database
}
// NewKeyLookupProxy returns a KeyLookupProxy with the specified configuration.
......@@ -100,8 +95,15 @@ func NewKeyLookupProxy(config *Config) (*KeyLookupProxy, error) {
return nil, err
}
// There is only one supported backend type, ldap.
ldap, err := backend.NewLDAPBackend(config.LDAPConfig)
var db backend.Database
switch config.Backend.Type {
case "ldap":
db, err = ldapBE.New(config.Backend.Params)
case "sql":
db, err = sqlBE.New(config.Backend.Params)
default:
err = errors.New("unknown backend type")
}
if err != nil {
return nil, err
}
......@@ -109,7 +111,7 @@ func NewKeyLookupProxy(config *Config) (*KeyLookupProxy, error) {
return &KeyLookupProxy{
config: config,
keystore: ksc,
db: ldap,
db: db,
}, nil
}
......
......@@ -13,6 +13,8 @@ import (
"git.autistici.org/id/go-sso"
"git.autistici.org/id/keystore/backend"
ldapBE "git.autistici.org/id/keystore/backend/ldap"
sqlBE "git.autistici.org/id/keystore/backend/sql"
)
var (
......@@ -22,12 +24,6 @@ var (
errInvalidTTL = errors.New("invalid ttl")
)
// Database represents the interface to the underlying backend for
// encrypted user keys.
type Database interface {
GetPrivateKeys(context.Context, string) ([][]byte, error)
}
type userKey struct {
pkey []byte
expiry time.Time
......@@ -39,8 +35,7 @@ type Config struct {
SSOService string `yaml:"sso_service"`
SSODomain string `yaml:"sso_domain"`
//Backend string `yaml:"backend"`
LDAPConfig *backend.LDAPConfig `yaml:"ldap"`
Backend *backend.Config `yaml:"backend"`
}
func (c *Config) check() error {
......@@ -53,10 +48,10 @@ func (c *Config) check() error {
if c.SSODomain == "" {
return errors.New("sso_domain is empty")
}
if c.LDAPConfig == nil {
if c.Backend == nil {
return errors.New("missing backend config")
}
return c.LDAPConfig.Valid()
return nil
}
// KeyStore holds decrypted secrets for users in memory for a short
......@@ -76,12 +71,12 @@ type KeyStore struct {
mx sync.Mutex
userKeys map[string]userKey
db Database
db backend.Database
service string
validator sso.Validator
}
func newKeyStoreWithBackend(config *Config, db Database) (*KeyStore, error) {
func newKeyStoreWithBackend(config *Config, db backend.Database) (*KeyStore, error) {
ssoKey, err := ioutil.ReadFile(config.SSOPublicKeyFile)
if err != nil {
return nil, err
......@@ -107,13 +102,20 @@ func NewKeyStore(config *Config) (*KeyStore, error) {
return nil, err
}
// There is only one supported backend type, ldap.
ldap, err := backend.NewLDAPBackend(config.LDAPConfig)
var db backend.Database
var err error
switch config.Backend.Type {
case "ldap":
db, err = ldapBE.New(config.Backend.Params)
case "sql":
db, err = sqlBE.New(config.Backend.Params)
default:
err = errors.New("unknown backend type")
}
if err != nil {
return nil, err
}
return newKeyStoreWithBackend(config, ldap)
return newKeyStoreWithBackend(config, db)
}
func (s *KeyStore) expire(t time.Time) {
......
......@@ -3,6 +3,7 @@ package server
import (
"bytes"
"context"
"errors"
"io/ioutil"
"os"
"path/filepath"
......@@ -67,6 +68,10 @@ func (t *testDB) GetPrivateKeys(_ context.Context, username string) ([][]byte, e
return keys, nil
}
func (t *testDB) GetPublicKey(_ context.Context, _ string) ([]byte, error) {
return nil, errors.New("not implemented")
}
var (
privKey *userenckey.Key
pw = []byte("equally secret password")
......
# This is the official list of Go-MySQL-Driver authors for copyright purposes.
# If you are submitting a patch, please add your name or the name of the
# organization which holds the copyright to this list in alphabetical order.
# Names should be added to this file as
# Name <email address>
# The email address is not required for organizations.
# Please keep the list sorted.
# Individual Persons
Aaron Hopkins <go-sql-driver at die.net>
Achille Roussel <achille.roussel at gmail.com>
Alexey Palazhchenko <alexey.palazhchenko at gmail.com>
Andrew Reid <andrew.reid at tixtrack.com>
Arne Hormann <arnehormann at gmail.com>
Asta Xie <xiemengjun at gmail.com>
Bulat Gaifullin <gaifullinbf at gmail.com>
Carlos Nieto <jose.carlos at menteslibres.net>
Chris Moos <chris at tech9computers.com>
Craig Wilson <craiggwilson at gmail.com>
Daniel Montoya <dsmontoyam at gmail.com>
Daniel Nichter <nil at codenode.com>
Daniël van Eeden <git at myname.nl>
Dave Protasowski <dprotaso at gmail.com>
DisposaBoy <disposaboy at dby.me>
Egor Smolyakov <egorsmkv at gmail.com>
Evan Shaw <evan at vendhq.com>
Frederick Mayle <frederickmayle at gmail.com>
Gustavo Kristic <gkristic at gmail.com>
Hajime Nakagami <nakagami at gmail.com>
Hanno Braun <mail at hannobraun.com>
Henri Yandell <flamefew at gmail.com>
Hirotaka Yamamoto <ymmt2005 at gmail.com>
ICHINOSE Shogo <shogo82148 at gmail.com>
Ilia Cimpoes <ichimpoesh at gmail.com>
INADA Naoki <songofacandy at gmail.com>
Jacek Szwec <szwec.jacek at gmail.com>
James Harr <james.harr at gmail.com>
Jeff Hodges <jeff at somethingsimilar.com>
Jeffrey Charles <jeffreycharles at gmail.com>
Jerome Meyer <jxmeyer at gmail.com>
Jian Zhen <zhenjl at gmail.com>
Joshua Prunier <joshua.prunier at gmail.com>
Julien Lefevre <julien.lefevr at gmail.com>
Julien Schmidt <go-sql-driver at julienschmidt.com>
Justin Li <jli at j-li.net>
Justin Nuß <nuss.justin at gmail.com>
Kamil Dziedzic <kamil at klecza.pl>
Kevin Malachowski <kevin at chowski.com>
Kieron Woodhouse <kieron.woodhouse at infosum.com>
Lennart Rudolph <lrudolph at hmc.edu>
Leonardo YongUk Kim <dalinaum at gmail.com>
Linh Tran Tuan <linhduonggnu at gmail.com>
Lion Yang <lion at aosc.xyz>
Luca Looz <luca.looz92 at gmail.com>
Lucas Liu <extrafliu at gmail.com>
Luke Scott <luke at webconnex.com>
Maciej Zimnoch <maciej.zimnoch at codilime.com>
Michael Woolnough <michael.woolnough at gmail.com>
Nicola Peduzzi <thenikso at gmail.com>
Olivier Mengué <dolmen at cpan.org>
oscarzhao <oscarzhaosl at gmail.com>
Paul Bonser <misterpib at gmail.com>
Peter Schultz <peter.schultz at classmarkets.com>
Rebecca Chin <rchin at pivotal.io>
Reed Allman <rdallman10 at gmail.com>
Richard Wilkes <wilkes at me.com>
Robert Russell <robert at rrbrussell.com>
Runrioter Wung <runrioter at gmail.com>
Shuode Li <elemount at qq.com>
Simon J Mudd <sjmudd at pobox.com>
Soroush Pour <me at soroushjp.com>
Stan Putrya <root.vagner at gmail.com>
Stanley Gunawan <gunawan.stanley at gmail.com>
Steven Hartland <steven.hartland at multiplay.co.uk>
Thomas Wodarek <wodarekwebpage at gmail.com>
Tim Ruffles <timruffles at gmail.com>
Tom Jenkinson <tom at tjenkinson.me>
Xiangyu Hu <xiangyu.hu at outlook.com>
Xiaobing Jiang <s7v7nislands at gmail.com>
Xiuming Chen <cc at cxm.cc>
Zhenye Xie <xiezhenye at gmail.com>
# Organizations
Barracuda Networks, Inc.
Counting Ltd.
GitHub Inc.
Google Inc.
InfoSum Ltd.
Keybase Inc.
Percona LLC
Pivotal Inc.
Stripe Inc.
Multiplay Ltd.
## Version 1.4 (2018-06-03)
Changes:
- Documentation fixes (#530, #535, #567)
- Refactoring (#575, #579, #580, #581, #603, #615, #704)
- Cache column names (#444)
- Sort the DSN parameters in DSNs generated from a config (#637)
- Allow native password authentication by default (#644)
- Use the default port if it is missing in the DSN (#668)
- Removed the `strict` mode (#676)
- Do not query `max_allowed_packet` by default (#680)
- Dropped support Go 1.6 and lower (#696)
- Updated `ConvertValue()` to match the database/sql/driver implementation (#760)
- Document the usage of `0000-00-00T00:00:00` as the time.Time zero value (#783)
- Improved the compatibility of the authentication system (#807)
New Features:
- Multi-Results support (#537)
- `rejectReadOnly` DSN option (#604)
- `context.Context` support (#608, #612, #627, #761)
- Transaction isolation level support (#619, #744)