Commit 056fbeee authored by ale's avatar ale

Add sql dependencies and documentation

parent aa79431d
......@@ -167,7 +167,7 @@ user, with the following attributes:
* `groups` is a list of group names that the user belongs to
## LDAP Backend
## LDAP backend
The *ldap* backend will look up user information in a LDAP database.
The backend connects to a single LDAP server and requires the
......@@ -231,6 +231,102 @@ App-specific passwords should be encoded as colon-separated strings:
The password should be encrypted. The comment is a free-form string
set by the user to tell the various credentials apart.
## SQL backend
The SQL backend allows you to use a SQL database to store user
information. It can adapt to any schema, provided that you can write
the queries it expects.
The parameters for the SQL backend configuration are:
* `driver` is the name of the database/sql driver (currently it must
be one of `sqlite3`, `mysql` or `postgres`, the built-in drivers)
* `db_uri` is the database URI (a.k.a. DSN), whose exact syntax will
depend on the chosen driver. Check out the documentation for the
database/sql [sqlite](,
[mysql]( and
[postgres]( drivers.
### Query definition
Each service can specify a set of different SQL queries. It can be
configured with the following attributes:
* `queries` holds the map of SQL queries that tell the auth-server
how to query your database.
The known queries are identified by name. It does not matter what
operations you do as long as the queries take the expected input
substitution parameters, and return rows with the expected number of
fields (column names do not matter). You should use the parameter
substitution symbol `?` as placeholder for query parameters.
* `get_user` takes a single parameter (the user name) and must return
a single row with *email*, *password*, *TOTP secret* and *shard*
fields for the matching user.
* `get_user_groups` takes a single parameter (the user name) and must
return rows with a single *group_name* field corresponding to the
user's group memberships.
* `get_user_u2f` takes a single parameter (user name) and must return
the user's U2F registrations as rows with *public_key* and
*key_handle* fields, in their native binary format.
* `get_user_asp` takes a single parameter (user name) and must return
the user's application-specific passwords as rows with *service* and
*password* fields.
The only mandatory query is *get_user*, if the other ones are not
specified the associated fields will be empty.
### Example database schema
The following could be a (very simple) example database schema for a
case where usernames are also email addresses, with support for all
authentication features:
email text NOT NULL,
password text NOT NULL,
totp_secret text,
shard text
CREATE UNIQUE INDEX users_email_idx ON users(email);
CREATE TABLE group_memberships (
email text NOT NULL,
group_name text NOT NULL
CREATE INDEX group_memberships_idx ON group_memberships(email);
CREATE TABLE u2f_registrations (
email text NOT NULL,
key_handle blob NOT NULL,
public_key blob NOT NULL
CREATE INDEX u2f_registrations_idx ON u2f_registrations(email);
CREATE TABLE service_passwords (
email text NOT NULL,
service text NOT NULL,
password text NOT NULL
CREATE INDEX service_passwords_idx ON service_passwords(email);
With this schema, one could use the following configuration for a
challenge_response: true
- backend: sql
get_user: "SELECT email, password, totp_secret, shard FROM users WHERE email = ?"
get_user_groups: "SELECT group_name FROM group_memberships WHERE email = ?"
get_user_u2f: "SELECT public_key, key_handle FROM u2f_registrations WHERE email = ?"
get_user_asp: "SELECT service, password FROM service_passwords WHERE email = ?"
# Usage
......@@ -6,10 +6,13 @@ import (
_ ""
_ ""
_ ""
_ ""
ct ""
# This is the official list of Go-MySQL-Driver authors for copyright purposes.
# If you are submitting a patch, please add your name or the name of the
# organization which holds the copyright to this list in alphabetical order.
# Names should be added to this file as
# Name <email address>
# The email address is not required for organizations.
# Please keep the list sorted.
# Individual Persons
Aaron Hopkins <go-sql-driver at>
Achille Roussel <achille.roussel at>
Alexey Palazhchenko <alexey.palazhchenko at>
Andrew Reid <andrew.reid at>
Arne Hormann <arnehormann at>
Asta Xie <xiemengjun at>
Bulat Gaifullin <gaifullinbf at>
Carlos Nieto <jose.carlos at>
Chris Moos <chris at>
Craig Wilson <craiggwilson at>
Daniel Montoya <dsmontoyam at>
Daniel Nichter <nil at>
Daniël van Eeden <git at>
Dave Protasowski <dprotaso at>
DisposaBoy <disposaboy at>
Egor Smolyakov <egorsmkv at>
Evan Shaw <evan at>
Frederick Mayle <frederickmayle at>
Gustavo Kristic <gkristic at>
Hajime Nakagami <nakagami at>
Hanno Braun <mail at>
Henri Yandell <flamefew at>
Hirotaka Yamamoto <ymmt2005 at>
ICHINOSE Shogo <shogo82148 at>
Ilia Cimpoes <ichimpoesh at>
INADA Naoki <songofacandy at>
Jacek Szwec <szwec.jacek at>
James Harr <james.harr at>
Jeff Hodges <jeff at>
Jeffrey Charles <jeffreycharles at>
Jerome Meyer <jxmeyer at>
Jian Zhen <zhenjl at>
Joshua Prunier <joshua.prunier at>
Julien Lefevre <julien.lefevr at>
Julien Schmidt <go-sql-driver at>
Justin Li <jli at>
Justin Nuß <nuss.justin at>
Kamil Dziedzic <kamil at>
Kevin Malachowski <kevin at>
Kieron Woodhouse <kieron.woodhouse at>
Lennart Rudolph <lrudolph at>
Leonardo YongUk Kim <dalinaum at>
Linh Tran Tuan <linhduonggnu at>
Lion Yang <lion at>
Luca Looz <luca.looz92 at>
Lucas Liu <extrafliu at>
Luke Scott <luke at>
Maciej Zimnoch <maciej.zimnoch at>
Michael Woolnough <michael.woolnough at>
Nicola Peduzzi <thenikso at>
Olivier Mengué <dolmen at>
oscarzhao <oscarzhaosl at>
Paul Bonser <misterpib at>
Peter Schultz <peter.schultz at>
Rebecca Chin <rchin at>
Reed Allman <rdallman10 at>
Richard Wilkes <wilkes at>
Robert Russell <robert at>
Runrioter Wung <runrioter at>
Shuode Li <elemount at>
Simon J Mudd <sjmudd at>
Soroush Pour <me at>
Stan Putrya <root.vagner at>
Stanley Gunawan <gunawan.stanley at>
Steven Hartland <steven.hartland at>
Thomas Wodarek <wodarekwebpage at>
Tim Ruffles <timruffles at>
Tom Jenkinson <tom at>
Xiangyu Hu < at>
Xiaobing Jiang <s7v7nislands at>
Xiuming Chen <cc at>
Zhenye Xie <xiezhenye at>
# Organizations
Barracuda Networks, Inc.
Counting Ltd.
GitHub Inc.
Google Inc.
InfoSum Ltd.
Keybase Inc.
Percona LLC
Pivotal Inc.
Stripe Inc.
Multiplay Ltd.
## Version 1.4 (2018-06-03)
- Documentation fixes (#530, #535, #567)
- Refactoring (#575, #579, #580, #581, #603, #615, #704)
- Cache column names (#444)
- Sort the DSN parameters in DSNs generated from a config (#637)
- Allow native password authentication by default (#644)
- Use the default port if it is missing in the DSN (#668)
- Removed the `strict` mode (#676)
- Do not query `max_allowed_packet` by default (#680)
- Dropped support Go 1.6 and lower (#696)
- Updated `ConvertValue()` to match the database/sql/driver implementation (#760)
- Document the usage of `0000-00-00T00:00:00` as the time.Time zero value (#783)
- Improved the compatibility of the authentication system (#807)
New Features:
- Multi-Results support (#537)
- `rejectReadOnly` DSN option (#604)
- `context.Context` support (#608, #612, #627, #761)
- Transaction isolation level support (#619, #744)
- Read-Only transactions support (#618, #634)
- `NewConfig` function which initializes a config with default values (#679)
- Implemented the `ColumnType` interfaces (#667, #724)
- Support for custom string types in `ConvertValue` (#623)
- Implemented `NamedValueChecker`, improving support for uint64 with high bit set (#690, #709, #710)
- `caching_sha2_password` authentication plugin support (#794, #800, #801, #802)
- Implemented `driver.SessionResetter` (#779)
- `sha256_password` authentication plugin support (#808)
- Use the DSN hostname as TLS default ServerName if `tls=true` (#564, #718)
- Fixed LOAD LOCAL DATA INFILE for empty files (#590)
- Removed columns definition cache since it sometimes cached invalid data (#592)
- Don't mutate registered TLS configs (#600)
- Make RegisterTLSConfig concurrency-safe (#613)
- Handle missing auth data in the handshake packet correctly (#646)
- Do not retry queries when data was written to avoid data corruption (#302, #736)
- Cache the connection pointer for error handling before invalidating it (#678)
- Fixed imports for appengine/cloudsql (#700)
- Fix sending STMT_LONG_DATA for 0 byte data (#734)
- Set correct capacity for []bytes read from length-encoded strings (#766)
- Make RegisterDial concurrency-safe (#773)
## Version 1.3 (2016-12-01)
- Go 1.1 is no longer supported
- Use decimals fields in MySQL to format time types (#249)
- Buffer optimizations (#269)
- TLS ServerName defaults to the host (#283)
- Refactoring (#400, #410, #437)
- Adjusted documentation for second generation CloudSQL (#485)
- Documented DSN system var quoting rules (#502)
- Made statement.Close() calls idempotent to avoid errors in Go 1.6+ (#512)
New Features:
- Enable microsecond resolution on TIME, DATETIME and TIMESTAMP (#249)
- Support for returning table alias on Columns() (#289, #359, #382)
- Placeholder interpolation, can be actived with the DSN parameter `interpolateParams=true` (#309, #318, #490)
- Support for uint64 parameters with high bit set (#332, #345)
- Cleartext authentication plugin support (#327)
- Exported ParseDSN function and the Config struct (#403, #419, #429)
- Read / Write timeouts (#401)
- Support for JSON field type (#414)
- Support for multi-statements and multi-results (#411, #431)
- DSN parameter to set the driver-side max_allowed_packet value manually (#489)
- Native password authentication plugin support (#494, #524)
- Fixed handling of queries without columns and rows (#255)
- Fixed a panic when SetKeepAlive() failed (#298)
- Handle ERR packets while reading rows (#321)
- Fixed reading NULL length-encoded integers in MySQL 5.6+ (#349)
- Fixed absolute paths support in LOAD LOCAL DATA INFILE (#356)
- Actually zero out bytes in handshake response (#378)
- Fixed race condition in registering LOAD DATA INFILE handler (#383)
- Fixed tests with MySQL 5.7.9+ (#380)
- QueryUnescape TLS config names (#397)
- Fixed "broken pipe" error by writing to closed socket (#390)
- Fixed LOAD LOCAL DATA INFILE buffering (#424)
- Fixed parsing of floats into float64 when placeholders are used (#434)
- Fixed DSN tests with Go 1.7+ (#459)
- Handle ERR packets while waiting for EOF (#473)
- Invalidate connection on error while discarding additional results (#513)
- Allow terminating packets of length 0 (#516)
## Version 1.2 (2014-06-03)
- We switched back to a "rolling release". `go get` installs the current master branch again
- Version v1 of the driver will not be maintained anymore. Go 1.0 is no longer supported by this driver
- Exported errors to allow easy checking from application code
- Enabled TCP Keepalives on TCP connections
- Optimized INFILE handling (better buffer size calculation, lazy init, ...)
- The DSN parser also checks for a missing separating slash
- Faster binary date / datetime to string formatting
- Also exported the MySQLWarning type
- mysqlConn.Close returns the first error encountered instead of ignoring all errors
- writePacket() automatically writes the packet size to the header
- readPacket() uses an iterative approach instead of the recursive approach to merge splitted packets
New Features:
- `RegisterDial` allows the usage of a custom dial function to establish the network connection
- Setting the connection collation is possible with the `collation` DSN parameter. This parameter should be preferred over the `charset` parameter
- Logging of critical errors is configurable with `SetLogger`
- Google CloudSQL support
- Allow more than 32 parameters in prepared statements
- Various old_password fixes
- Fixed TestConcurrent test to pass Go's race detection
- Fixed appendLengthEncodedInteger for large numbers
- Renamed readLengthEnodedString to readLengthEncodedString and skipLengthEnodedString to skipLengthEncodedString (fixed typo)
## Version 1.1 (2013-11-02)
- Go-MySQL-Driver now requires Go 1.1
- Connections now use the collation `utf8_general_ci` by default. Adding `&charset=UTF8` to the DSN should not be necessary anymore
- Made closing rows and connections error tolerant. This allows for example deferring rows.Close() without checking for errors
- `[]byte(nil)` is now treated as a NULL value. Before, it was treated like an empty string / `[]byte("")`
- DSN parameter values must now be url.QueryEscape'ed. This allows text values to contain special characters, such as '&'.
- Use the IO buffer also for writing. This results in zero allocations (by the driver) for most queries
- Optimized the buffer for reading
- stmt.Query now caches column metadata
- New Logo
- Changed the copyright header to include all contributors
- Improved the LOAD INFILE documentation
- The driver struct is now exported to make the driver directly accessible
- Refactored the driver tests
- Added more benchmarks and moved all to a separate file
- Other small refactoring
New Features:
- Added *old_passwords* support: Required in some cases, but must be enabled by adding `allowOldPasswords=true` to the DSN since it is insecure
- Added a `clientFoundRows` parameter: Return the number of matching rows instead of the number of rows changed on UPDATEs
- Added TLS/SSL support: Use a TLS/SSL encrypted connection to the server. Custom TLS configs can be registered and used
- Fixed MySQL 4.1 support: MySQL 4.1 sends packets with lengths which differ from the specification
- Convert to DB timezone when inserting `time.Time`
- Splitted packets (more than 16MB) are now merged correctly
- Fixed false positive `io.EOF` errors when the data was fully read
- Avoid panics on reuse of closed connections
- Fixed empty string producing false nil values
- Fixed sign byte for positive TIME fields
## Version 1.0 (2013-05-14)
Initial Release
# Contributing Guidelines
## Reporting Issues
Before creating a new Issue, please check first if a similar Issue [already exists]( or was [recently closed](
## Contributing Code
By contributing to this project, you share your code under the Mozilla Public License 2, as specified in the LICENSE file.
Don't forget to add yourself to the AUTHORS file.
### Code Review
Everyone is invited to review and comment on pull requests.
If it looks fine to you, comment with "LGTM" (Looks good to me).
If changes are required, notice the reviewers with "PTAL" (Please take another look) after committing the fixes.
Before merging the Pull Request, at least one [team member]( must have commented with "LGTM".
## Development Ideas
If you are looking for ideas for code contributions, please check our [Development Ideas]( Wiki page.
This diff is collapsed.
This diff is collapsed.
// Go MySQL Driver - A MySQL-Driver for Go's database/sql package
// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved.
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at
// +build appengine
package mysql
import (
func init() {
RegisterDial("cloudsql", cloudsql.Dial)
This diff is collapsed.
// Go MySQL Driver - A MySQL-Driver for Go's database/sql package
// Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved.
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at
package mysql
import (
const defaultBufSize = 4096
// A buffer which is used for both reading and writing.
// This is possible since communication on each connection is synchronous.
// In other words, we can't write and read simultaneously on the same connection.
// The buffer is similar to bufio.Reader / Writer but zero-copy-ish
// Also highly optimized for this particular use case.
type buffer struct {
buf []byte // buf is a byte buffer who's length and capacity are equal.
nc net.Conn
idx int
length int
timeout time.Duration
// newBuffer allocates and returns a new buffer.
func newBuffer(nc net.Conn) buffer {
return buffer{
buf: make([]byte, defaultBufSize),
nc: nc,
// fill reads into the buffer until at least _need_ bytes are in it
func (b *buffer) fill(need int) error {
n := b.length
// move existing data to the beginning
if n > 0 && b.idx > 0 {
copy(b.buf[0:n], b.buf[b.idx:])
// grow buffer if necessary
// TODO: let the buffer shrink again at some point
// Maybe keep the org buf slice and swap back?
if need > len(b.buf) {
// Round up to the next multiple of the default size
newBuf := make([]byte, ((need/defaultBufSize)+1)*defaultBufSize)
copy(newBuf, b.buf)
b.buf = newBuf
b.idx = 0
for {
if b.timeout > 0 {
if err :=; err != nil {
return err
nn, err :=[n:])
n += nn
switch err {
case nil:
if n < need {
b.length = n
return nil
case io.EOF:
if n >= need {
b.length = n
return nil
return io.ErrUnexpectedEOF
return err
// returns next N bytes from buffer.
// The returned slice is only guaranteed to be valid until the next read
func (b *buffer) readNext(need int) ([]byte, error) {
if b.length < need {
// refill
if err := b.fill(need); err != nil {
return nil, err
offset := b.idx
b.idx += need
b.length -= need
return b.buf[offset:b.idx], nil
// takeBuffer returns a buffer with the requested size.
// If possible, a slice from the existing buffer is returned.
// Otherwise a bigger buffer is made.
// Only one buffer (total) can be used at a time.
func (b *buffer) takeBuffer(length int) ([]byte, error) {
if b.length > 0 {
return nil, ErrBusyBuffer
// test (cheap) general case first
if length <= cap(b.buf) {
return b.buf[:length], nil
if length < maxPacketSize {
b.buf = make([]byte, length)
return b.buf, nil
// buffer is larger than we want to store.
return make([]byte, length), nil
// takeSmallBuffer is shortcut which can be used if length is
// known to be smaller than defaultBufSize.
// Only one buffer (total) can be used at a time.
func (b *buffer) takeSmallBuffer(length int) ([]byte, error) {
if b.length > 0 {
return nil, ErrBusyBuffer
return b.buf[:length], nil
// takeCompleteBuffer returns the complete existing buffer.
// This can be used if the necessary buffer size is unknown.
// cap and len of the returned buffer will be equal.
// Only one buffer (total) can be used at a time.
func (b *buffer) takeCompleteBuffer() ([]byte, error) {
if b.length > 0 {
return nil, ErrBusyBuffer
return b.buf, nil
// store stores buf, an updated buffer, if its suitable to do so.
func (b *buffer) store(buf []byte) error {
if b.length > 0 {
return ErrBusyBuffer
} else if cap(buf) <= maxPacketSize && cap(buf) > cap(b.buf) {
b.buf = buf[:cap(buf)]
return nil
// Go MySQL Driver - A MySQL-Driver for Go's database/sql package
// Copyright 2014 The Go-MySQL-Driver Authors. All rights reserved.
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at
package mysql
const defaultCollation = "utf8_general_ci"