Merge branch 'txn' into 'master'

Txn

See merge request ai3/accountserver!1

API.md

@@ -130,3 +130,157 @@ Actions:
 ### web hosting (website / db / FTP account or equivalent)
 
 ...
+
+
+# API Reference
+
+## User endpoints
+
+The following API endpoints invoke operations on an individual
+user. Access is allowed for admins, and for the user itself.
+
+Most requests dealing with encryption keys are so-called *privileged*
+requests, and will require the user's password in the *cur_password*
+parameter in order to decrypt the existing encryption keys.
+
+### `/api/user/get`
+
+Retrieve information about a user and all its associated resources.
+
+Request parameters:
+
+* `username` - user to fetch
+* `sso` - SSO ticket
+
+### `/api/user/change_password`
+
+Change the primary authentication password for a user. This operation
+will update the user's storage encryption keys, or initialize them if
+they do not exist.
+
+Request parameters:
+
+* `username` - name of the user
+* `sso` - SSO ticket
+* `cur_password` - current valid password for the user
+* `password` - new password (unencrypted)
+
+### `/api/user/set_password_recovery_hint`
+
+Sets the secondary authentication password (a hint / response pair,
+used to recover the primary credentials) for a user. This operation
+will update the user's storage encryption keys, or initialize them if
+they do not exist yet.
+
+Request parameters:
+
+* `username` - name of the user
+* `sso` - SSO ticket
+* `cur_password` - current valid password for the user
+* `hint` - the secondary authentication hint
+* `response` - the secondary authentication response (effectively a
+  password, unencrypted)
+
+### `/api/user/enable_otp`
+
+Enable TOTP for a user. The server can generate a new TOTP secret if
+necessary, or it can be generated by the caller (usually better as it
+allows for a better validation UX).
+
+Request parameters:
+
+* `username` - name of the user
+* `sso` - SSO ticket
+* `totp_secret` - new TOTP secret (optional)
+
+### `/api/user/disable_otp`
+
+Disable TOTP for a user. Existing 2FA credentials will be wiped as
+well.
+
+Request parameters:
+
+* `username` - name of the user
+* `sso` - SSO ticket
+
+### `/api/user/create_app_specific_password`
+
+Create a new application-specific password. 2FA must already be
+enabled for the user. A new random password will be generated by the
+server and returned in the response. A new copy of the encryption key
+will be encrypted with the new application-specific password.
+
+ASPs are identified by a unique random ID that is also automatically
+generated by the server.
+
+Request parameters:
+
+* `username` - name of the user
+* `sso` - SSO ticket
+* `cur_password` - current valid password for the user
+* `service` - service that the password should be valid for
+* `notes` - a user-controlled comment about the client
+
+### `/api/user/delete_app_specific_password`
+
+Delete an application-specific password (and the associated encryption
+key).
+
+Request parameters:
+
+* `username` - name of the user
+* `sso` - SSO ticket
+* `asp_id` - ID of the app-specific password
+
+
+## Resource endpoints
+
+These API endpoints manipulate individual resources regardless of
+which user they belong to. Access is normally granted to admins and to
+the user that owns a resource, but some operations are restricted to
+admins only.
+
+### `/api/resource/enable`
+
+Enable a resource (admin-only).
+
+Request parameters:
+
+* `resource_id` - resource ID
+* `sso` - SSO ticket
+* `comment` - notes on the operation
+
+### `/api/resource/disable`
+
+Disable a resource (admin-only).
+
+Request parameters:
+
+* `resource_id` - resource ID
+* `sso` - SSO ticket
+* `comment` - notes on the operation
+
+### `/api/resource/move`
+
+Move a resource between shards (admin-only).
+
+Resources that are part of a group (for instance websites and DAV
+accounts) are moved together, so this request might end up moving more
+than one resource.
+
+Request parameters:
+
+* `resource_id` - resource ID
+* `sso` - SSO ticket
+* `shard` - new shard
+
+### `/api/resource/create`
+
+Create one or more resources associated with a user. Note that if
+creating multiple resources, they must all belong to the same user.
+
+Request parameters:
+
+* `sso` - SSO ticket
+* `resources` - list of resource objects to create
+

DATAMODEL.md

+A few notes on the data model
+======
+
+The data model used by the accountserver tries to be straightforward
+and simple, but there are numerous aspects about the assumed semantics
+that require further discussion.
+
+At its core, it represents *users* and their associated
+*resources*. Resources can represent accounts on services of different
+type, and can be nested if there is a direct and relevant hierarchical
+relation.
+
+## Authentication
+
+The major issue with the LDAP backend is that it's not easy to do
+joins (you need to do two separate queries, and most open source
+clients aren't even coded to do that). This makes it hard to go, for
+example, from the email resource object to the main user account
+object.
+
+The issue of authentication is made complex by the choice of specific
+services and desired UX that we have adopted. Specifically, the
+decision of having the e-mail coincide with the primary user identity
+(along with the availability of web-based single sign-on), and the UX
+decision of presenting a single "authentication control surface" to
+the user (so that, for instance, you don't have to separately set up
+2FA for the main user account and your email) determine the need to
+unify authentication details for the main user object and the email
+resource(s).
+
+In practice, this means that the API offers its authentication-related
+methods on the user object rather than on the email resource. While
+the implementation on a SQL backend would be straightforward
+(credentials would simply be fields, or subtables, of the *user*
+table), in the LDAP backend we are going to rely on denormalization
+(i.e. copying the same attribute to multiple objects) and careful
+splitting of some attributes to where they are used. For an example of
+the latter, consider 2FA, where the split between interactive and
+non-interactive clients allows us to:
+
+* put the TOTP secret on the user object, because that's used by the
+  interactive authentication service (the single sign-on login server
+  application);
+* put application-specific passwords (used by non-interactive clients)
+  on the email resource LDAP objects, which is where the email service
+  authenticates its users. Same for the storage encryption keys.

README.md

@@ -50,3 +50,18 @@ esprimendo una relazione di associazione di qualche tipo (come ad
 esempio tra siti web e database MySQL).
 
 Lo schema è definito esplicitamente in [types.go](types.go).
+
+
+# Testing
+
+Per testare questo attrezzo purtroppo serve Java (basta un JRE, il
+runtime environment). Questo perché non è affatto facile implementare
+un server LDAP per i test, quindi si è scelto di usarne uno esistente:
+in particolare un'implementazione Java... su un sistema Debian:
+
+```
+sudo apt install default-jre-headless
+```
+
+dovrebbe bastare (oltre a *golang-go* ovviamente) per poter eseguire i
+test con successo.

actions_test.go

@@ -2,6 +2,7 @@ package accountserver
 
 import (
 	"context"
+	"errors"
 	"testing"
 
 	sso "git.autistici.org/id/go-sso"
@@ -15,15 +16,38 @@ type fakeBackend struct {
 	encryptionKeys       map[string][]*UserEncryptionKey
 }
 
+func (b *fakeBackend) NewTransaction() (TX, error) {
+	return b, nil
+}
+
+func (b *fakeBackend) Commit(_ context.Context) error {
+	return nil
+}
+
 func (b *fakeBackend) GetUser(_ context.Context, username string) (*User, error) {
 	return b.users[username], nil
 }
 
-func (b *fakeBackend) GetResource(_ context.Context, username, resourceID string) (*Resource, error) {
-	return b.resources[username][resourceID], nil
+func (b *fakeBackend) CreateUser(_ context.Context, user *User) error {
+	b.users[user.Name] = user
+	return nil
+}
+
+func (b *fakeBackend) GetResource(_ context.Context, resourceID ResourceID) (*Resource, error) {
+	return b.resources[resourceID.User()][resourceID.String()], nil
 }
 
-func (b *fakeBackend) UpdateResource(_ context.Context, username string, r *Resource) error {
+func (b *fakeBackend) UpdateResource(_ context.Context, r *Resource) error {
+	b.resources[r.ID.User()][r.ID.String()] = r
+	return nil
+}
+
+func (b *fakeBackend) CreateResource(_ context.Context, r *Resource) error {
+	if _, ok := b.resources[r.ID.User()][r.ID.String()]; ok {
+		return errors.New("resource already exists")
+	}
+
+	b.resources[r.ID.User()][r.ID.String()] = r
 	return nil
 }
 
@@ -32,7 +56,11 @@ func (b *fakeBackend) SetUserPassword(_ context.Context, user *User, password st
 	return nil
 }
 
-func (b *fakeBackend) SetResourcePassword(_ context.Context, username string, r *Resource, password string) error {
+func (b *fakeBackend) SetPasswordRecoveryHint(_ context.Context, user *User, hint, response string) error {
+	return nil
+}
+
+func (b *fakeBackend) SetResourcePassword(_ context.Context, r *Resource, password string) error {
 	return nil
 }
 
@@ -42,6 +70,7 @@ func (b *fakeBackend) GetUserEncryptionKeys(_ context.Context, user *User) ([]*U
 
 func (b *fakeBackend) SetUserEncryptionKeys(_ context.Context, user *User, keys []*UserEncryptionKey) error {
 	b.encryptionKeys[user.Name] = keys
+	b.users[user.Name].HasEncryptionKeys = true
 	return nil
 }
 
@@ -66,7 +95,16 @@ func (b *fakeBackend) DeleteUserTOTPSecret(_ context.Context, user *User) error
 	return nil
 }
 
-func (b *fakeBackend) HasAnyResource(_ context.Context, rsrcs []string) (bool, error) {
+func (b *fakeBackend) HasAnyResource(_ context.Context, rsrcs []FindResourceRequest) (bool, error) {
+	for _, fr := range rsrcs {
+		for _, ur := range b.resources {
+			for _, r := range ur {
+				if r.ID.Type() == fr.Type && r.ID.Name() == fr.Name {
+					return true, nil
+				}
+			}
+		}
+	}
 	return false, nil
 }
 
@@ -76,7 +114,7 @@ type fakeValidator struct {
 	adminUser string
 }
 
-func (v *fakeValidator) Validate(tkt string, nonce string, service string, _ []string) (*sso.Ticket, error) {
+func (v *fakeValidator) Validate(tkt, nonce, service string, _ []string) (*sso.Ticket, error) {
 	// The sso ticket username is just the ticket itself.
 	var groups []string
 	if tkt == v.adminUser {
@@ -90,40 +128,79 @@ func (v *fakeValidator) Validate(tkt string, nonce string, service string, _ []s
 	}, nil
 }
 
+func (b *fakeBackend) addUser(user *User) {
+	b.users[user.Name] = user
+	b.resources[user.Name] = make(map[string]*Resource)
+	for _, r := range user.Resources {
+		b.resources[user.Name][r.ID.String()] = r
+	}
+}
+
 func createFakeBackend() *fakeBackend {
 	fb := &fakeBackend{
-		users: map[string]*User{
-			"testuser": &User{
-				Name: "testuser",
-				Resources: []*Resource{
-					{
-						ID:     "email/testuser@example.com",
-						Name:   "testuser@example.com",
-						Type:   ResourceTypeEmail,
-						Status: ResourceStatusActive,
-						Email:  &Email{},
-					},
-				},
-			},
+		users: make(map[string]*User),
+		resources: map[string]map[string]*Resource{
+			// For global (user-less) resources, where CreateUser is not called.
+			"": make(map[string]*Resource),
 		},
-		resources:            make(map[string]map[string]*Resource),
 		passwords:            make(map[string]string),
 		appSpecificPasswords: make(map[string][]*AppSpecificPasswordInfo),
 		encryptionKeys:       make(map[string][]*UserEncryptionKey),
 	}
+	fb.addUser(&User{
+		Name: "testuser",
+		Resources: []*Resource{
+			{
+				ID:     NewResourceID(ResourceTypeEmail, "testuser", "testuser@example.com"),
+				Name:   "testuser@example.com",
+				Status: ResourceStatusActive,
+				Email: &Email{
+					Maildir: "example.com/testuser",
+				},
+			},
+			{
+				ID:     NewResourceID(ResourceTypeDAV, "testuser", "dav1"),
+				Name:   "dav1",
+				Status: ResourceStatusActive,
+				DAV: &WebDAV{
+					Homedir: "/home/dav1",
+				},
+			},
+		},
+	})
 	return fb
 }
 
 func testConfig() *Config {
 	var c Config
+	c.ForbiddenUsernames = []string{"root"}
+	c.AvailableDomains = map[string][]string{
+		ResourceTypeEmail:       []string{"example.com"},
+		ResourceTypeMailingList: []string{"example.com"},
+	}
 	c.SSO.Domain = "mydomain"
 	c.SSO.Service = "service/"
 	c.SSO.AdminGroup = testAdminGroupName
+	c.Shards.Available = map[string][]string{
+		ResourceTypeEmail:       []string{"host1", "host2", "host3"},
+		ResourceTypeMailingList: []string{"host1", "host2", "host3"},
+		ResourceTypeWebsite:     []string{"host1", "host2", "host3"},
+		ResourceTypeDomain:      []string{"host1", "host2", "host3"},
+		ResourceTypeDAV:         []string{"host1", "host2", "host3"},
+	}
+	c.Shards.Allowed = c.Shards.Available
 	return &c
 }
 
+func testService(admin string) (*AccountService, TX) {
+	be := createFakeBackend()
+	svc, _ := newAccountServiceWithSSO(be, testConfig(), &fakeValidator{admin})
+	tx, _ := be.NewTransaction()
+	return svc, tx
+}
+
 func TestService_GetUser(t *testing.T) {
-	svc := newAccountServiceWithSSO(createFakeBackend(), testConfig(), &fakeValidator{})
+	svc, tx := testService("")
 
 	req := &GetUserRequest{
 		RequestBase: RequestBase{
@@ -131,7 +208,7 @@ func TestService_GetUser(t *testing.T) {
 			SSO:      "testuser",
 		},
 	}
-	resp, err := svc.GetUser(context.TODO(), req)
+	resp, err := svc.GetUser(context.TODO(), tx, req)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -141,7 +218,7 @@ func TestService_GetUser(t *testing.T) {
 }
 
 func TestService_Auth(t *testing.T) {
-	svc := newAccountServiceWithSSO(createFakeBackend(), testConfig(), &fakeValidator{"adminuser"})
+	svc, tx := testService("adminuser")
 
 	for _, td := range []struct {
 		sso        string
@@ -157,7 +234,7 @@ func TestService_Auth(t *testing.T) {
 				SSO:      td.sso,
 			},
 		}
-		_, err := svc.GetUser(context.TODO(), req)
+		_, err := svc.GetUser(context.TODO(), tx, req)
 		if err != nil {
 			if !IsAuthError(err) {
 				t.Errorf("error for sso_user=%s is not an auth error: %v", td.sso, err)
@@ -172,21 +249,38 @@ func TestService_Auth(t *testing.T) {
 
 func TestService_ChangePassword(t *testing.T) {
 	fb := createFakeBackend()
-	svc := newAccountServiceWithSSO(fb, testConfig(), &fakeValidator{})
+	tx, _ := fb.NewTransaction()
+	svc, _ := newAccountServiceWithSSO(fb, testConfig(), &fakeValidator{})
 
-	req := &ChangeUserPasswordRequest{
-		PrivilegedRequestBase: PrivilegedRequestBase{
-			RequestBase: RequestBase{
-				Username: "testuser",
-				SSO:      "testuser",
-			},
-			CurPassword: "cur",
-		},
-		Password: "password",
+	testdata := []struct {
+		password    string
+		newPassword string
+		expectedOk  bool
+	}{
+		// Ordering is important as it is meant to emulate
+		// setting the password, failing to reset it, then
+		// succeeding.
+		{"password", "new_password", true},
+		{"BADPASS", "new_password_2", false},
+		{"new_password", "new_password_2", true},
 	}
-	err := svc.ChangeUserPassword(context.TODO(), req)
-	if err != nil {
-		t.Fatal(err)
+	for _, td := range testdata {
+		req := &ChangeUserPasswordRequest{
+			PrivilegedRequestBase: PrivilegedRequestBase{
+				RequestBase: RequestBase{
+					Username: "testuser",
+					SSO:      "testuser",
+				},
+				CurPassword: td.password,
+			},
+			Password: td.newPassword,
+		}
+		err := svc.ChangeUserPassword(context.TODO(), tx, req)
+		if err == nil && !td.expectedOk {
+			t.Fatalf("ChangeUserPassword(old=%s new=%s) should have failed but didn't", td.password, td.newPassword)
+		} else if err != nil && td.expectedOk {
+			t.Fatalf("ChangeUserPassword(old=%s new=%s) failed: %v", td.password, td.newPassword, err)
+		}
 	}
 
 	if _, ok := fb.passwords["testuser"]; !ok {
@@ -196,3 +290,186 @@ func TestService_ChangePassword(t *testing.T) {
 		t.Errorf("no encryption keys were set")
 	}
 }
+
+// Lower level test that basically corresponds to the same operations
+// as TestService_ChangePassword above, but exercises the
+// initializeUserEncryptionKeys / updateUserEncryptionKeys code path
+// directly.
+func TestService_EncryptionKeys(t *testing.T) {
+	fb := createFakeBackend()
+	svc, _ := newAccountServiceWithSSO(fb, testConfig(), &fakeValidator{})
+	tx, _ := fb.NewTransaction()
+	ctx := context.Background()
+
+	user, _ := getUserOrDie(ctx, tx, "testuser")
+
+	// Set the keys to something.
+	keys, _, err := svc.initializeEncryptionKeys(ctx, tx, user, "password")
+	if err != nil {
+		t.Fatal("init", err)
+	}
+	if err := tx.SetUserEncryptionKeys(ctx, user, keys); err != nil {
+		t.Fatal("SetUserEncryptionKeys", err)
+	}
+	if n := len(fb.encryptionKeys["testuser"]); n != 1 {
+		t.Fatalf("found %d encryption keys, expected 1", n)
+	}
+
+	// Try to read (decrypt) them again using bad / good passwords.
+	if _, _, err := svc.readOrInitializeEncryptionKeys(ctx, tx, user, "BADPASS", "new_password"); err == nil {
+		t.Fatal("read with bad password did not fail")
+	}
+	if _, _, err := svc.readOrInitializeEncryptionKeys(ctx, tx, user, "password", "new_password"); err != nil {
+		t.Fatal("readOrInitialize", err)
+	}
+}
+
+// Try adding aliases to the email resource.
+func TestService_AddEmailAlias(t *testing.T) {
+	svc, tx := testService("")
+
+	testdata := []struct {
+		addr       string
+		expectedOk bool
+	}{
+		{"alias@example.com", true},
+		{"another-example-address@example.com", true},
+		{"root@example.com", false},
+		{"alias@other-domain.com", false},
+	}
+	for _, td := range testdata {
+		req := &AddEmailAliasRequest{
+			ResourceRequestBase: ResourceRequestBase{
+				SSO:        "testuser",
+				ResourceID: NewResourceID(ResourceTypeEmail, "testuser", "testuser@example.com"),
+			},
+			Addr: td.addr,
+		}
+		err := svc.AddEmailAlias(context.TODO(), tx, req)
+		if err != nil && td.expectedOk {
+			t.Errorf("AddEmailAlias(%s) failed: %v", td.addr, err)
+		} else if err == nil && !td.expectedOk {
+			t.Errorf("AddEmailAlias(%s) did not fail but should have", td.addr)
+		}
+	}
+}
+
+func TestService_CreateResource(t *testing.T) {
+	svc, tx := testService("admin")
+
+	req := &CreateResourcesRequest{
+		SSO: "admin",
+		Resources: []*Resource{
+			&Resource{
+				ID:            NewResourceID(ResourceTypeDAV, "testuser", "dav2"),
+				Name:          "dav2",
+				Status:        ResourceStatusActive,
+				Shard:         "host2",
+				OriginalShard: "host2",
+				DAV: &WebDAV{
+					Homedir: "/home/dav2",
+				},
+			},
+		},
+	}
+
+	// The request should succeed the first time around.
+	_, err := svc.CreateResources(context.Background(), tx, req)
+	if err != nil {
+		t.Fatal("CreateResources", err)
+	}
+
+	// The object already exists, so the same request should fail now.
+	_, err = svc.CreateResources(context.Background(), tx, req)
+	if err == nil {
+		t.Fatal("creating a duplicate resource did not fail")
+	}
+}
+
+func TestService_CreateResource_List(t *testing.T) {
+	svc, tx := testService("admin")
+
+	// A list is an example of a user-less (global) resource.
+	req := &CreateResourcesRequest{
+		SSO: "admin",
+		Resources: []*Resource{
+			&Resource{
+				ID:            NewResourceID(ResourceTypeMailingList, "list@example.com"),
+				Name:          "list@example.com",
+				Status:        ResourceStatusActive,
+				Shard:         "host2",
+				OriginalShard: "host2",
+				List: &MailingList{
+					Admins: []string{"testuser@example.com"},
+				},
+			},
+		},
+	}
+
+	// The request should succeed.
+	_, err := svc.CreateResources(context.Background(), tx, req)
+	if err != nil {
+		t.Fatal("CreateResources", err)
+	}
+}
+
+func TestService_CreateUser(t *testing.T) {
+	svc, tx := testService("admin")
+
+	req := &CreateUserRequest{
+		SSO: "admin",
+		User: &User{
+			Name: "testuser2@example.com",
+			Resources: []*Resource{
+				&Resource{
+					ID:            NewResourceID(ResourceTypeEmail, "testuser2@example.com", "testuser2@example.com"),