gitlab-deps

Gitlab-deps is a simple build orchestration toolkit: it tracks dependencies between projects on a Gitlab instance and it can automatically rebuild dependencies when a project pipeline completes successfully.

It can track dependencies between projects by either of two supported ways:

• projects can add a .gitlab-deps file to the root of their repository, containing the fully qualified project URLs of their dependencies;
• gitlab-deps can scan Dockerfiles (in the repository root) and automatically infer dependencies based on FROM lines.

The implementation depends on HTTP hooks triggered by pipeline events: gitlab-deps runs a small HTTP server to respond to these requests and trigger new builds.

By default, since it is meant to be used as a trigger as the last step in a CI script, it will not navigate the dependency tree recursively but only look at the first-level dependencies: this way, further CI pipelines downstream will each trigger their own deps once the image has been rebuilt.

Installation

The tools require Python 3.

Install the tool either in a virtualenv of or system-wide with any of the standard Python installation mechanisms, for instance (using setuptools):

sudo python3 setup.py install

This will install the gitlab-deps command-line tool in /usr/local/bin. The tool has few dependencies (just the Gitlab API, Flask and Cheroot).

Usage

The toolkit is split into functional components (all wrapped in a single executable with different sub-commands):

• scan Gitlab and generate a dependency map
• manually trigger builds using the dependency map
• run a server that listens for Gitlab notifications and trigger builds

The tools talk to Gitlab using its API, so you're going to need an admin token in order to create new pipelines.

Common options

The tool must be pointed at your Gitlab instance with the --url command-line option, or alternatively using the GITLAB_URL environment variable.

You can pass an authentication token using the --token or --token-file command-line options. This is usually required in order to trigger CI pipelines, or to access private projects: the access token must have at least the api scope. Credentials can also be provided using the GITLAB_AUTH_TOKEN or GITLAB_AUTH_TOKEN_FILE environment variables.

Listing projects

The list-projects sub-command can be used to create a list of projects (and their branches) in the Gitlab instance. It provides some basic functionality for filtering (using the --search option), but it generates output suitable for grep, e.g. to filter a specific path prefix (Gitlab group):

gitlab-deps list-projects | grep ^path/to/my/group/

or to only select "master" branches:

gitlab-deps list-projects | grep ':master$'

The output from this command is just a list of project paths (with namespaces) and branch names, separated by a ':', one per line:

path/to/project1:master
path/to/project1:test-branch
path/to/project2:master
...

Computing dependencies

The deps sub-command will scan the projects and their repositories, and it will produce a list of all the edges in the dependency graph. It takes a list of project_path:branch specs as input (as produced by the list-projects sub-command), and it will produce a list of edges as whitespace-separated project:branch pairs, e.g.:

project:master dependency1:master
project:master dependency2:master

The output format is once again meant to be processed with standard UNIX tools such as awk and grep.

Configuring pipeline_events hooks

To work, gitlab-deps needs a HTTP hook for pipeline_events on all projects that have dependencies. Since setting this up in Gitlab is a manual and laborious process, the set-hooks sub-command is provided to do this automatically using the API. The intended usage is to run it on the right-hand side of the dependency edges (i.e. the list of projects/branches that actually have dependencies):

gitlab-deps deps | awk '{print $2}' | gitlab-deps set-hooks

One-off rebuilds

The rebuild sub-Command will trigger a rebuild of all the dependencies of a given project, possibly waiting for the CI pipelines to complete. Pass a qualified project name and branch as a command-line argument. The dependency graph (list of edges as produced by the deps sub-command) must also be provided, either as a file or on standard input.

The --recurse option will traverse the dependency tree recursively, waiting for CI pipelines to complete so that they are built in the right order.

Running the server

The gitlab-deps tool has a server command to start a simple HTTP server that receives the pipeline_events webhooks from Gitlab, and trigger builds for project dependencies.

The server command requires an address to bind to, specified using the --host and --port options. It is also possible to enforce authentication of the webhook with a secret token (X-Gitlab-Token) using the --webhook-auth-token option.

Also note that the server does not have any TLS support: if necessary, it is best to use a dedicated reverse proxy (Apache, NGINX, etc).

For example, assuming the webhook server will be running on the same machine as Gitlab itself, and that the Gitlab authentication token is stored in /etc/gitlab_docker_token:

gitlab-deps deps \
| gitlab-deps server \
    --url=https://my.gitlab \
    --token-file=/etc/gitlab_docker_token \
    --host=127.0.0.1 --port=14001

If configuring webhooks manually (rather than with set-hooks), create a new webhook with the URL http://localhost:14001/, and with the Trigger checkbox set only on Pipeline events.