gitlab-docker-autodep

Automatically rebuild all the downstream dependencies of Docker-based projects on a Gitlab instance.

It scans the master branch of all repositories containing a Dockerfile looking for FROM lines and navigates the resulting dependency tree to find all projects that needs to be rebuilt when their base image (or an upstream image thereof) changes.

By default, since it is meant to be used as a trigger as the last step in a CI script, it will not navigate the dependency tree recursively but only look at the first-level dependencies: this way, further CI pipelines downstream will each trigger their own deps once the image has been rebuilt.

Installation

Install the tool either in a virtualenv of or system-wide with any of the standard Python installation mechanisms, for instance (using setuptools):

sudo python setup.py install

This will install the gitlab-docker-autodep command-line tool in /usr/local/bin. The tool should work with either Python 2 and Python 3, and it has few dependencies (just the Gitlab API, and Flask).

Usage

The tool is split into functional components:

• scan Gitlab and generate a dependency map (stored as a JSON file)
• manually trigger builds using the dependency map
• run a server that listens for Gitlab notifications and trigger builds

In all cases, the program is configured via command-line options.

Common options

The tool must be pointed at your Gitlab instance with the --url command-line option,

You can pass an authentication token using the --token or --token-file command-line options. This is usually required in order to trigger CI pipelines: the access token must have the api scope.

The tool will only examine Docker images hosted on the Docker registry associated with the Gitlab instance. By default the registry name is automatically derived from the server URL (adding a registry prefix), but it can be changed with the --registry command-line option.

Scope

On larger Gitlab instances, parsing Dockerfiles for all projects can be an expensive (long) operation. The program offers two options to manage the scope of the dependency analysis: --match and --filter.

The former, --match, allows for filtering the project list on the server side, using a Gitlab search query. The latter, --filter, applies a regular expression to the project names (including namespaces) before parsing their dependencies. Combining the two, for example, it is possible to efficiently limit the scope of the tool to a specific namespace:

gitlab-docker-autodep deps --match myns --filter ^myns/ ...

Note that, when building the dependency tree:

• tags in FROM lines are ignored
• only the master branch of repositories is scanned for Dockerfiles

This might lead to more rebuilds than strictly necessary.

Command-line

The rebuild command will trigger a rebuild of all the dependencies of a given Docker image, possibly waiting for the CI pipelines to complete. Pass the qualified Docker image name (as it appears on FROM lines in Dockerfiles) as a command-line argument.

The tool will print out the project names of the dependencies it found. The --recurse option will traverse the dependency tree recursively, waiting for CI pipelines to complete so that they are built in the right order.

It is possible to limit the scope of the initial dependency scan (which is an expensive operation) to projects matching a Gitlab search keyword using the --match option.

Gitlab CI integration

In order to automatically rebuild the dependencies of a Docker image built using Gitlab CI, it is possible to run gitlab-docker-autodep as a webhook: this way, whenever a successful CI pipeline completes, you can trigger a rebuild of the dependencies.

To do this, use the server command of gitlab-docker-autodep, and specify the address to bind to using the --host and --port options. It is also possible to enforce authentication of the webhook with a secret token (X-Gitlab-Token) using the --webhook-auth-token option.

When running in this mode, it is assumed that all your Docker-related projects have webhooks set up to rebuild their dependencies, so gitlab-docker-autodep will only trigger a build of the immediate dependencies of a project.

Also note that the server does not have any TLS support: if necessary, it is best to use a dedicated reverse proxy (Apache, NGINX, etc).

For example, assuming the webhook server will be running on the same machine as Gitlab itself, and that the Gitlab authentication token is stored in /etc/gitlab_docker_token:

gitlab-docker-autodep \
    --url=https://my.gitlab \
    --token-file=/etc/gitlab_docker_token \
    server \
    --deps=deps.json
    --host=127.0.0.1 --port=14001

You can then configure your project's webhooks with the URL http://localhost:14001/, with the Trigger checkbox set only on Pipeline events.

Then you should generate the deps.json dependency map periodically, for instance with a cron job:

*/30 * * * * root gitlab-docker-autodep
                  --url=https://my.gitlab 
                  --token-file=/etc/gitlab_docker_token
                  deps > deps.json

It can be useful to run the rebuild command from a cron job, for instance in order to rebuild images on a periodic schedule, and assuming all your projects share a common base image:

50 5 * * * root gitlab-docker-autodep 
                --url=https://my.gitlab
                --token-file=/etc/gitlab_docker_token
                rebuild $MY_BASE_IMAGE