5.25 KB
Newer Older
ale's avatar
ale committed
1 2 3 4 5 6

Automatically rebuild all the downstream dependencies of Docker-based
projects on a Gitlab instance.

7 8 9 10
It scans the *master* branch of all repositories containing a
Dockerfile looking for FROM lines and navigates the resulting
dependency tree to find all projects that needs to be rebuilt when
their base image (or an upstream image thereof) changes.
ale's avatar
ale committed
11 12 13 14 15 16 17

By default, since it is meant to be used as a trigger as the last step
in a CI script, it will not navigate the dependency tree recursively
but only look at the first-level dependencies: this way, further CI
pipelines downstream will each trigger their own deps once the image
has been rebuilt.

# Installation
ale's avatar
ale committed

20 21 22
Install the tool either in a virtualenv of or system-wide with any of
the standard Python installation mechanisms, for instance (using
ale's avatar
ale committed
23 24 25 26 27

sudo python install

28 29 30 31 32 33 34 35 36 37 38
This will install the *gitlab-docker-autodep* command-line tool in
/usr/local/bin. The tool should work with either Python 2 and Python
3, and it has few dependencies (just the [Gitlab
API](, and

# Usage

There are two modes of using this software: either as a one-shot
command-line tool, or as a standalone HTTP server responding to Gitlab
Webhook requests, to integrate Docker dependencies with Gitlab CI.
ale's avatar
ale committed

40 41 42 43 44 45
In both cases, the program is configured via command-line options.

## Common options

The tool must be pointed at your Gitlab instance with the *--url*
command-line option, 
ale's avatar
ale committed
46 47

You can pass an authentication token using the *--token* command-line
option. This is usually required in order to trigger CI pipelines: the
access token must have the *api* scope.
ale's avatar
ale committed
50 51 52 53 54 55 56

The tool will only examine Docker images hosted on the Docker registry
associated with the Gitlab instance. By default the registry name is
automatically derived from the server URL (adding a *registry*
prefix), but it can be changed with the *--registry* command-line

57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
### Scope

On larger Gitlab instances, parsing Dockerfiles for all projects can
be an expensive (long) operation. The program offers two options to
manage the scope of the dependency analysis: *--match* and *--filter*.

The former, *--match*, allows for filtering the project list on the
server side, using a Gitlab search query. The latter, *--filter*,
applies a regular expression to the project names (including
namespaces) before parsing their dependencies. Combining the two, for
example, it is possible to efficiently limit the scope of the tool to
a specific namespace:

gitlab-docker-autodep ... --match myns --filter ^myns/ ...

Note that, when building the dependency tree:

* tags in FROM lines are ignored
* only the *master* branch of repositories is scanned for Dockerfiles

This might lead to more rebuilds than strictly necessary.

81 82 83 84 85 86 87
## Command-line

The `rebuild` command will trigger a rebuild of all the dependencies
of a given Docker image, possibly waiting for the CI pipelines to
complete. Pass the qualified Docker image name (as it appears on FROM
lines in Dockerfiles) as a command-line argument.

ale's avatar
ale committed
The tool will print out the project names of the dependencies it
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131
found. The *--recurse* option will traverse the dependency tree
recursively, waiting for CI pipelines to complete so that they are
built in the right order.

It is possible to limit the scope of the initial dependency scan
(which is an expensive operation) to projects matching a Gitlab search
keyword using the *--match* option.

## Gitlab CI integration

In order to automatically rebuild the dependencies of a Docker image
built using Gitlab CI, it is possible to run *gitlab-docker-autodep*
as a webhook: this way, whenever a successful CI pipeline completes,
you can trigger a rebuild of the dependencies.

To do this, use the *server* command of *gitlab-docker-autodep*, and
specify the address to bind to using the *--host* and *--port*
options. It is also possible to enforce authentication of the webhook
with a secret token
using the *--webhook-auth-token* option.

When running in this mode, it is assumed that all your Docker-related
projects have webhooks set up to rebuild their dependencies, so
*gitlab-docker-autodep* will only trigger a build of the immediate
dependencies of a project.

Also note that the server does not have any TLS support: if necessary,
it is best to use a dedicated reverse proxy (Apache, NGINX, etc).

For example, assuming the webhook server will be running on the same
machine as Gitlab itself, and that the Gitlab authentication token is
stored in */etc/gitlab_docker_token*:

gitlab-docker-autodep \
    --url=https://my.gitlab \
    --token=$(< /etc/gitlab_docker_token) \
    server \
    --host= --port=14001

You can then configure your project's webhooks with the URL
ale's avatar
ale committed
`http://localhost:14001/`, with the *Trigger* checkbox set only
133 134 135 136 137 138 139 140 141 142 143 144
on *Pipeline events*.

It can be useful to run the *rebuild* command from a cron job, for
instance in order to rebuild images on a periodic schedule, and
assuming all your projects share a common base image:

50 5 * * * root gitlab-docker-autodep 
                --token=$(< /etc/gitlab_docker_token)
                rebuild $MY_BASE_IMAGE