README.md 5.25 KB
Newer Older
ale's avatar
ale committed
1 2 3 4 5 6
gitlab-docker-autodep
===

Automatically rebuild all the downstream dependencies of Docker-based
projects on a Gitlab instance.

7 8 9 10
It scans the *master* branch of all repositories containing a
Dockerfile looking for FROM lines and navigates the resulting
dependency tree to find all projects that needs to be rebuilt when
their base image (or an upstream image thereof) changes.
ale's avatar
ale committed
11 12 13 14 15 16 17

By default, since it is meant to be used as a trigger as the last step
in a CI script, it will not navigate the dependency tree recursively
but only look at the first-level dependencies: this way, further CI
pipelines downstream will each trigger their own deps once the image
has been rebuilt.

18
# Installation
ale's avatar
ale committed
19

20 21 22
Install the tool either in a virtualenv of or system-wide with any of
the standard Python installation mechanisms, for instance (using
*setuptools*):
ale's avatar
ale committed
23 24 25 26 27

```
sudo python setup.py install
```

28 29 30 31 32 33 34 35 36 37 38
This will install the *gitlab-docker-autodep* command-line tool in
/usr/local/bin. The tool should work with either Python 2 and Python
3, and it has few dependencies (just the [Gitlab
API](https://python-gitlab.readthedocs.io/en/stable/), and
[Flask](https://flask.pocoo.org/)).

# Usage

There are two modes of using this software: either as a one-shot
command-line tool, or as a standalone HTTP server responding to Gitlab
Webhook requests, to integrate Docker dependencies with Gitlab CI.
ale's avatar
ale committed
39

40 41 42 43 44 45
In both cases, the program is configured via command-line options.

## Common options

The tool must be pointed at your Gitlab instance with the *--url*
command-line option, 
ale's avatar
ale committed
46 47

You can pass an authentication token using the *--token* command-line
48
option. This is usually required in order to trigger CI pipelines: the
49
access token must have the *api* scope.
ale's avatar
ale committed
50 51 52 53 54 55 56

The tool will only examine Docker images hosted on the Docker registry
associated with the Gitlab instance. By default the registry name is
automatically derived from the server URL (adding a *registry*
prefix), but it can be changed with the *--registry* command-line
option.

57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
### Scope

On larger Gitlab instances, parsing Dockerfiles for all projects can
be an expensive (long) operation. The program offers two options to
manage the scope of the dependency analysis: *--match* and *--filter*.

The former, *--match*, allows for filtering the project list on the
server side, using a Gitlab search query. The latter, *--filter*,
applies a regular expression to the project names (including
namespaces) before parsing their dependencies. Combining the two, for
example, it is possible to efficiently limit the scope of the tool to
a specific namespace:

```
gitlab-docker-autodep ... --match myns --filter ^myns/ ...
```

Note that, when building the dependency tree:

* tags in FROM lines are ignored
* only the *master* branch of repositories is scanned for Dockerfiles

This might lead to more rebuilds than strictly necessary.

81 82 83 84 85 86 87
## Command-line

The `rebuild` command will trigger a rebuild of all the dependencies
of a given Docker image, possibly waiting for the CI pipelines to
complete. Pass the qualified Docker image name (as it appears on FROM
lines in Dockerfiles) as a command-line argument.

ale's avatar
ale committed
88
The tool will print out the project names of the dependencies it
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131
found. The *--recurse* option will traverse the dependency tree
recursively, waiting for CI pipelines to complete so that they are
built in the right order.

It is possible to limit the scope of the initial dependency scan
(which is an expensive operation) to projects matching a Gitlab search
keyword using the *--match* option.

## Gitlab CI integration

In order to automatically rebuild the dependencies of a Docker image
built using Gitlab CI, it is possible to run *gitlab-docker-autodep*
as a webhook: this way, whenever a successful CI pipeline completes,
you can trigger a rebuild of the dependencies.

To do this, use the *server* command of *gitlab-docker-autodep*, and
specify the address to bind to using the *--host* and *--port*
options. It is also possible to enforce authentication of the webhook
with a secret token
([X-Gitlab-Token](https://docs.gitlab.com/ee/user/project/integrations/webhooks.html))
using the *--webhook-auth-token* option.

When running in this mode, it is assumed that all your Docker-related
projects have webhooks set up to rebuild their dependencies, so
*gitlab-docker-autodep* will only trigger a build of the immediate
dependencies of a project.

Also note that the server does not have any TLS support: if necessary,
it is best to use a dedicated reverse proxy (Apache, NGINX, etc).

For example, assuming the webhook server will be running on the same
machine as Gitlab itself, and that the Gitlab authentication token is
stored in */etc/gitlab_docker_token*:

```
gitlab-docker-autodep \
    --url=https://my.gitlab \
    --token=$(< /etc/gitlab_docker_token) \
    server \
    --host=127.0.0.1 --port=14001
```

You can then configure your project's webhooks with the URL
ale's avatar
ale committed
132
`http://localhost:14001/`, with the *Trigger* checkbox set only
133 134 135 136 137 138 139 140 141 142 143 144
on *Pipeline events*.

It can be useful to run the *rebuild* command from a cron job, for
instance in order to rebuild images on a periodic schedule, and
assuming all your projects share a common base image:

```
50 5 * * * root gitlab-docker-autodep 
                --url=https://my.gitlab
                --token=$(< /etc/gitlab_docker_token)
                rebuild $MY_BASE_IMAGE
```