Skip to content
Snippets Groups Projects
Select Git revision
  • dc20869668759c0d6c50257313869adb0c5d09d4
  • master default protected
2 results

gitlab-deps

Forked from pipelines / tools / gitlab-deps
58 commits behind the upstream repository.
ale's avatar
Serve webhook on /
ale authored 
Get rid of the useless / handler and move /webhook in its place.
dc208696
History
ale's avatar dc208696
History
Name Last commit Last update
gitlab_docker_autodep
.gitignore
LICENSE
README.md
setup.py
README.md

gitlab-docker-autodep

Automatically rebuild all the downstream dependencies of Docker-based projects on a Gitlab instance.

It scans the master branch of all repositories containing a Dockerfile looking for FROM lines and navigates the resulting dependency tree to find all projects that needs to be rebuilt when their base image (or an upstream image thereof) changes.

By default, since it is meant to be used as a trigger as the last step in a CI script, it will not navigate the dependency tree recursively but only look at the first-level dependencies: this way, further CI pipelines downstream will each trigger their own deps once the image has been rebuilt.

Installation

Install the tool either in a virtualenv of or system-wide with any of the standard Python installation mechanisms, for instance (using setuptools):

sudo python setup.py install

This will install the gitlab-docker-autodep command-line tool in /usr/local/bin. The tool should work with either Python 2 and Python 3, and it has few dependencies (just the Gitlab API, and Flask).

Usage

There are two modes of using this software: either as a one-shot command-line tool, or as a standalone HTTP server responding to Gitlab Webhook requests, to integrate Docker dependencies with Gitlab CI.

In both cases, the program is configured via command-line options.

Common options

The tool must be pointed at your Gitlab instance with the --url command-line option,

You can pass an authentication token using the --token command-line option. This is usually required in order to trigger CI pipelines: the access token must have the api scope.

The tool will only examine Docker images hosted on the Docker registry associated with the Gitlab instance. By default the registry name is automatically derived from the server URL (adding a registry prefix), but it can be changed with the --registry command-line option.

Scope

On larger Gitlab instances, parsing Dockerfiles for all projects can be an expensive (long) operation. The program offers two options to manage the scope of the dependency analysis: --match and --filter.

The former, --match, allows for filtering the project list on the server side, using a Gitlab search query. The latter, --filter, applies a regular expression to the project names (including namespaces) before parsing their dependencies. Combining the two, for example, it is possible to efficiently limit the scope of the tool to a specific namespace:

gitlab-docker-autodep ... --match myns --filter ^myns/ ...

Note that, when building the dependency tree:

  • tags in FROM lines are ignored
  • only the master branch of repositories is scanned for Dockerfiles

This might lead to more rebuilds than strictly necessary.

Command-line

The rebuild command will trigger a rebuild of all the dependencies of a given Docker image, possibly waiting for the CI pipelines to complete. Pass the qualified Docker image name (as it appears on FROM lines in Dockerfiles) as a command-line argument.

The tool will print out the project names of the dependencies it found. The --recurse option will traverse the dependency tree recursively, waiting for CI pipelines to complete so that they are built in the right order.

It is possible to limit the scope of the initial dependency scan (which is an expensive operation) to projects matching a Gitlab search keyword using the --match option.

Gitlab CI integration

In order to automatically rebuild the dependencies of a Docker image built using Gitlab CI, it is possible to run gitlab-docker-autodep as a webhook: this way, whenever a successful CI pipeline completes, you can trigger a rebuild of the dependencies.

To do this, use the server command of gitlab-docker-autodep, and specify the address to bind to using the --host and --port options. It is also possible to enforce authentication of the webhook with a secret token (X-Gitlab-Token) using the --webhook-auth-token option.

When running in this mode, it is assumed that all your Docker-related projects have webhooks set up to rebuild their dependencies, so gitlab-docker-autodep will only trigger a build of the immediate dependencies of a project.

Also note that the server does not have any TLS support: if necessary, it is best to use a dedicated reverse proxy (Apache, NGINX, etc).

For example, assuming the webhook server will be running on the same machine as Gitlab itself, and that the Gitlab authentication token is stored in /etc/gitlab_docker_token:

gitlab-docker-autodep \
    --url=https://my.gitlab \
    --token=$(< /etc/gitlab_docker_token) \
    server \
    --host=127.0.0.1 --port=14001

You can then configure your project's webhooks with the URL http://localhost:14001/, with the Trigger checkbox set only on Pipeline events.

It can be useful to run the rebuild command from a cron job, for instance in order to rebuild images on a periodic schedule, and assuming all your projects share a common base image:

50 5 * * * root gitlab-docker-autodep 
                --url=https://my.gitlab
                --token=$(< /etc/gitlab_docker_token)
                rebuild $MY_BASE_IMAGE