Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result

gitlab-deps

  • Clone with SSH
  • Clone with HTTPS
    • ale's avatar
    Implement a server mode for webhooks
    ale authored 
    Support better integration with Gitlab CI, for automatic dependency
rebuilds via Gitlab Webhooks (minimal configuration required).

Update the documentation accordingly.
    d5a3d46e
    History
    ale's avatar d5a3d46e
    History
    Name Last commit Last update
    gitlab_docker_autodep
    .gitignore
    LICENSE
    README.md
    setup.py
    README.md

    gitlab-docker-autodep

    Automatically rebuild all the downstream dependencies of Docker-based projects on a Gitlab instance.

    It scans the master branch of all repositories containing a Dockerfile looking for FROM lines and navigates the resulting dependency tree to find all projects that needs to be rebuilt when their base image (or an upstream image thereof) changes.

    By default, since it is meant to be used as a trigger as the last step in a CI script, it will not navigate the dependency tree recursively but only look at the first-level dependencies: this way, further CI pipelines downstream will each trigger their own deps once the image has been rebuilt.

    Installation

    Install the tool either in a virtualenv of or system-wide with any of the standard Python installation mechanisms, for instance (using setuptools):

    sudo python setup.py install

    This will install the gitlab-docker-autodep command-line tool in /usr/local/bin. The tool should work with either Python 2 and Python 3, and it has few dependencies (just the Gitlab API, and Flask).

    Usage

    There are two modes of using this software: either as a one-shot command-line tool, or as a standalone HTTP server responding to Gitlab Webhook requests, to integrate Docker dependencies with Gitlab CI.

    In both cases, the program is configured via command-line options.

    Common options

    The tool must be pointed at your Gitlab instance with the --url command-line option,

    You can pass an authentication token using the --token command-line option. This is usually required in order to trigger CI pipelines: the token must have sufficient permissions to do so.

    The tool will only examine Docker images hosted on the Docker registry associated with the Gitlab instance. By default the registry name is automatically derived from the server URL (adding a registry prefix), but it can be changed with the --registry command-line option.

    Command-line

    The rebuild command will trigger a rebuild of all the dependencies of a given Docker image, possibly waiting for the CI pipelines to complete. Pass the qualified Docker image name (as it appears on FROM lines in Dockerfiles) as a command-line argument.

    The tool will print out the project names of the dependencies it found. The --recurse option will traverse the dependency tree recursively, waiting for CI pipelines to complete so that they are built in the right order.

    It is possible to limit the scope of the initial dependency scan (which is an expensive operation) to projects matching a Gitlab search keyword using the --match option.

    Gitlab CI integration

    In order to automatically rebuild the dependencies of a Docker image built using Gitlab CI, it is possible to run gitlab-docker-autodep as a webhook: this way, whenever a successful CI pipeline completes, you can trigger a rebuild of the dependencies.

    To do this, use the server command of gitlab-docker-autodep, and specify the address to bind to using the --host and --port options. It is also possible to enforce authentication of the webhook with a secret token (X-Gitlab-Token) using the --webhook-auth-token option.

    When running in this mode, it is assumed that all your Docker-related projects have webhooks set up to rebuild their dependencies, so gitlab-docker-autodep will only trigger a build of the immediate dependencies of a project.

    Also note that the server does not have any TLS support: if necessary, it is best to use a dedicated reverse proxy (Apache, NGINX, etc).

    For example, assuming the webhook server will be running on the same machine as Gitlab itself, and that the Gitlab authentication token is stored in /etc/gitlab_docker_token:

    gitlab-docker-autodep \
    --url=https://my.gitlab \
    --token=$(< /etc/gitlab_docker_token) \
    server \
    --host=127.0.0.1 --port=14001

    You can then configure your project's webhooks with the URL http://localhost:14001/webhook, with the Trigger checkbox set only on Pipeline events.

    It can be useful to run the rebuild command from a cron job, for instance in order to rebuild images on a periodic schedule, and assuming all your projects share a common base image:

    50 5 * * * root gitlab-docker-autodep 
                --url=https://my.gitlab
                --token=$(< /etc/gitlab_docker_token)
                rebuild $MY_BASE_IMAGE