↓ Skip to main content
  1. Posts/

Publishing your Python package in your Gitlab Package Registry

Table of Contents

I joined iib in January and since the beginning it was clear that I’d have a lot of fun automating a few manual process. The development team is formed by three persons: the CTO, another developer and me. The team is small but step by step we’re going towards the technical excellence we want. :)

One of the things that needed automation was the release of an internal library. The process to install it was basically git pull + python setup.py install. Since we have many other things to do, this problem was waiting a little longer to be solved. But the things became ugly when I had to setup a test pipeline for Airflow (must say that now I understand why the most famous blog post on this topic is called Data’s Inferno: 7 Circles of Data Testing Hell with Airflow - but this discussion I’ll leave for another day).

At iib we use a self-hosted Gitlab instance, recently updated to the version 13. Gitlab offers a private Package Registry out of the box. The package registry was there, we had our project using poetry and a test pipeline. Publishing the package would be a piece of cake, right?

Where is this Package Registry after all?
#

I’m going to start by the most obvious thing that I took a few minutes to figure: you have to enable the Package Registry in your project in order to use it. Go to General > Visibility, project features, permissions > Repository > Packages.

Forbidden
#

As you can imagine, you’ll need the right permissions to upload the package to your own pypi. This can be done using CI_JOB_TOKEN, an variable provided by Gitlab during a pipeline run. Below you can see how we did it:

release:
  stage: deploy
  image: python:3.8
  script:
    - pip install poetry twine
    - poetry version ${BUMP_RULE}  # missing a step to commit this change!
    - poetry build
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --verbose --repository-url https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi dist/*
  only:
    - tags

The username for the token ``CI_JOB_TOKENisgitlab-ci-token. Another important variable provided during the pipeline execution is CI_PROJECT_ID` (you can see the number in the repository page too).

Automation FTW
#

The thing that got me really excited was the automation flow for our releases. With the job shown previously, we only needed to create a new tag and then a new version would be released. I added BUMP_RULE as an environment variable because we can control the rule used by just changing the repository environment variable. Poetry provides different options (following the SEMVER style).

Poetry publish #fail
#

Note that we used Twine instead of Poetry. Unfortunately it wasn’t possible to use poetry publish due to a different errors:

poetry publish --repository gitlab -u gitlab-ci-token -p $env.CI_JOB_TOKEN
  • RuntimeError Repository gitlab is not defined (when having it configured in pyproject.toml)
  • UploadError HTTP Error 404: Not Found (when configuring the repository via CLI)

There are open issues for it, so I hope it will be possible in the future. By now, Twine is doing the trick just fine.

pip install
#

Since the package is published in the private package registry, we’ll need to adapt the command a little bit.

A trick here is: you’ll need the index URL and the extra index URL for the dependencies that are coming from Pypi. In the end the command will look like this:

pip install \
    --index-url https://$PERSONAL_ACCESS_TOKEN_NAME:$PERSONAL_ACCESS_TOKEN@gitlab.com/api/v4/projects/44/packages/pypi/simple \
    --extra-index-url https://pypi.org/simple \
    mypackage

Without the --extra-index-url you’ll get errors like ERROR: Could not find a version that satisfies the requirement.

Unfortunately, Gitlab’s documentation is missing some details but you can find their step by step here.

Hope it helps! See ya next time.