gitlab ci cache and key
I had some hard time configure cache in gitlab ci.
Still need to check it deeply, but it seems what define a cache is the key, not the path. Hence having the same key in a pipeline with different paths might do things not expected.
It seems that when a cache it pushed, with a given path, previous cache push for the same key with different paths are erased.
I ended up with the following: a cache configuration snippet to ensure one key = one set of paths, then recall this snippet with yaml anchors. Here is an simplified excerpt.
# ...
stages:
- warmup
- build
variables:
# ...
SHARED_CACHE_PATH: "${CI_PROJECT_DIR}/.shared_buildcache"
BRANCH_CACHE_PATH: "${CI_PROJECT_DIR}/.branch_buildcache"
EXTERNAL_BUILD_DIR: "${SHARED_CACHE_PATH}/builds/external"
EXTERNAL_INSTALL_DIR: "${SHARED_CACHE_PATH}/installs/external"
PIP_VENV_CACHE_DIR: "${CI_PROJECT_DIR}/.venv"
# ...
.cache-configuration:
python: &cache-python
key: "python3"
paths:
- ${PIP_VENV_CACHE_DIR}
policy: pull-push
build: &cache-build
key: ${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}-${BUILD_TYPE}
paths:
- "${BUILD_DIR}/${BUILD_TYPE}"
policy: pull-push
external: &cache-external
key:
files:
- external/CMakeLists.txt
paths:
- "${EXTERNAL_BUILD_DIR}"
- "${EXTERNAL_INSTALL_DIR}"
policy: pull-push
# ...
.before-script-python:
before_script:
- >
apt-get --no-install-recommends -y
install python3 python3-pip python3-venv
- python3 -m venv ${PIP_VENV_CACHE_DIR}
- source ${PIP_VENV_CACHE_DIR}/bin/activate
- pip3 install -r requirements.txt
warmup-python:
stage: warmup
script:
- !reference [.before-script-python, before_script]
cache:
- <<: *cache-python
build-externals:
stage: build
before_script:
- !reference [.before-script-python, before_script]
needs:
- job: warmup-python
script:
- python3 build-external.py
cache:
- <<: *cache-external
- <<: *cache-python
policy: pull
artifacts:
paths:
- "${EXTERNAL_INSTALL_DIR}"
.build-type-matrix:
matrix:
- BUILD_TYPE: ["Release", "Debug"]
build-linux:
stage: build
before_script:
- !reference [.before-script-python, before_script]
parallel: !reference [ .build-type-matrix ]
needs:
- job: build-externals
artifacts: true
- job: warmup-python
variables:
BUILD_DIR: "${BRANCH_CACHE_PATH}/builds/${PROJECT_NAME}"
script:
- python3 build.py
cache:
- <<: *cache-build
- <<: *cache-python
policy: pull
Leave a Reply