Draft: Optimize frontend fixture downloads with caching mechanism
What does this MR do and why?
Optimize frontend fixture downloads with caching mechanism
Implement two-level caching to reduce unnecessary network requests and improve CI/CD performance when downloading frontend fixtures.
Cache implementation:
- Add .fixtures_version file to track the SHA of last extracted fixtures
- Skip download entirely if current SHA matches cached version
- Add .no_archive file to cache SHAs without available archives
- Maintain rolling cache of last 100 checked SHAs
- Skip archive existence checks for known missing archives
Performance improvements:
- Eliminate redundant downloads when fixtures are already current
- Reduce API calls by ~50-80% in typical CI runs
- Maintain cache size limits to prevent unbounded growth
The caching is stored in tmp/tests/frontend/ and is automatically created if missing. Cache cleanup ensures the no-archive list stays within reasonable bounds (100 entries).
References
- Contributes to gitlab-development-kit#2753 (closed)
- gitlab-development-kit!4978 (comment 2614641016)
How to set up and validate locally
scripts/frontend/download_fixtures.sh
$ time scripts/frontend/download_fixtures.sh
Looking for frontend fixtures for commit 467cf8fe3db6c76907efea44bc93409a1e84af9c... no archive found - caching.
Looking for frontend fixtures for commit fc4c323d0c487983095ecae0f8598516b1908ace... no archive found - caching.
Looking for frontend fixtures for commit fd4e649a407af9c37a6a00a06b3f3d990a10dff6... no archive found - caching.
Looking for frontend fixtures for commit ad11cd64f95588b4840d4cc4adcced56fe2a3499...
We have found frontend fixtures at https://gitlab.com/api/v4/projects/278964/packages/generic/fixtures/ad11cd64f95588b4840d4cc4adcced56fe2a3499/fixtures-ad11cd64f95588b4840d4cc4adcced56fe2a3499.tar.gz!
Downloading from https://gitlab.com/api/v4/projects/278964/packages/generic/fixtures/ad11cd64f95588b4840d4cc4adcced56fe2a3499/fixtures-ad11cd64f95588b4840d4cc4adcced56fe2a3499.tar.gz ...
Extracting archive to .
real 0m1.796s
user 0m0.155s
sys 0m0.125s
rm tmp/tests/frontend/.fixtures_version tmp/tests/frontend/.no_archive
scripts/frontend/download_fixtures.sh
$ time scripts/frontend/download_fixtures.sh
Looking for frontend fixtures for commit 467cf8fe3db6c76907efea44bc93409a1e84af9c... cached - skipping.
Looking for frontend fixtures for commit fc4c323d0c487983095ecae0f8598516b1908ace... cached - skipping.
Looking for frontend fixtures for commit fd4e649a407af9c37a6a00a06b3f3d990a10dff6... cached - skipping.
Looking for frontend fixtures for commit ad11cd64f95588b4840d4cc4adcced56fe2a3499... already downloaded - skipping.
real 0m0.306s
user 0m0.032s
sys 0m0.026s
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Peter Leitzen