Version Control

CI Cache Design: Repository Size, Shallow Clones and Cache Keys

When CI stalls at the clone step, repository size and cache policy are usually to blame. Shallow clone, partial clone, filter flags, cache key design, and separating artifacts from source.

By LaoHand Team·9 min read·Updated 2026-09-30

Measure the Size, Then Choose a Strategy

The first step when CI slows is quantification. Use git count-objects -vH for object store size, git rev-list --count HEAD for commit count, and du -sh .git for disk usage. Past roughly 1GB or tens of thousands of commits, the cost stops being network transfer and becomes pack inflation plus index building.

Locate the bloat with git verify-pack -v .git/objects/pack/*.idx | sort -k3 -n -r | head -20, then map objects to paths with git rev-list --objects --all. Almost every "the repo is too big" case comes down to a handful of causes: accidentally committed binaries (video, model weights, build output), long-lived unpruned branches, or vendored dependencies.

Remember history only grows: even if the current tree is 1MB, a 2GB tar committed two years ago still has to be downloaded during clone. That is why cleanup requires git filter-repo history rewriting plus a force push, not merely git rm.

# Quantify repository size
git count-objects -vH
git rev-list --count HEAD
du -sh .git

# Find the largest historical objects and their paths
git verify-pack -v .git/objects/pack/*.idx | sort -k3 -n -r | head -20
git rev-list --objects --all | git cat-file --batch-check="%(objecttype) %(objectname) %(objectsize) %(rest)" \
  | awk "$1==\"blob\" {print $3, $4}" | sort -n -r | head -20

# Purge large files from history (rewrites all commit shas, requires a force push)
git filter-repo --path build/ --path "*.zip" --invert-paths
git push --force --mirror origin

The Real Boundaries of Shallow and Partial Clone

--depth 1 fetches only the tip commit, turning a multi-minute clone into seconds — at the cost of breaking history operations. git log shows nothing meaningful, git blame reports missing commits, and git describe --tags finds no tags. Recover on demand with git fetch --deepen=100, or git fetch --unshallow when you truly need everything.

--filter=blob:none is a partial clone: it fetches commits and trees but pulls blob contents lazily from the remote. It shines on repositories whose history holds large binaries, because checking out the current branch transparently fetches just the blobs it needs. The cost is one network round-trip per lazy fetch, which can be slower on poor connections.

The two combine, subject to server support: --filter requires uploadpack.allowFilter on the remote, and --depth with partial clone has compatibility gaps on older servers. A safe combination is --depth 50 --filter=blob:none --no-tags, which keeps blame functional while avoiding a full historical blob transfer.

# Everyday CI clone: sufficient and blame still works
git clone --depth 50 --filter=blob:none --no-tags --branch "$CI_BRANCH" \
  "https://github.com/org/repo.git" .

# Deepen on demand when full history or tags are needed
git fetch --deepen=200
git fetch --tags

# Explicitly drop history (build-only scenarios)
git clone --depth 1 --single-branch --no-tags "https://github.com/org/repo.git" .

# Use sparse-checkout to fetch only the needed directories when history is large
git sparse-checkout init --cone
git sparse-checkout set packages/api packages/shared

Cache Key Design: Hit Rate Versus Correctness

A CI cache key trades hit rate against correctness. A common mistake is putting the branch name in the key, which gives every branch and every PR its own cache and wrecks the hit rate. A more workable design is layered: a primary key of lockfile hash plus runtime version plus architecture, with the branch name as a prefix and a prefix fallback lookup.

Fallback lookups via restore-keys drive the hit rate. Restoring a slightly older cache under a prefix like node_modules- and then running npm ci to repair it is far faster than a cold install. Over-precise keys — the full lockfile hash — mean a single dependency tweak sends the whole pipeline back to cold.

Another trap to avoid is caching node_modules without reinstalling on restore. Native modules tied to a platform or runtime version get silently corrupted across versions. The reliable pattern is caching the package manager download cache (npm _cacache, the pnpm store) and re-running installation every time.

# GitHub Actions: exact primary key + prefix fallback
- name: Cache pnpm store
  uses: actions/cache@v4
  with:
    path: ~/.pnpm-store
    key: pnpm-${{ runner.os }}-${{ runner.arch }}-node${{ hashFiles("pnpm-lock.yaml") }}
    restore-keys: |
      pnpm-${{ runner.os }}-${{ runner.arch }}-node

# GitLab CI: staged caching, also avoid an over-narrow key
stages: [deps, test]

cache:
  key:
    files:
      - pnpm-lock.yaml
    prefix: "pnpm-$CI_OS"
  paths:
    - .pnpm-store/
    - node_modules/

deps:
  script:
    - corepack enable
    - pnpm install --frozen-lockfile

test:
  script:
    - pnpm test

Separate Artifacts from Source

Mixing build output with source is where cache design starts to rot. The correct structure is: the source repository holds source only, CI produces immutable artifacts (image tarballs, tar.gz, jar, wheel), those go to an artifact registry (GHCR, Nexus, S3), and deployment pulls by immutable tag without ever checking out source.

This separation buys three things directly: rollback to any historical tag because artifacts are immutable; the build runs once per source change while deployment becomes a pure pull; and production needs no compiler toolchain, allowing small images with a minimal attack surface.

Enforce one rule: deployment may only consume signed artifacts produced by CI, and build commands are forbidden in the deploy stage. The moment deployment builds locally, artifacts stop being reproducible and both the caching and rollback benefits evaporate.

# CI: build and push an immutable artifact (the tag contains the git sha, immutable by nature)
docker build -t ghcr.io/org/app:$GIT_SHA -t ghcr.io/org/app:latest .
docker push ghcr.io/org/app:$GIT_SHA

echo "artifact=ghcr.io/org/app:$GIT_SHA" >> $GITHUB_OUTPUT

# Deploy: pull the artifact by tag only, no build, no source checkout
git checkout $TARGET_SHA
git pull --ff-only
docker pull ghcr.io/org/app:$ARTIFACT_TAG
docker tag ghcr.io/org/app:$ARTIFACT_TAG ghcr.io/org/app:running
docker compose up -d --no-build

# Rollback: just pull with an older sha tag, no rebuild needed
# ARTIFACT_TAG=<previous-sha> ./deploy.sh

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.