Storing evaluation data for AI projects

When building ML systems, test and evaluation datasets are as critical as the code itself. But where should you store them? The answer depends on size, sensitivity, and how often they change.

This post explores three practical approaches to managing evaluation data, from simple Git commits to cloud storage backends. All examples are drawn from real ML project setups.

The baseline: Plain files in Git

The simplest option is to commit test data directly into your Git repository, typically in a tests/data/ directory. I would consider this an anti-pattern in many cases, but one that is used often enough.

my-project/
├── src/
├── tests/
│   └── data/
│       ├── sample_001.pdf
│       ├── sample_002.pdf
│       └── labels.csv
└── pyproject.toml

This works fine for minimal test fixtures, but breaks down quickly as datasets grow. It’s also an absolute no-go if there is personal or restricted data involved (which has been the case in most of my projects so far). But it’s fine and common practice to use this approach for simple artificial or anonymized datasets that are needed, for example, for unit testing.

Pros

• Dead simple: git clone gives you everything
• Versioning comes for free (data evolves with code)
• Great PR workflow (diff shows data changes)

Cons

• Git performs badly with large binaries and frequent rewrites
• Easy to accidentally commit sensitive data
• Makes the entire repo unshareable if data is confidential
• Testing different code versions with the same dataset becomes awkward

Option 1: Separate test-data repository as Git submodule

This is a variation of the plain files in Git approach, but with a separate repository for the test data. The advantage is that the main repository stays clean and fast, while the test data can be versioned and kept private.

The setup is very simple. You need a separate repository and just add it as a submodule to your main repository:

❯ git submodule add https://github.com/youraccount/testdata.git test/data

Cloning into '/foo/bar/test/data'...
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 33 (delta 0), reused 21 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (33/33), 35.21 KiB | 7.04 MiB/s, done.

❯ git add .gitmodules test/data
❯ git commit -m "Add test data submodule"

[main 8078da5] Add test data submodule
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 test/data

❯ git push

Other users can clone the repository and initialize the submodule with one additional command:

# Main repo
git clone https://github.com/myorg/ml-project
cd ml-project
git submodule update --init  # Pulls test-data repo

The parent repo pins the submodule to a specific commit. Optionally, you can set a default branch in .gitmodules to make remote updates more convenient:

# .gitmodules
[submodule "testdata"]
    path = testdata
    url = https://github.com/myorg/test-data-private.git
    branch = v2.1

When to use this

This is a good option if you have lightweight test data and a Git server that’s secure enough for your data sensitivity. It offers independent versioning of the test data and allows you to share the same dataset between multiple services or use it both alongside the codebase and in external system tests.

Pros

• Keeps main repo fast
• Clear separation of concerns
• Good access control (private data repo, public code repo)

Cons

• Versions must be coordinated (e.g. code only works with data versions from x to y).
• CI and developers need to fetch two repositories and keep them in sync.

Option 2: Including external storage with Git LFS

Git LFS is a plugin for Git which replaces large files with pointers in Git while storing the actual files externally. It is a well-established solution for storing larger files with Git, but it is also a bit more complex to set up and use.

Setup

Because Git LFS is a plugin for Git, every user and CI runner needs to install it. Then, you would first want to add a .gitattributes file which configures which files should be tracked with LFS:

# .gitattributes
testdata/*.pdf filter=lfs diff=lfs merge=lfs -text
testdata/*.parquet filter=lfs diff=lfs merge=lfs -text

After adding .gitattributes and running git lfs install, new matching files will be uploaded to LFS automatically:

git lfs install
git add .gitattributes

# Add files (uploaded to LFS automatically)
git add testdata/
git commit -m "Add evaluation dataset"
git push

Instead of using GitHub/GitLab’s built-in LFS, you can configure custom backends like Artifactory:

# .lfsconfig
[lfs]
url = "https://artifactory.example.com/api/lfs/my-lfs-repo"

# ~/.git-credentials
https://<username>:<token>@artifactory.example.com

This keeps the data out of your main Git history while retaining a Git-like workflow.

When to use this

Git LFS is a solid option if you have to store binaries with code (PDFs, images, model checkpoints) and especially if the dataset is somewhat larger, but still fast enough for fetching it on-the-fly.

Pros

• The Git repository stays usable while storing large binaries
• Data versions are clearly tied to commits/tags
• Native Git workflow

Cons

• Requires LFS support and understanding everywhere (dev machines, CI, collaborators)
• Hosting quotas and bandwidth limits can become expensive
• Fork workflows can get messy depending on platform settings
• The data is not necessarily protected if you use a public Git repository unless you have a private LFS backend

Option 3: Object storage with a fetch script

Because it’s simple, this is the option I recommend most of the time. The idea is to store datasets in a private cloud bucket (S3/GCS/Azure/MinIO) and add a small fetch script wherever the data is needed (repo + CI).

Setup

You only need object storage, e.g. an S3 or GCS bucket. Of course, you should use strong encryption by default. That’s the place where you store your datasets. The repository would simply contain a script as follows to fetch (and/or push) the data. Note that I would recommend just using separate folders for separate versions of the data unless you have a specific reason not to.

#!/bin/bash

TESTDATA_BUCKET="gs://my-test-bucket"
TESTDATA_VERSION="2025-10-01"
LOCAL_DIR="./testdata"

mkdir -p "${LOCAL_DIR}"
gcloud storage cp -r "${TESTDATA_BUCKET}/${TESTDATA_VERSION}/*" "${LOCAL_DIR}/"

In your CI pipeline, you’d do the same. And of course you would have to manage the authorization accordingly. This can be as simple as this for typical S3 storage:

# .github/workflows/test.yml
jobs:
  test:
    steps:
      - uses: actions/checkout@v4
      - name: Fetch test data
        run: ./scripts/fetch-testdata.sh
        env:
          TESTDATA_TOKEN: ${{ secrets.TESTDATA_TOKEN }}
      - name: Run tests
        run: ...

When to use this

This is a good option for all sizes of datasets (well, you wouldn’t use terabytes for testing, would you?). It works well with all file types and clearly separates access to code and data.

Pros

• Good scaling, cheap storage
• Lifecycle rules possible (auto-deletion of old versions)
• Fits well into the CI (download only what you need)
• You can keep data private while code stays public

Cons

• Must implement versioning yourself (folder naming, tags)
• Credential management in CI and development team required
• Reproducibility suffers without clear rules

A word about data versioning

Since this external storage option does not provide versioning out of the box, a different approach is needed. This can be simple folder naming, but you can also use tools like DVC, Delta Lake, or others to add Git-like versioning on top of object storage.

Comparison and decision guide

Summary

The best approach for storing evaluation data depends on three factors:

1. Size - Git works well up to ~10 MB, Git LFS up to ~10 GB, object storage beyond that
2. Sensitivity - Separate storage or already a second repository allows public code with private data
3. Change frequency - Frequent updates favor object storage over Git-based solutions

For most ML projects, I recommend:

• Start with plain Git for initial fixtures
• Decide for either a separate repository or object storage as soon as you add substantial data

The key is picking the simplest solution that meets your (and your colleagues!) needs while staying compliant.