HCA Validation Tools

A multi-service collection of tools for validating Human Cell Atlas (HCA) ingest data, built with LinkML schemas and deployed as AWS Lambda functions.

Overview

This repository contains validation tools organized in a multi-service architecture:

• Entry Sheet Validator: Validates Google Sheets against LinkML schemas with AWS Lambda deployment
• Data Dictionary Generator: Creates data dictionaries from LinkML schema definitions
• Schema Validation: Validates LinkML schemas and generates Python models

Repository Layout

Top-level directories and their roles (each package/service has its own pyproject.toml, uv environment, and tests/):

• shared/ — core validation library (Pydantic validation, entry-sheet logic, data dictionary build) that the services depend on via a uv path dependency
• packages/ — publishable PyPI packages: hca-metadata-schema (the LinkML schema and its generated Pydantic models, which shared and hca-anndata-tools depend on), hca-schema-validator, hca-anndata-tools, hca-anndata-mcp; and the private (unpublished) hca-tracker-client / hca-tracker-mcp, installed from git or a local path
• services/ — deployable services: entry-sheet-validator (Lambda), dataset-validator (Batch), and the cellxgene-validator / hca-schema-validator wrappers
• deployment/ — Dockerfiles and per-service deployment configs
• data_dictionaries/ — generated data dictionaries

Quick Start

This project uses uv for dependency management and Make for build automation.

Install uv first (it also provisions the required Python version, so you don't need to install Python yourself) — see the uv install docs:

curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/clevercanary/hca-validation-tools.git
cd hca-validation-tools

# Build everything (schemas, data dictionaries, Docker images, run tests)
# Note: build-all builds the Lambda container, so it needs Docker running and an
# AWS profile with access to the Lambda Extension layer (LAMBDA_PROFILE in .env.make).
make build-all

# Or just install the shared library (no Docker/AWS needed)
cd shared
uv sync

Available Commands

Build Commands

make build-all                 # Build shared lib + containers and run tests (needs Docker + AWS)
make build-lambda-container   # Build the entry-sheet Lambda image (AWS profile: LAMBDA_PROFILE in .env.make)

Schema and data-dictionary generation live in the shared library's Makefile:

cd shared
make gen-schema                # Generate Pydantic models from LinkML schemas
make validate-schema           # Validate all LinkML schema files
make generate-data-dictionary  # Generate the core data dictionary
make build                     # All of the above in one step

Test Commands

Each package and service has its own uv environment, so tests are run per project. CI's test matrix in .github/workflows/ci.yml is the list of suites it runs on every pull request: run each as uv run pytest tests/ <pytest-args> in the entry's project directory, with the entry's pytest-args, after installing what its apt names (the tracker packages' download tests need aria2c) and on its python-version where it sets one. Type checking runs separately:

make typecheck   # pyright across every typed project (one pass per venv)

Integration tests (which hit live Google Sheets and need credentials in .env) are marked integration. CI's matrix excludes them with -m "not integration"; a bare pytest tests/ would run them. Run only the integration tests with:

cd shared && uv run pytest tests/ -m integration

make test-all runs the shared and service suites in one go, but it is not what CI runs: it invokes the entry-sheet container smoke test, so it requires Docker and a built image (and fails without them); it runs shared's integration tests, which need credentials; it covers only packages/hca-metadata-schema of the packages/ suites; and it does not check coverage floors.

The entry-sheet-validator container smoke test is not in CI's matrix; it boots the built Lambda image in Docker (so it needs a built image), and its happy-path assertion additionally needs GOOGLE_SERVICE_ACCOUNT in the environment — without it that assertion skips. See deployment/entry-sheet-validator/README.md.

Checks (mirror CI)

Before pushing, run the same lint, format, and type checks CI gates on. Ruff is pinned to match CI exactly:

uvx ruff@0.11.8 check .          # lint
uvx ruff@0.11.8 format --check . # formatting
make typecheck                   # pyright

Optionally install the pre-commit hook so equivalent checks run automatically on git commit — it applies ruff --fix and formatting and runs make typecheck (the auto-fixing counterparts of the read-only checks above). It is opt-in and does nothing until installed:

pip install pre-commit && pre-commit install

Deployment Commands

The environment is selected with ENV=dev (default) or ENV=prod:

make deploy-lambda-container ENV=dev    # Deploy the entry-sheet Lambda (dev)
make deploy-lambda-container ENV=prod   # Deploy the entry-sheet Lambda (prod)
make invoke-lambda SHEET_ID=<id>        # Invoke the deployed Lambda

# Dataset-validator Batch service
make batch-publish-container ENV=dev    # Build, push to ECR, register the job def
make batch-submit-job ENV=dev           # Submit a Batch validation job

See CLAUDE.md for the full deployment and release workflow.

Configuration

Environment Files

Both are gitignored — keep credentials and machine-specific values out of the repo.

• .env - Contains GOOGLE_SERVICE_ACCOUNT credentials for Google Sheets API
• .env.make - AWS deployment variables (account IDs, regions, roles) and the AWS CLI profiles: one to build the Lambda image, and one per environment to deploy and invoke it (DEV_LAMBDA_DEPLOY_PROFILE, PROD_LAMBDA_DEPLOY_PROFILE). The deploy, invoke and Batch targets take their profile only from this file. Each one stops when PROFILE, its own profile variable (LAMBDA_DEPLOY_PROFILE or BATCH_PROFILE) or that variable's DEV_/PROD_ form for the current ENV is set on the command line, or in the shell environment and not in this file. AWS_PROFILE is ignored.

One-time AWS setup — copy the committed template and fill in your values:

cp .env.make.example .env.make
# then edit .env.make: set LAMBDA_PROFILE to an AWS CLI profile that can fetch
# the Lambda Extension layer, and fill in the remaining vars

With .env.make in place, make build-lambda-container and make build-all pick up the profile automatically — no PROFILE= argument needed (pass PROFILE=<name> only to override).

Google Sheets Integration

To run integration tests or use the validator with private sheets:

1. Create a Google Service Account with Sheets API access
2. Download the service account JSON key
3. Add to .env file:

GOOGLE_SERVICE_ACCOUNT='{"type": "service_account", "project_id": "your-project", ...}'

Development

Multi-Service Architecture

• shared/ - Core validation library used by all services
• services/entry-sheet-validator/ - AWS Lambda service for sheet validation
• Each service has its own uv environment and test suite

Adding New Services

1. Create new directory under services/
2. Add pyproject.toml with dependency on shared library
3. Update Makefile with service-specific commands

Virtual Environment Management

This project uses uv, which creates a project-local .venv/ directory in each package/service:

• Shared library: shared/.venv/
• Services: Each service gets its own .venv/ alongside its pyproject.toml
• Reproducibility: uv resolves dependencies into a uv.lock; uv sync --frozen installs exactly those versions without re-resolving (a plain uv sync may update the lock). The services commit their uv.lock, so --frozen works there; the library projects (shared/, packages/*) gitignore their lock (see #483), so a fresh clone re-resolves from pyproject.toml
• VS Code: Open hca-validation-tools.code-workspace so each folder resolves its own .venv, or select the interpreter via Ctrl+Shift+P → "Python: Select Interpreter"

License

See the LICENSE file for details.