Skip to content
AndreaBozzoPublic

About

Data profiling and quality gates for CSV, JSON, Parquet and DataFrames. Rust core, Python API.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

34 stars

Watchers

0 watching

Forks

Latest commit

 

History

1,277 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dataprof logo

dataprof

Know what's in your data before it ships.

PyPI Crates.io docs.rs License: MIT OR Apache-2.0

Website · Getting started · Python API · Release notes

dataprof profiles CSV, JSON, JSONL, Parquet, DataFrames and Arrow tables in one call: types, nulls, distributions, patterns and a quality score. Then it can gate your pipeline on what it found, including changes from a saved baseline. It runs locally and records when the evidence cannot settle a check.

A Rust library with a Python package on top. Local file profiling needs no server, account or network access.

Install

pip install dataprof          # or: uv pip install dataprof

Wheels for CPython 3.11 to 3.15 on Linux, macOS and Windows, with no Python dependencies.

Profile a file

import dataprof as dp

report = dp.profile("orders.csv")        # also .json, .jsonl, .parquet, DataFrames, dicts
print(report.rows, "rows,", report.columns, "columns, quality", report.quality_score)

amount = report["amount"]
print(amount.data_type, amount.mean, amount.null_percentage)

Ask it what deserves attention. Each finding has a stable code and the evidence behind it, never a raw value:

for finding in report.findings():
    print(finding.severity, finding.code, finding.column)
# warning locale_numbers price       ("10,50"-style numbers, left out of the stats)
# warning null_heavy email
# info sensitive_pattern email

Gate a pipeline

State what "good enough" means and get a verdict: pass, fail, or inconclusive when the data can't prove it either way.

result = report.check(
    min_quality_score=90, max_null_count={"customer_id": 0}, max_null_percentage={"*": 20}
)
if not result.passed:
    for check in result.violations:
        print(check.code, check.column, check.message)

Or from CI, with no code at all:

python -m dataprof.check orders.csv --min-quality 90 --max-null "*=20"
# exit 0 = pass, 1 = fail, 2 = inconclusive, 3 = an argument, policy, input, baseline or output error

# Or against the last accepted run: fail if quality drops more than 5 points.
# A rolling clock lets each run judge its dates against its own day.
python -m dataprof.check orders.csv --baseline accepted.json --max-quality-drop 5 --timeliness-clock rolling

Keep the candidate report with --report candidate.json. Promote it to the baseline only on a pass, including the first run, which uses absolute rules alone. A changed or unrecorded metric definition can leave a comparison definition_not_comparable; inspect the reason and regenerate the baseline deliberately. See the checkpoint recipe.

0.14 adds these checks:

  • Freshness and missing periods: check(max_date_age={"published_at": 3}, max_date_gap={"published_at": 1}) limits the age of the newest date and the largest gap between consecutive distinct days. dp.profile("orders.csv", reference_date="2026-06-10") fixes the timeliness clock.
  • Volume: check(baseline=baseline, max_row_count_drop=10, max_row_count_growth=20) limits percentage changes in row count; min_rows=1 sets an absolute floor.
  • Keys: check(unique_columns=["order_id"], max_null_count={"order_id": 0}) requires distinct non-null keys and no nulls. From CI, use --unique-column order_id --max-null-count order_id=0. identifier_columns=["order_id"] selects the key for the uniqueness score and key_duplicates finding.
  • Schema drift: forbid_removed_columns=True rejects removed columns. compare() and baseline diagnostics also report inferred type changes; these diagnostics alone do not fail a gate. A column's native_type records its source's declared type where available.

See the cookbook for complete examples.

Hand it to an agent

A token-bounded summary for an LLM. Values that look like personal data are never echoed:

print(report.to_llm_context(max_tokens=500))

Use it from Rust

cargo add dataprof
use dataprof::{Profiler, QualityPolicy, Verdict};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let report = Profiler::new().analyze_file("orders.csv")?;
    let result = QualityPolicy::new().min_quality_score(90.0).evaluate(&report)?;
    assert_eq!(result.verdict, Verdict::Pass);
    Ok(())
}

Minimum supported Rust: 1.96.

What you can count on

  • Same data, same serialized numbers. Rust Serde output and Python to_dict() / to_json() agree across routes and engines for the same logical data, schema semantics, options and analyzed population, within the numeric contract. Source and execution provenance are compared separately. Raw floats may differ in their final digits; extreme values near the limits of f64 belong to an approximate domain. A retained sample and a full scan need not agree. Reports flag sampled numeric statistics with is_approximate, estimated distinct counts with unique_count_is_approximate, and sampled quality components with quality_sampled_dimensions.
  • Bounded memory. Files larger than RAM stream through fixed-size accumulators. The exact duplicate-row count holds up to 1,000,000 distinct rows by default, about 10 to 20 bytes each, inside the memory budget, and becomes an estimate under memory pressure. A duplicate_rows_exact_limit you set yourself is the exception: those rows stay exact and are kept outside the budget.
  • Honest verdicts. A sampled or partial scan passes a claim about the whole file only when what it read already proves it, such as a row minimum it reached; otherwise the check is inconclusive. A sampled score carries an interval in score_bounds whenever it can be bounded.
  • Absence is not zero. A metric that wasn't computed is None, never a plausible default.
  • Local by default. Profiling runs on your machine. Reports carry statistics, including numeric minima, maxima and modes; to_llm_context() leaves raw values out by default.

The Python wheels include file and bytes profiling, Arrow C array and stream inputs, and dataprof.asyncio file, bytes and HTTP helpers. Install the table library you use, plus pyarrow for pandas and polars conversion; native Parquet file and bytes profiling needs none of them. PostgreSQL, MySQL and SQLite require a source build with connector features. See input routes.

Status

Beta, and moving fast toward a stable 1.0 contract. Some things are still narrow, and we'd rather tell you than have you find out: numbers written with a decimal comma are detected and flagged but not yet parsed, and beyond empty, null and nan, only a fixed set of markers (NA, N/A, n/a, #N/A, \N, None, matched exactly, ignoring surrounding whitespace) counts as null, so a sentinel like - or -999 is still a value. The release notes list what changed and what's known.

Found something wrong? Open an issue. A file that profiles badly is the most useful bug report there is.

Learn more

Citing dataprof

Use Cite this repository in the GitHub sidebar, which builds APA and BibTeX from CITATION.cff. The citation is for the software; benchmark material lives in scalcom2026-dataprof.

License

Either the MIT License or the Apache License, Version 2.0, at your option.

About

Data profiling and quality gates for CSV, JSON, Parquet and DataFrames. Rust core, Python API.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

34 stars

Watchers

0 watching

Forks

Releases

Used by

Contributors

Languages