Argo+Rust lab

Some common Argo NetCDF data manipulation operations implemented in Rust, with a Python interface provided by PyO3.

The goal is to explore how much faster and simpler Argo file handling (QC filtering, DATA_MODE selection, ...) can be when the heavy lifting is done in Rust, while keeping a NumPy-friendly API.

Setup

You need:

• Rust via rustup. On Apple Silicon, make sure the toolchain is native (rustc -vV | grep host should print aarch64-apple-darwin).
• A C compiler and cmake (xcode-select --install and brew install cmake on macOS, sudo apt install build-essential cmake on Debian/Ubuntu).
• uv to manage the Python environment.

HDF5 and libnetcdf do not need to be installed on your system: the static feature of the netcdf and hdf5-metno crates builds them from source. The first build therefore takes several minutes, but later builds reuse the cached result in target/.

Avoid cargo clean unless something is really broken, as it throws that cache away.

Develop

uv sync                 # installs dependencies and builds the native module
uv run maturin develop  # fast rebuild loop after editing src/lib.rs
uv run pytest

uv sync rebuilds automatically when pyproject.toml, Cargo.toml or src/**/*.rs change (see tool.uv.cache-keys). To force a rebuild:

uv sync --reinstall-package argo-rust-lab

For quick feedback on Rust compile errors, cargo check is much faster than a full build.

Layout

Path	Content
src/lib.rs	Rust implementation (compiled to argo_rust_lab.native)
python/argo_rust_lab/__init__.py	Public Python API (re-exports from native)
Cargo.toml	Rust dependencies
pyproject.toml	Python packaging, built with maturin

Usage

import argo_rust_lab as lab

ncfile = "..."

lab.dims(ncfile)                # {dimension name: length}
lab.n_prof_levels(ncfile)       # (N_PROF, N_LEVELS)
lab.read_var(ncfile, "PRES")    # float64 array, shape=(N_PROF, N_LEVELS)

Filter on QC flags

filter_var reads a float variable and its QC variable, and returns an array with the same shape and dtype as the variable, with NaN where the QC flag is not accepted.

lab.filter_var(ncfile, "PRES", keep="1")                       # good only
lab.filter_var(ncfile, "PRES", keep="12")                      # good + probably good
lab.filter_var(ncfile, "PRES", qc_name="PRES_QC", keep="12 ")  # also keep blank (not QC'd) flags
# each returns: array(..., shape=(N_PROF, N_LEVELS), dtype=float32)

• keep is a string of the QC flags to accept ("1" by default). A blank flag is rejected unless the string contains a space.
• qc_name defaults to <PARAM>_QC.

Select PARAM or PARAM_ADJUSTED according to DATA_MODE

select_mode reads DATA_MODE for each profile and assembles one (N_PROF, N_LEVELS) array per parameter:

DATA_MODE	Variable used
R	<PARAM>
A or D	<PARAM_ADJUSTED>
blank / other	NaN for the whole profile

It returns a dict {<PARAM>: array} with the original shape and dtype. By default, params is ["PRES", "TEMP", "PSAL"].

core = lab.select_mode(ncfile)
core = lab.select_mode(ncfile, params=["PRES", "DOXY"])

# If `keep` is given (e.g. "12"), the matching QC variable is selected the same way
# (<PARAM>_QC for R, <PARAM>_ADJUSTED_QC for A/D) and rejected values become NaN
core = lab.select_mode(ncfile, keep="12")

Create an xarray dataset from the result

import xarray as xr

xr.merge([xr.DataArray(core[k], name=k, dims=["N_PROF", "N_LEVELS"]) for k in core])

<xarray.Dataset> Size: 12MB
Dimensions:  (N_PROF: 742, N_LEVELS: 1369)
Dimensions without coordinates: N_PROF, N_LEVELS
Data variables:
    PRES     (N_PROF, N_LEVELS) float32 4MB 3.7 4.5 5.5 6.4 ... nan nan nan nan
    TEMP     (N_PROF, N_LEVELS) float32 4MB 25.88 25.89 25.89 ... nan nan nan
    PSAL     (N_PROF, N_LEVELS) float32 4MB nan nan nan nan ... nan nan nan nan

API summary

Function	Returns
dims(path)	dict[str, int] of dimension lengths (key order is not guaranteed)
n_prof_levels(path)	(N_PROF, N_LEVELS)
read_var(path, name)	numeric variable as a float64 array
filter_var(path, name, qc_name=None, keep="1")	array, same shape and dtype, NaN where QC fails
select_mode(path, params=None, keep=None)	dict[str, array], selected by DATA_MODE

Notes and limitations

• Missing data are not converted. Fill values (typically 99999) are returned as is, unless a QC filter removes them. Handle NaN and fill values in Python, for instance with np.ma.masked_invalid.
• No attribute handling. _FillValue, scale_factor and add_offset are not applied.
• filter_var and select_mode return Float variables only (float32 and float64). Other types raise a TypeError.
• read_var always returns float64, whatever the type stored in the file.
• Missing variables raise a KeyError naming the variable (e.g. no DOXY_ADJUSTED in the file).
• The GIL is released during file reading, so calls can run in parallel from Python threads.

Troubleshooting

• uname -m prints x86_64 on an Apple Silicon Mac: your shell is running under Rosetta (often an Intel build of bash or zsh set as login shell). Switch to a native shell, then check that which brew gives /opt/homebrew/bin/brew.
• Linker or CMake errors mentioning a strange path: look for stray LDFLAGS, LIBRARY_PATH or similar variables in your shell startup files (env | grep -i -E "ldflags|library_path|cmake").
• HDF5_DIR or NETCDF_DIR set in your environment: unset them, as they make the build use a system HDF5 instead of the bundled one.
• ModuleNotFoundError: numpy: the Rust numpy crate does not install NumPy itself, so keep numpy in the project dependencies.