Skip to content

[cli] Add mosaic command-line tool#66

Merged
JingsongLi merged 55 commits into
apache:mainfrom
jianguotian:feat/mosaic-cli
Jun 30, 2026
Merged

[cli] Add mosaic command-line tool#66
JingsongLi merged 55 commits into
apache:mainfrom
jianguotian:feat/mosaic-cli

Conversation

@jianguotian

@jianguotian jianguotian commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Inspecting a Mosaic file used to mean writing Rust against the library API. This adds a mosaic binary to inspect, query and import files from the shell, with a command surface aligned to parquet-cli so the workflow is familiar.

Commands — all support --json

Command Shows
schema column names, Arrow types, nullability, bucket assignment
meta row groups, row counts, per-column stats (null_count/min/max)
cat / head rows as a table; -n, --all, -c/--columns, --where (stats pushdown)
count total row count
convert import CSV or JSON lines into a new file (schema inferred, --stats)
pages per-column encoding (plain/const/dict/all_null) + slot size
footer magic, version, bucket count, compression
column-size on-disk bytes per column + total compression ratio
dictionary dump a dict-encoded column (-c/--column)
buckets bucket layout per row group (monolithic vs paged) + ratio
mosaic convert data.csv -o data.mosaic --stats id
mosaic cat data.mosaic --all --where "id>100"   # skips row groups via min/max
mosaic count data.mosaic

Structure

  • New cli workspace crate: clap commands, text/JSON renderers, where filter + stats pushdown, CSV/JSON import; reads via a file-backed InputFile (pread)
  • Core: read-only additive accessors only — no format or behavior change; convert uses the existing MosaicWriter
  • Docs: docs/cli.html and cli/README.md cover every command with text/JSON examples

Tests

221 pass, 0 fail — cli e2e covers all commands incl. compression ratio, where filter, stats pushdown boundaries and CSV/JSON round-trip; core unchanged (126 + 53 + 21)

mingfeng and others added 3 commits June 16, 2026 04:22
Mosaic previously shipped no viewer tooling — inspecting a file meant
writing Rust against the library API. Add a `mosaic` binary (a new `cli`
workspace crate) mirroring parquet-cli:

- schema: column names, Arrow types, nullability, bucket assignment
- meta:   row groups, rows, per-column stats (null_count/min/max)
- cat:    first N rows as a table, with -n and --columns projection
- pages:  per-column encoding (plain/const/dict/all_null) + slot size

All commands support --json. The reader is driven over a new file-backed
InputFile (pread). Core gains three small read-only accessors used by
`pages`: BucketReader::encodings(), ColumnPageReader::encoding(), and
MosaicReader::page_infos(). No format/behavior change; 199 core tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a core regression test for MosaicReader::page_infos asserting
plain/dict/const detection on a paged-bucket file, and CLI unit tests
for the fmt helpers (json escaping, value/encoding rendering, ndjson
null handling, table truncation).
Drive the mosaic binary against a fixture file (via CARGO_BIN_EXE) and
assert stdout for schema/meta/pages/cat, --json output, projection,
row truncation and missing-file failure. No external dev-deps.
@JingsongLi

Copy link
Copy Markdown
Contributor

Can you also add documentation? And can you compare the API design differences with Parquet CLI?

Adds docs/cli.html documenting the mosaic inspector (schema/meta/pages/cat,
text + JSON) with a parquet-cli command mapping and design-difference table,
addressing the review asks on apache#66. Adds CLI to the nav across doc pages.
mingfeng added 8 commits June 22, 2026 03:26
Align the viewer command set with parquet-cli/arrow-rs: head (alias of
cat), footer (magic/version/buckets/compression), column-size (on-disk
bytes per column), dictionary (dump dict-encoded entries). Core gains
compression()/dict_values()/dictionary() read-only accessors. e2e tests
cover the new commands.
Mosaic's column-bucket grouping has no parquet equivalent. Add a
buckets command printing, per row group, each bucket's kind
(empty/monolithic/paged), on-disk size and member columns. Core gains
MosaicReader::bucket_infos(). e2e covered.
Align dictionary column selection with parquet-cli's -c flag instead of
a positional argument; update e2e.
Completes JSON output across all 9 commands; dict columns emit an array,
non-dict row groups emit null. e2e extended.
Expand docs/cli.html and cli/README.md to cover every command
(schema/meta/footer/buckets/pages/dictionary/column-size/cat/head) with
usage and example output. Drop all comparison content per maintainer
preference.
Remove the near-trivial encoding_names mapping test; extend footer and
buckets e2e to cover their --json output, improving CLI feature coverage.
The e2e tests carry their own fixture writer; the standalone gen.rs
example duplicated it and was unreferenced.
@jianguotian jianguotian changed the title feat(cli): add mosaic inspector CLI (schema/meta/cat/pages) feat(cli): add mosaic inspector CLI (9 commands, text + JSON) Jun 22, 2026
@jianguotian jianguotian changed the title feat(cli): add mosaic inspector CLI (9 commands, text + JSON) feat(cli): add mosaic inspector CLI Jun 22, 2026
@jianguotian jianguotian changed the title feat(cli): add mosaic inspector CLI feat(cli): add mosaic inspector CLI Jun 22, 2026
mingfeng added 5 commits June 22, 2026 08:15
column-size summed PageInfo.slot_size, which is 0 for monolithic buckets
(the default for small files), so every column reported 0 B. Attribute each
bucket's on-disk size to its columns via bucket_infos instead; add a
default-threshold e2e so monolithic files are covered. Also emit Date32 as a
bare epoch-day integer in cat --json for consistency with other numerics.
BucketInfo now carries uncompressed size (exact for monolithic, unknown for
paged). column-size prints a total with uncompressed bytes + ratio; buckets
shows per-bucket ratio when known. Tests + docs updated.
count prints total rows; --all drops the -n cap; --where applies one
column/op/value condition (=,!=,>,>=,<,<=), numeric or string. Tests + docs.
convert imports a CSV into a new Mosaic file (schema inferred, --stats picks
min/max columns). cat --where now skips row groups whose min/max provably
exclude the predicate, conservative (any missing stat keeps the group).
Tests + docs.
convert dispatches on extension: .json/.ndjson/.jsonl read one object per line
via arrow-json (CSV path unchanged). Test + docs.
@jianguotian jianguotian changed the title feat(cli): add mosaic inspector CLI feat(cli): add mosaic command-line inspector (11 commands) Jun 23, 2026
mingfeng added 2 commits June 22, 2026 21:36
… atomicity

- page_infos validates paged slot sizes sum to total (rejects forged sizes
  that could drive a ~4GiB read, matching the projected reader)
- cat --where: ordering ops on a non-numeric column/value error instead of
  silently dropping all rows
- cat --json: NaN/Infinity emit null (was invalid JSON)
- convert writes a temp file and renames on success (no truncated output)
@jianguotian jianguotian changed the title feat(cli): add mosaic command-line inspector (11 commands) [cli] Add mosaic command-line tool Jun 23, 2026

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few focused comments on CLI semantics and Parquet CLI alignment.

Comment thread cli/src/fmt.rs Outdated
Float32 if !arr.as_any().downcast_ref::<Float32Array>().unwrap().value(row).is_finite() => "null".into(),
Float64 if !arr.as_any().downcast_ref::<Float64Array>().unwrap().value(row).is_finite() => "null".into(),
// Date32 is an epoch-day integer; emit bare like other numerics.
_ => cell(arr, row),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cat --json can emit invalid JSON for Mosaic types that are supported by the core reader but not rendered by cell(). For example Binary, Decimal, Time32, Timestamp, List, and Map currently fall through to ?, and this branch writes that value unquoted, producing output like "col":?. Parquet/Arrow CLI paths use a real JSON serializer or type-aware value rendering; please either route this through Arrow JSON writing or cover all supported Mosaic types explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cat --json can emit invalid JSON for Mosaic types that are supported by the core reader but not rendered by cell(). For example Binary, Decimal, Time32, Timestamp, List, and Map currently fall through to ?, and this branch writes that value unquoted, producing output like "col":?. Parquet/Arrow CLI paths use a real JSON serializer or type-aware value rendering; please either route this through Arrow JSON writing or cover all supported Mosaic types explicitly.

Fixed, cat --json now goes through Arrow's JSON writer, so all reader types render as valid JSON.

Comment thread cli/src/fmt.rs Outdated
let mask: Vec<bool> = (0..batch.num_rows()).map(|r| {
if col.is_null(r) { return false; }
let lhs = cell(col.as_ref(), r);
match (lhs.parse::<f64>(), w.value.parse::<f64>()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes equality semantics depend on whether the rendered strings happen to parse as numbers. On a Utf8 column, --where s=1 matches both "1" and "01" because both sides parse as f64. Since Parquet CLI does not define this filter behavior, Mosaic should make the semantics type-driven: exact string comparison for Utf8, numeric comparison only for numeric columns.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes equality semantics depend on whether the rendered strings happen to parse as numbers. On a Utf8 column, --where s=1 matches both "1" and "01" because both sides parse as f64. Since Parquet CLI does not define this filter behavior, Mosaic should make the semantics type-driven: exact string comparison for Utf8, numeric comparison only for numeric columns.

Fixed, comparison is type-driven now: numeric columns compare numerically, others as exact strings. s=01 no longer matches 1.

Comment thread cli/src/main.rs
// paged layouts); split a bucket's size across its member columns.
for b in reader.bucket_infos(rg)? {
if b.columns.is_empty() { continue; }
split_evenly(b.size, &b.columns, &mut bytes);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This differs materially from Parquet CLI column-size, which sums exact column-chunk sizes from metadata. Here the bucket size is split evenly across member columns, so the per-column bytes can be quite misleading; for paged buckets we should be able to use per-column slot sizes instead. If monolithic bucket attribution must remain approximate, please label it as approximate in the command output/docs rather than presenting it as exact per-column size.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This differs materially from Parquet CLI column-size, which sums exact column-chunk sizes from metadata. Here the bucket size is split evenly across member columns, so the per-column bytes can be quite misleading; for paged buckets we should be able to use per-column slot sizes instead. If monolithic bucket attribution must remain approximate, please label it as approximate in the command output/docs rather than presenting it as exact per-column size.

Fixed, paged buckets use exact per-column slot sizes; monolithic multi-column buckets are split and labeled (approx).

Comment thread cli/README.md Outdated

## Commands

Every command accepts `--json`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert does not accept --json, so this statement is currently inaccurate. Please scope this to the inspection/query commands or add a JSON mode to convert.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert does not accept --json, so this statement is currently inaccurate. Please scope this to the inspection/query commands or add a JSON mode to convert.

Fixed, scoped the statement to inspection/query commands; convert writes a file.

mingfeng added 2 commits June 23, 2026 00:44
…tial

Paged buckets don't record uncompressed size, so summing across a mixed file
gave a misleading total ratio (e.g. 0.01x). Only report the total ratio when
every non-empty bucket's uncompressed size is known.
… column-size

- cat --json renders via Arrow JSON writer: all reader types valid, NaN->null
- where: numeric cols compare numerically, others exact string; ordering numeric-only
- column-size: paged uses exact slot sizes, monolithic multi-col marked (approx)
- README: scope --json to inspection/query commands (convert writes a file)
mingfeng and others added 19 commits June 26, 2026 09:20
After the ARRAY merge, paged directories carry a child header plus one slot
per physical column; page_infos/slot_sizes parsed it as N logical u32s and
mis-read or failed validation. Add paged_dir() to skip the header and read
physical slots; attribute child bytes to the parent. Test over a List column.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Float32 cell widened to f64 never equals the f64-parsed RHS (stored 0.1f32
!= 0.1f64), so price=0.1 dropped every row. Parse the RHS as f32 for Float32
columns. Test added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the f32 where fix, stats_exclude still parsed the RHS as f64 while
apply_where used f32, so a Float32 group with a matching row could be pruned
('no rows'). Round the RHS to f32 when stats are Float; test now uses stats.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cat/head printed Utf8 cells verbatim, so a crafted file's ANSI escapes
manipulated the inspector terminal (CWE-150); the JSON path was already
escaped. Replace control chars with U+FFFD; test added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A crafted file's strings reach the terminal raw, so attacker-controlled
bytes could inject ANSI escapes. Add fmt::safe() and route every text-mode
sink through it: stats min/max (render_value), table cells, dict entries,
and schema column names in inspect/meta/pages/column-size/buckets/headers.
JSON output stays escaped by json_str/the writer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 65536x ratio let a ~1 MiB compressed block legally request 64 GiB
before reading a byte. Clamp the cap to a 512 MiB absolute ceiling so a
tiny file can't drive a giant pre-alloc; a real slot/page is far below it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Float32 path re-parsed the RHS and unwrap_or(NaN) was unreachable (any
f64-parseable literal also f32-parses, saturating to inf). Reuse the
already-parsed rhs rounded to f32 width; symmetric with stats_exclude.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
meta/dictionary --json wrapped render_value(), leaking human suffixes like
"18627 (epoch-day)" / "5 (ms)" into JSON — unparseable downstream. Add
render_json() that strips units (nanos collapse to a count); text keeps them.
Also sum row-group counts via ? so meta/count fail loud instead of unwrap_or(0)
silently undercounting a corrupt file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cell() re-ran the data_type match + downcast for every (row,col); cat
renders all rows by default. Replace with col_formatter() built once per
column per batch, mirroring apply_where. Same output, no per-cell downcast.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
slot_sizes re-ran expand_col_types just to map ARRAY child slots to parents
right after paged_dir already computed it. Return children from paged_dir and
reuse them — one expand per paged bucket, one fewer reach into bucket_writer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FileOut was implemented twice (convert path + e2e tests) with subtly
different pos accounting. Move it to core::writer as FileSink (where the
OutputFile trait lives); both reuse it, so tests now cover the real sink.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The comparison operator was a stringly-typed &'static str threaded through
cmp_op/excl/apply_where with _ => false fallthrough. Replace with an Op enum;
cmp_op/excl are now exhaustive, so a new operator must be handled everywhere.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cat collected every row group's batch before printing; cat --json on a big
file held the whole result in memory. Stream each group out as ndjson; the
table path still buffers (column widths need all rows first).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fmt.rs mixed presentation (render/table/json) with query filtering (Where,
Op, apply_where, stats_exclude). Move the filter half to filter.rs verbatim;
pure relocation, no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eight commands hand-assembled JSON via format!/json_str; a missed comma or
escape was silent. Add jsonout models (#[derive(Serialize)]) and serialize
each; output byte-identical (verified across all 8). serde was already in the
tree via arrow. Removes json_str.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…name helpers

- arch-2: pages/column_size repeated the -c membership check three ways; one
  selected() helper now backs all of them.
- arch-3: bucket_kind returns String like encoding_name, so callers handle one type.
- arch-4: convert's two passes share one open() closure instead of four File::open.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jianguotian

Copy link
Copy Markdown
Contributor Author

Please make sure your ci passed, you can run it in your local.

I missed cargo fmt locally before. Now fixed it, and verified clippy + all rust/java/python tests pass locally.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest revision with a focus on Parquet CLI alignment and API design.

Comment thread cli/src/main.rs
.iter()
.position(|c| c.name == p.column)
});
let mut batches: Vec<RecordBatch> = Vec::new();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that cat defaults to all rows, matching parquet-cli, the non-JSON path should not collect every row group into batches. Parquet CLI streams records as it reads them; this implementation makes mosaic cat file load the whole file before printing. Could we either stream text output per batch/row group, or reserve the global pretty table for bounded head / -n output?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: unbounded text cat now streams per row group, while bounded output still buffers for table width.

Comment thread cli/src/main.rs
if json {
let mut row_groups = Vec::new();
for rg in 0..nrg {
let pgs = reader

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-c/--columns is currently only an output filter: page_infos(rg) has already inspected every column. In parquet-cli, column selection is applied before page iteration, so pages -c one_col avoids scanning unrelated columns. It would be better for the CLI to call a projected reader API, for example page_infos_projected(rg, selected_indices), so the option's semantics and cost match the API surface.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: pages -c now uses page_infos_projected, so unrelated columns are not inspected.

Comment thread core/src/reader.rs Outdated
column_index: gi,
bucket: b,
encoding: Encoding::from_code(enc),
slot_size: sizes[local],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For paged buckets with ARRAY/LIST columns, sizes[local] is only the primary physical slot; child slots returned by paged_dir() are ignored here. slot_sizes() already attributes child bytes to the logical parent, so pages and column-size can disagree for nested columns. Please sum child slot sizes into the parent PageInfo.slot_size and add an assertion in the existing paged-array test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: PageInfo.slot_size now includes ARRAY/LIST child slots, with a paged-array test assertion.

Comment thread cli/src/fmt.rs
millis,
nanos_of_milli,
} => (*millis as i128 * 1_000_000 + *nanos_of_milli as i128).to_string(),
_ => render_value(v),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fallback routes string values through render_value(), which calls safe() and replaces control characters. JSON output should be lossless and machine-readable, leaving escaping to serde/Arrow; terminal sanitization should stay in text output only. Please keep raw strings for JSON, and apply the same principle to JSON column names such as buckets --json.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: JSON string values and bucket column names now stay lossless; only text output uses safe().

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review on the latest revision.

Comment thread cli/src/main.rs Outdated
#[arg(long)]
json: bool,
},
/// Print rows as a table (default: first 10; use --all to scan all).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reintroduces the Parquet CLI mismatch we fixed earlier. In parquet-cli, cat defaults to no row limit and head is the preview command with default 10 rows; adding --all makes mosaic cat file behave like head, so users coming from parquet-cli will silently see only a preview. Please keep cat unbounded by default and address memory usage by streaming/bounded table formatting rather than changing the command semantics. The docs/tests added in this commit should be updated accordingly too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: cat is unbounded by default again, and memory usage is handled by streaming output.

mingfeng and others added 3 commits June 29, 2026 02:45
convert: use a process-unique sibling temp name instead of fixed
out.mosaic.tmp, so a stray user file is never clobbered/removed.
dictionary: reject List/LargeList/Map columns (first physical slot is
the length column) instead of printing misleading junk, matching
parquet-cli's primitive-leaf rule.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review on the latest revision.

Comment thread cli/src/main.rs Outdated
/// Limit to N rows.
#[arg(short = 'n', long)]
num: Option<usize>,
/// Print every row.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that cat is unbounded by default again, this flag no longer adds a distinct mode; its only visible effect is that mosaic cat -n 10 --all silently ignores the limit. Parquet CLI keeps this surface smaller (cat = all rows, head/-n = bounded output). Could we either drop --all, or at least mark it as conflicting with -n so the API contract stays unambiguous?

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit 9605afd into apache:main Jun 30, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants