[cli] Add mosaic command-line tool#66
Conversation
Mosaic previously shipped no viewer tooling — inspecting a file meant writing Rust against the library API. Add a `mosaic` binary (a new `cli` workspace crate) mirroring parquet-cli: - schema: column names, Arrow types, nullability, bucket assignment - meta: row groups, rows, per-column stats (null_count/min/max) - cat: first N rows as a table, with -n and --columns projection - pages: per-column encoding (plain/const/dict/all_null) + slot size All commands support --json. The reader is driven over a new file-backed InputFile (pread). Core gains three small read-only accessors used by `pages`: BucketReader::encodings(), ColumnPageReader::encoding(), and MosaicReader::page_infos(). No format/behavior change; 199 core tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a core regression test for MosaicReader::page_infos asserting plain/dict/const detection on a paged-bucket file, and CLI unit tests for the fmt helpers (json escaping, value/encoding rendering, ndjson null handling, table truncation).
Drive the mosaic binary against a fixture file (via CARGO_BIN_EXE) and assert stdout for schema/meta/pages/cat, --json output, projection, row truncation and missing-file failure. No external dev-deps.
|
Can you also add documentation? And can you compare the API design differences with Parquet CLI? |
Adds docs/cli.html documenting the mosaic inspector (schema/meta/pages/cat, text + JSON) with a parquet-cli command mapping and design-difference table, addressing the review asks on apache#66. Adds CLI to the nav across doc pages.
Align the viewer command set with parquet-cli/arrow-rs: head (alias of cat), footer (magic/version/buckets/compression), column-size (on-disk bytes per column), dictionary (dump dict-encoded entries). Core gains compression()/dict_values()/dictionary() read-only accessors. e2e tests cover the new commands.
Mosaic's column-bucket grouping has no parquet equivalent. Add a buckets command printing, per row group, each bucket's kind (empty/monolithic/paged), on-disk size and member columns. Core gains MosaicReader::bucket_infos(). e2e covered.
Align dictionary column selection with parquet-cli's -c flag instead of a positional argument; update e2e.
Completes JSON output across all 9 commands; dict columns emit an array, non-dict row groups emit null. e2e extended.
Expand docs/cli.html and cli/README.md to cover every command (schema/meta/footer/buckets/pages/dictionary/column-size/cat/head) with usage and example output. Drop all comparison content per maintainer preference.
Remove the near-trivial encoding_names mapping test; extend footer and buckets e2e to cover their --json output, improving CLI feature coverage.
The e2e tests carry their own fixture writer; the standalone gen.rs example duplicated it and was unreferenced.
6100aa5 to
924ac4b
Compare
mosaic inspector CLI (9 commands, text + JSON)
mosaic inspector CLI (9 commands, text + JSON)mosaic inspector CLI
mosaic inspector CLIcolumn-size summed PageInfo.slot_size, which is 0 for monolithic buckets (the default for small files), so every column reported 0 B. Attribute each bucket's on-disk size to its columns via bucket_infos instead; add a default-threshold e2e so monolithic files are covered. Also emit Date32 as a bare epoch-day integer in cat --json for consistency with other numerics.
BucketInfo now carries uncompressed size (exact for monolithic, unknown for paged). column-size prints a total with uncompressed bytes + ratio; buckets shows per-bucket ratio when known. Tests + docs updated.
count prints total rows; --all drops the -n cap; --where applies one column/op/value condition (=,!=,>,>=,<,<=), numeric or string. Tests + docs.
convert imports a CSV into a new Mosaic file (schema inferred, --stats picks min/max columns). cat --where now skips row groups whose min/max provably exclude the predicate, conservative (any missing stat keeps the group). Tests + docs.
convert dispatches on extension: .json/.ndjson/.jsonl read one object per line via arrow-json (CSV path unchanged). Test + docs.
mosaic command-line inspector (11 commands)
… atomicity - page_infos validates paged slot sizes sum to total (rejects forged sizes that could drive a ~4GiB read, matching the projected reader) - cat --where: ordering ops on a non-numeric column/value error instead of silently dropping all rows - cat --json: NaN/Infinity emit null (was invalid JSON) - convert writes a temp file and renames on success (no truncated output)
mosaic command-line inspector (11 commands)
JingsongLi
left a comment
There was a problem hiding this comment.
I left a few focused comments on CLI semantics and Parquet CLI alignment.
| Float32 if !arr.as_any().downcast_ref::<Float32Array>().unwrap().value(row).is_finite() => "null".into(), | ||
| Float64 if !arr.as_any().downcast_ref::<Float64Array>().unwrap().value(row).is_finite() => "null".into(), | ||
| // Date32 is an epoch-day integer; emit bare like other numerics. | ||
| _ => cell(arr, row), |
There was a problem hiding this comment.
cat --json can emit invalid JSON for Mosaic types that are supported by the core reader but not rendered by cell(). For example Binary, Decimal, Time32, Timestamp, List, and Map currently fall through to ?, and this branch writes that value unquoted, producing output like "col":?. Parquet/Arrow CLI paths use a real JSON serializer or type-aware value rendering; please either route this through Arrow JSON writing or cover all supported Mosaic types explicitly.
There was a problem hiding this comment.
cat --jsoncan emit invalid JSON for Mosaic types that are supported by the core reader but not rendered bycell(). For example Binary, Decimal, Time32, Timestamp, List, and Map currently fall through to?, and this branch writes that value unquoted, producing output like"col":?. Parquet/Arrow CLI paths use a real JSON serializer or type-aware value rendering; please either route this through Arrow JSON writing or cover all supported Mosaic types explicitly.
Fixed, cat --json now goes through Arrow's JSON writer, so all reader types render as valid JSON.
| let mask: Vec<bool> = (0..batch.num_rows()).map(|r| { | ||
| if col.is_null(r) { return false; } | ||
| let lhs = cell(col.as_ref(), r); | ||
| match (lhs.parse::<f64>(), w.value.parse::<f64>()) { |
There was a problem hiding this comment.
This makes equality semantics depend on whether the rendered strings happen to parse as numbers. On a Utf8 column, --where s=1 matches both "1" and "01" because both sides parse as f64. Since Parquet CLI does not define this filter behavior, Mosaic should make the semantics type-driven: exact string comparison for Utf8, numeric comparison only for numeric columns.
There was a problem hiding this comment.
This makes equality semantics depend on whether the rendered strings happen to parse as numbers. On a Utf8 column,
--where s=1matches both"1"and"01"because both sides parse asf64. Since Parquet CLI does not define this filter behavior, Mosaic should make the semantics type-driven: exact string comparison for Utf8, numeric comparison only for numeric columns.
Fixed, comparison is type-driven now: numeric columns compare numerically, others as exact strings. s=01 no longer matches 1.
| // paged layouts); split a bucket's size across its member columns. | ||
| for b in reader.bucket_infos(rg)? { | ||
| if b.columns.is_empty() { continue; } | ||
| split_evenly(b.size, &b.columns, &mut bytes); |
There was a problem hiding this comment.
This differs materially from Parquet CLI column-size, which sums exact column-chunk sizes from metadata. Here the bucket size is split evenly across member columns, so the per-column bytes can be quite misleading; for paged buckets we should be able to use per-column slot sizes instead. If monolithic bucket attribution must remain approximate, please label it as approximate in the command output/docs rather than presenting it as exact per-column size.
There was a problem hiding this comment.
This differs materially from Parquet CLI
column-size, which sums exact column-chunk sizes from metadata. Here the bucket size is split evenly across member columns, so the per-column bytes can be quite misleading; for paged buckets we should be able to use per-column slot sizes instead. If monolithic bucket attribution must remain approximate, please label it as approximate in the command output/docs rather than presenting it as exact per-column size.
Fixed, paged buckets use exact per-column slot sizes; monolithic multi-column buckets are split and labeled (approx).
|
|
||
| ## Commands | ||
|
|
||
| Every command accepts `--json`. |
There was a problem hiding this comment.
convert does not accept --json, so this statement is currently inaccurate. Please scope this to the inspection/query commands or add a JSON mode to convert.
There was a problem hiding this comment.
convertdoes not accept--json, so this statement is currently inaccurate. Please scope this to the inspection/query commands or add a JSON mode toconvert.
Fixed, scoped the statement to inspection/query commands; convert writes a file.
…tial Paged buckets don't record uncompressed size, so summing across a mixed file gave a misleading total ratio (e.g. 0.01x). Only report the total ratio when every non-empty bucket's uncompressed size is known.
… column-size - cat --json renders via Arrow JSON writer: all reader types valid, NaN->null - where: numeric cols compare numerically, others exact string; ordering numeric-only - column-size: paged uses exact slot sizes, monolithic multi-col marked (approx) - README: scope --json to inspection/query commands (convert writes a file)
After the ARRAY merge, paged directories carry a child header plus one slot per physical column; page_infos/slot_sizes parsed it as N logical u32s and mis-read or failed validation. Add paged_dir() to skip the header and read physical slots; attribute child bytes to the parent. Test over a List column. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A Float32 cell widened to f64 never equals the f64-parsed RHS (stored 0.1f32 != 0.1f64), so price=0.1 dropped every row. Parse the RHS as f32 for Float32 columns. Test added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After the f32 where fix, stats_exclude still parsed the RHS as f64 while
apply_where used f32, so a Float32 group with a matching row could be pruned
('no rows'). Round the RHS to f32 when stats are Float; test now uses stats.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cat/head printed Utf8 cells verbatim, so a crafted file's ANSI escapes manipulated the inspector terminal (CWE-150); the JSON path was already escaped. Replace control chars with U+FFFD; test added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A crafted file's strings reach the terminal raw, so attacker-controlled bytes could inject ANSI escapes. Add fmt::safe() and route every text-mode sink through it: stats min/max (render_value), table cells, dict entries, and schema column names in inspect/meta/pages/column-size/buckets/headers. JSON output stays escaped by json_str/the writer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 65536x ratio let a ~1 MiB compressed block legally request 64 GiB before reading a byte. Clamp the cap to a 512 MiB absolute ceiling so a tiny file can't drive a giant pre-alloc; a real slot/page is far below it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Float32 path re-parsed the RHS and unwrap_or(NaN) was unreachable (any f64-parseable literal also f32-parses, saturating to inf). Reuse the already-parsed rhs rounded to f32 width; symmetric with stats_exclude. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
meta/dictionary --json wrapped render_value(), leaking human suffixes like "18627 (epoch-day)" / "5 (ms)" into JSON — unparseable downstream. Add render_json() that strips units (nanos collapse to a count); text keeps them. Also sum row-group counts via ? so meta/count fail loud instead of unwrap_or(0) silently undercounting a corrupt file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cell() re-ran the data_type match + downcast for every (row,col); cat renders all rows by default. Replace with col_formatter() built once per column per batch, mirroring apply_where. Same output, no per-cell downcast. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
slot_sizes re-ran expand_col_types just to map ARRAY child slots to parents right after paged_dir already computed it. Return children from paged_dir and reuse them — one expand per paged bucket, one fewer reach into bucket_writer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FileOut was implemented twice (convert path + e2e tests) with subtly different pos accounting. Move it to core::writer as FileSink (where the OutputFile trait lives); both reuse it, so tests now cover the real sink. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The comparison operator was a stringly-typed &'static str threaded through cmp_op/excl/apply_where with _ => false fallthrough. Replace with an Op enum; cmp_op/excl are now exhaustive, so a new operator must be handled everywhere. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cat collected every row group's batch before printing; cat --json on a big file held the whole result in memory. Stream each group out as ndjson; the table path still buffers (column widths need all rows first). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fmt.rs mixed presentation (render/table/json) with query filtering (Where, Op, apply_where, stats_exclude). Move the filter half to filter.rs verbatim; pure relocation, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eight commands hand-assembled JSON via format!/json_str; a missed comma or escape was silent. Add jsonout models (#[derive(Serialize)]) and serialize each; output byte-identical (verified across all 8). serde was already in the tree via arrow. Removes json_str. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…name helpers - arch-2: pages/column_size repeated the -c membership check three ways; one selected() helper now backs all of them. - arch-3: bucket_kind returns String like encoding_name, so callers handle one type. - arch-4: convert's two passes share one open() closure instead of four File::open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
I missed |
JingsongLi
left a comment
There was a problem hiding this comment.
Reviewed the latest revision with a focus on Parquet CLI alignment and API design.
| .iter() | ||
| .position(|c| c.name == p.column) | ||
| }); | ||
| let mut batches: Vec<RecordBatch> = Vec::new(); |
There was a problem hiding this comment.
Now that cat defaults to all rows, matching parquet-cli, the non-JSON path should not collect every row group into batches. Parquet CLI streams records as it reads them; this implementation makes mosaic cat file load the whole file before printing. Could we either stream text output per batch/row group, or reserve the global pretty table for bounded head / -n output?
There was a problem hiding this comment.
Fixed: unbounded text cat now streams per row group, while bounded output still buffers for table width.
| if json { | ||
| let mut row_groups = Vec::new(); | ||
| for rg in 0..nrg { | ||
| let pgs = reader |
There was a problem hiding this comment.
-c/--columns is currently only an output filter: page_infos(rg) has already inspected every column. In parquet-cli, column selection is applied before page iteration, so pages -c one_col avoids scanning unrelated columns. It would be better for the CLI to call a projected reader API, for example page_infos_projected(rg, selected_indices), so the option's semantics and cost match the API surface.
There was a problem hiding this comment.
Fixed: pages -c now uses page_infos_projected, so unrelated columns are not inspected.
| column_index: gi, | ||
| bucket: b, | ||
| encoding: Encoding::from_code(enc), | ||
| slot_size: sizes[local], |
There was a problem hiding this comment.
For paged buckets with ARRAY/LIST columns, sizes[local] is only the primary physical slot; child slots returned by paged_dir() are ignored here. slot_sizes() already attributes child bytes to the logical parent, so pages and column-size can disagree for nested columns. Please sum child slot sizes into the parent PageInfo.slot_size and add an assertion in the existing paged-array test.
There was a problem hiding this comment.
Fixed: PageInfo.slot_size now includes ARRAY/LIST child slots, with a paged-array test assertion.
| millis, | ||
| nanos_of_milli, | ||
| } => (*millis as i128 * 1_000_000 + *nanos_of_milli as i128).to_string(), | ||
| _ => render_value(v), |
There was a problem hiding this comment.
This fallback routes string values through render_value(), which calls safe() and replaces control characters. JSON output should be lossless and machine-readable, leaving escaping to serde/Arrow; terminal sanitization should stay in text output only. Please keep raw strings for JSON, and apply the same principle to JSON column names such as buckets --json.
There was a problem hiding this comment.
Fixed: JSON string values and bucket column names now stay lossless; only text output uses safe().
JingsongLi
left a comment
There was a problem hiding this comment.
Follow-up review on the latest revision.
| #[arg(long)] | ||
| json: bool, | ||
| }, | ||
| /// Print rows as a table (default: first 10; use --all to scan all). |
There was a problem hiding this comment.
This reintroduces the Parquet CLI mismatch we fixed earlier. In parquet-cli, cat defaults to no row limit and head is the preview command with default 10 rows; adding --all makes mosaic cat file behave like head, so users coming from parquet-cli will silently see only a preview. Please keep cat unbounded by default and address memory usage by streaming/bounded table formatting rather than changing the command semantics. The docs/tests added in this commit should be updated accordingly too.
There was a problem hiding this comment.
Fixed: cat is unbounded by default again, and memory usage is handled by streaming output.
convert: use a process-unique sibling temp name instead of fixed out.mosaic.tmp, so a stray user file is never clobbered/removed. dictionary: reject List/LargeList/Map columns (first physical slot is the length column) instead of printing misleading junk, matching parquet-cli's primitive-leaf rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JingsongLi
left a comment
There was a problem hiding this comment.
Follow-up review on the latest revision.
| /// Limit to N rows. | ||
| #[arg(short = 'n', long)] | ||
| num: Option<usize>, | ||
| /// Print every row. |
There was a problem hiding this comment.
Now that cat is unbounded by default again, this flag no longer adds a distinct mode; its only visible effect is that mosaic cat -n 10 --all silently ignores the limit. Parquet CLI keeps this surface smaller (cat = all rows, head/-n = bounded output). Could we either drop --all, or at least mark it as conflicting with -n so the API contract stays unambiguous?
|
+1 |
Inspecting a Mosaic file used to mean writing Rust against the library API. This adds a mosaic binary to inspect, query and import files from the shell, with a command surface aligned to parquet-cli so the workflow is familiar.
Commands — all support
--json-n,--all,-c/--columns,--where(stats pushdown)--stats)-c/--column)Structure
Tests
221 pass, 0 fail — cli e2e covers all commands incl. compression ratio, where filter, stats pushdown boundaries and CSV/JSON round-trip; core unchanged (126 + 53 + 21)