feat: add sparse structural encoding

docs/src/format/file/encoding.md

@@ -327,6 +327,85 @@ The protobuf for the full zip layout describes the compression of the data buffe
 size of the control words and how many bits we have per value (for fixed-width data) or how many bits we
 have per offset (for variable-width data).
 
+### Sparse Page Layout
+
+The sparse page layout is a 2.3+ structural layout for sparse nested data. It is selected explicitly with
+`lance-encoding:structural-encoding=sparse`, or automatically in 2.3+ when dense mini-block structural information
+would exceed the mini-block structural page budget. Writers must not emit this layout for file versions before 2.3.
+
+Sparse layout stores Arrow structure as native slot-domain mappings instead of dense repetition / definition events.
+The value buffers are still mini-block compressed. The structural layers are stored from outer-most to inner-most:
+
+- validity layer: null slots in the parent slot domain
+- list layer: non-empty parent slots, child counts, and null parent slots
+- fixed-size-list layer: the fixed dimension and null parent slots
+
+Valid empty list slots are not stored in the non-empty positions. They are represented by their absence from both the
+non-empty positions and null positions for that list layer.
+
+Sparse structural sets are semantic, not necessarily materialized position arrays. Position sets are encoded as one of:
+
+- empty: no slot is selected
+- all: every slot in the layer domain is selected
+- range: one contiguous slot range is selected
+- explicit: a delta-compressed `u64` position buffer
+
+List counts are encoded as one of:
+
+- empty: there are no non-empty list slots
+- constant: every non-empty list slot has the same child count
+- explicit: a compressed `u64` count buffer
+
+#### Selection and Compatibility
+
+Writers may emit sparse layout only for file versions 2.3 and later. Explicit sparse selection is requested with field
+metadata `lance-encoding:structural-encoding=sparse`; the request is invalid for older file versions. In 2.3 and later,
+the automatic writer path may select sparse layout when the page has native sparse structural layers and the equivalent
+mini-block repetition / definition stream would exceed the mini-block structural page budget. When sparse is not requested
+and the structural budget is satisfied, writers use the normal mini-block / full-zip selection path.
+
+Sparse layout is a page layout, not a column-level promise. Readers must identify it from `PageLayout.sparse_layout`.
+The `lance-encoding:structural-encoding` field metadata is a writer input only and must not be used by readers to infer
+the layout of existing pages.
+
+#### Buffers
+
+Sparse pages always have a metadata buffer followed by one value buffer, then zero or more structural buffers.
+
+| Buffer | Meaning |
+| ------ | ------- |
+| 0      | Value chunk metadata |
+| 1      | Mini-block encoded value chunks |
+| 2+     | Structural position/count buffers, in `SparseStructuralLayer` order |
+
+Buffer 0 stores one 8-byte entry per value chunk. The first 4 bytes store the chunk size divided by 8 minus one. The
+second 4 bytes store the number of visible values in the chunk. The sum of chunk value counts must match
+`SparseLayout.num_visible_items`, and the sum of chunk byte sizes must exactly match the value buffer size.
+
+Structural position buffers are present only for explicit position sets. They store delta-encoded `u64` positions.
+Positions must be strictly increasing after delta decoding and must be within the layer's parent slot domain. List count
+buffers are present only for explicit count sets. They store `u64` child counts for each non-empty list slot.
+
+`SparseStructuralLayer.non_empty_positions`, `SparseStructuralLayer.counts`, and
+`SparseStructuralLayer.null_positions` are the canonical structural representation. `LIST` layers require
+`non_empty_positions`, `counts`, and `null_positions`; `VALIDITY` layers require only `null_positions`;
+`FIXED_SIZE_LIST` layers require `null_positions` and `fixed_size_list_dimension`.
+
+#### Protobuf
+
+```protobuf
+%%% proto.message.SparseLayout %%%
+```
+
+```protobuf
+%%% proto.message.SparseStructuralLayer %%%
+```
+
+The `value_compression` field is required. The physical page buffer list must match the structural layer metadata:
+each explicit position/count set requires exactly one corresponding compressed structural buffer, and empty/all/range or
+constant sets must not include an unused compression descriptor. Readers should reject malformed sparse pages with a
+format error.
+
 ### Constant Page Layout
 
 This layout is used when all (visible) values in the page are the same scalar value.
@@ -549,7 +628,7 @@ options. However, they can also be set in the field metadata in the schema.
 | `lance-encoding:dict-values-compression-level` | Integers (scheme dependent) | Varies by scheme | Compression level for dictionary values general compression                             |
 | `lance-encoding:general`             | `off`, `on`                          | `off`            | Whether to apply general compression.                                                   |
 | `lance-encoding:packed`              | Any string                           | Not set          | Whether to apply packed struct encoding (see above).                                    |
-| `lance-encoding:structural-encoding` | `miniblock`, `fullzip`               | Not set          | Force a particular structural encoding to be applied (only useful for testing purposes) |
+| `lance-encoding:structural-encoding` | `miniblock`, `fullzip`, `sparse`     | Not set          | Force a structural encoding. `sparse` requires file version 2.3+.                      |
 
 ### Configuration Details
 

docs/src/format/file/versioning.md

@@ -21,7 +21,7 @@ The following values are supported:
 | 2.0            | 0.16.0                | Any                   | Rework of the Lance file format that removed row groups and introduced null support for lists, fixed size lists, and primitives |
 | 2.1            | 0.38.1                | Any                   | Enhances integer and string compression, adds support for nulls in struct fields, and improves random access performance with nested fields. |
 | 2.2            | None                  | Any                   | Adds support for newer nested type/encoding capabilities (including map support) and 2.2-era storage features. |
-| 2.3 (unstable) | None                  | Any                   | Adds experimental encodings for upcoming features. |
+| 2.3 (unstable) | None                  | Any                   | Adds experimental encodings for upcoming features, including sparse structural encoding. |
 | legacy         | N/A                   | N/A                   | Alias for 0.1 |
 | stable         | N/A                   | N/A                   | Alias for the default version for new datasets in the Lance release you are running. |
 | next           | N/A                   | N/A                   | Alias for the latest unstable version in the Lance release you are running.|

protos/encodings_v2_1.proto

@@ -147,6 +147,106 @@ message FullZipLayout {
   repeated RepDefLayer layers = 8;
 }
 
+// A layout used for sparse nested pages where the Arrow structure is discrete.
+//
+// Sparse layout stores structural layers as native slot-domain mappings instead of dense
+// repetition / definition events:
+//
+// - validity layer: slot -> valid/null
+// - list layer: parent slot -> child slot range, plus null parent slots if present
+// - fixed-size-list layer: parent slot -> deterministic child slot range, plus null parent slots
+//
+// Value buffers are still mini-block compressed, but the structural side is a sparse index over
+// Arrow structure.
+message SparseLayout {
+  // Description of the compression of values.
+  CompressiveEncoding value_compression = 3;
+  // The number of value buffers in each chunk. This does NOT include repetition or definition
+  // buffers.
+  uint64 num_buffers = 5;
+  // The number of structural items in the page, including invisible structural items.
+  uint64 num_items = 7;
+  // The number of visible items in the page.
+  uint64 num_visible_items = 8;
+  // If true, chunk-local value buffer sizes are stored as u32 values. If false, they are stored
+  // as u16 values, matching mini-block chunk payload metadata.
+  bool has_large_chunk = 9;
+  // Structural layers, stored from outer-most to inner-most.
+  repeated SparseStructuralLayer structural_layers = 10;
+}
+
+message SparseStructuralLayer {
+  enum Kind {
+    SPARSE_LAYER_UNSPECIFIED = 0;
+    // A nullable item / struct layer. Null slots are recorded in the parent slot domain.
+    SPARSE_LAYER_VALIDITY = 1;
+    // A list-like layer. Non-empty parent slots map to child ranges; missing valid slots are empty.
+    SPARSE_LAYER_LIST = 2;
+    // A fixed-size-list layer. Child ranges are deterministic from fixed_size_list_dimension.
+    SPARSE_LAYER_FIXED_SIZE_LIST = 3;
+  }
+
+  Kind kind = 1;
+  // Number of slots in this layer's parent domain.
+  uint64 num_slots = 2;
+  // Number of slots in this layer's child domain after this layer is applied. For validity
+  // layers this matches num_slots because Arrow child slot numbering is unchanged.
+  uint64 num_child_slots = 3;
+  // Semantic representation for non-empty LIST slots. Required for LIST and absent otherwise.
+  SparsePositionSet non_empty_positions = 4;
+  // Semantic representation for LIST child counts. Required for LIST and absent otherwise.
+  SparseCountSet counts = 5;
+  // Semantic representation for null slots. Required for all sparse structural layers.
+  SparsePositionSet null_positions = 6;
+  // Fixed list dimension. Required for FIXED_SIZE_LIST and zero otherwise.
+  uint64 fixed_size_list_dimension = 7;
+}
+
+message SparsePositionEmpty {}
+
+message SparsePositionAll {}
+
+message SparsePositionRange {
+  // First position in the layer slot domain.
+  uint64 start = 1;
+  // Number of positions in the contiguous range.
+  uint64 length = 2;
+}
+
+message SparsePositionSet {
+  oneof positions {
+    // Delta-compressed u64 positions. Cardinality comes from num_positions.
+    CompressiveEncoding explicit = 1;
+    // A contiguous non-empty position range.
+    SparsePositionRange range = 2;
+    // Every slot in the layer domain is present.
+    SparsePositionAll all = 3;
+    // No slots are present.
+    SparsePositionEmpty empty = 4;
+  }
+  // Number of selected positions. For explicit sets this is the number of compressed u64
+  // positions. For empty/all/range sets this must match the semantic cardinality.
+  uint64 num_positions = 5;
+}
+
+message SparseCountEmpty {}
+
+message SparseCountConstant {
+  // Child count for every selected non-empty LIST slot.
+  uint64 value = 1;
+}
+
+message SparseCountSet {
+  oneof counts {
+    // Compressed u64 child counts. Cardinality is the containing LIST layer's non-empty position count.
+    CompressiveEncoding explicit = 1;
+    // A single child count for every non-empty LIST slot.
+    SparseCountConstant constant = 2;
+    // No counts; only valid when the containing LIST layer has no non-empty positions.
+    SparseCountEmpty empty = 3;
+  }
+}
+
 // A layout used for pages where all (visible) values are the same scalar value.
 //
 // This generalizes the prior AllNullLayout semantics for file_version >= 2.2.
@@ -206,6 +306,8 @@ message PageLayout {
     // A layout where large binary data is encoded externally
     // and only the descriptions are put in the page
     BlobLayout blob_layout = 4;
+    // A layout used for sparse nested pages with native structural layers (2.3+).
+    SparseLayout sparse_layout = 5;
   }
 }
 

rust/lance-encoding/src/constants.rs

@@ -55,6 +55,8 @@ pub const STRUCTURAL_ENCODING_META_KEY: &str = "lance-encoding:structural-encodi
 pub const STRUCTURAL_ENCODING_MINIBLOCK: &str = "miniblock";
 /// Value for fullzip structural encoding
 pub const STRUCTURAL_ENCODING_FULLZIP: &str = "fullzip";
+/// Value for sparse structural encoding
+pub const STRUCTURAL_ENCODING_SPARSE: &str = "sparse";
 
 // Byte stream split metadata keys
 /// Metadata key for byte stream split encoding configuration