`add_columns_with_schema` rejects all-null `Map<K, V>` columns

[kushudai]

Dataset.add_columns(pa.Schema) errors with Invalid user input: All-null columns must be nullable. on any Arrow Map<K, V> field, even when the outer Map is declared nullable=true.

Schema::all_fields_nullable pre-order walks every nested field and requires f.nullable on each. Arrow Map layout mandates entries.nullable = false and entries.key.nullable = false
(spec), which Lance itself enforces at field construction (field.rs#L1123-L1128).
The two checks contradict each other, so no Map column can pass the metadata-only AllNulls path regardless of the outer field's nullable flag.

Reproduction

import lance
import pyarrow as pa

ds = lance.write_dataset(
    pa.table({"x": pa.array([1, 2, 3], type=pa.int64())}),
    "/tmp/lance_map_allnulls_repro",
    mode="overwrite",
)
ds.add_columns(
    pa.schema([
        pa.field("cutoffs", pa.map_(pa.string(), pa.float64()), nullable=True),
    ])
)

OSError: Invalid user input: All-null columns must be nullable.,
.../rust/lance/src/dataset/schema_evolution.rs:361:28

Reproduced on pylance 4.0.0 and 6.0.0; Bug exists on main - all_fields_nullable body is byte-identical from v4.0.1 to as of this writing.

Related

NewColumnTransform::AllNulls introduced in #3391; existing test at schema_evolution.rs#L1177-L1209 covers the leaf non-nullable-outer-field case but no nested types.

Proposed Fix

I'm not actually sure but collection types like lists and maps should be allowed to be null-able?

[westonpace]

The all_fields_nullable check is only used during column addition / schema evolution because we have no default value and we are going to use null. However, it should be sufficient that the top-level of any nested (e.g. list/struct) type is nullable. I don't believe we need to recurse into the children. Do you want to create a PR with this fix?

[jiangxt2]

I'd like to pick this up.

The root cause is that schema.all_fields_nullable() (schema.rs:763) uses SchemaFieldIterPreOrder to recursively walk all nested fields. For a Map column like Map<Utf8, Float64> with nullable=true, the Arrow spec mandates that entries and key are non-nullable — so all_fields_nullable() returns false and the AllNulls path rejects the column.

The same contradiction applies to any nested type where an inner field is non-nullable — mandated by the Arrow spec for Map's entries/key, or user-defined for List/Struct children.

Proposed fix in schema_evolution.rs:370:

// before
if !schema.all_fields_nullable() {
// after
if !schema.fields.iter().all(|f| f.nullable) {

This checks only top-level fields, consistent with the guidance above that "top-level nullable is sufficient, no need to recurse into children." The all_fields_nullable() method itself will be left as-is.

I'll add test cases covering Map<K, V>, List<non-null child>, and Struct<non-null child> with NewColumnTransform::AllNulls.