Projection rewrites do not rebase physical LambdaVariable bindings across schema boundaries

[lonless9]

Describe the bug

Physical lambda expressions can fail during execution after an optimizer rewrite moves the expression across a schema boundary.

This affects at least two paths:

1. collapsing consecutive ProjectionExecs;
2. pushing a join-filter expression below a NestedLoopJoinExec.

In both cases, the query plans successfully, but the physical LambdaVariable retains an index and FieldRef that no longer match the RecordBatch against which it is evaluated.

To Reproduce

Reproducer 1: projection-chain collapse

SET datafusion.sql_parser.dialect = spark;

SELECT array_transform(arr, x -> x)
FROM (
  SELECT arr
  FROM (VALUES ([1, 2], 7)) AS t(arr, padding)
) AS q;

Actual result:

Error: Execution error: Field of physical LambdaVariable with index 0 doesn't match batch field during evaluation Field { "x": nullable Int64 } != Field { "column2": Int64 }

The optimized physical plan has collapsed the intermediate projection:

ProjectionExec
  array_transform(column1, (x) -> x@1)
  DataSourceExec

Reproducer 2: NestedLoopJoin filter pushdown

SET datafusion.sql_parser.dialect = spark;

SELECT l.id, r.k
FROM (
  VALUES
    (1, [1], 10, 20),
    (2, arrow_cast([NULL], 'List(Int64)'), 30, 40)
) AS l(id, arr, padding_1, padding_2)
JOIN (VALUES (0), (1), (2)) AS r(k)
ON (cardinality(array_filter(l.arr, x -> x IS NOT NULL)) > 0)
   = (l.id > r.k)
ORDER BY l.id, r.k;

Actual result:

Error: Execution error: Field of physical LambdaVariable with index 0 doesn't match batch field during evaluation Field { "x": nullable Int64 } != Field { "column4": Int64 }

EXPLAIN confirms that this goes through NestedLoopJoinExec and join-filter projection pushdown:

NestedLoopJoinExec
  ProjectionExec
    arr: column2
    id: column1
    join_proj_push_down_1:
      cardinality(
        array_filter(column2, (x) -> x@3 IS NOT NULL)
      ) > 0

Expected behavior

The first query should return:

[1, 2]

The second query should return:

1, 0
2, 2

Additional context

Related lakehq/sail#2413

[lonless9]

• [EPIC] Full lambda support #21172

[shinzoxD]

take

[shinzoxD]

Opened #24406