Skip to content

# [fix](topn lazy materialization) Disable topn lazy materialization on non-light-schema-change tables#65415

Open
liutang123 wants to merge 1 commit into
apache:masterfrom
liutang123:fix-topn-lazy-non-lsc-table
Open

# [fix](topn lazy materialization) Disable topn lazy materialization on non-light-schema-change tables#65415
liutang123 wants to merge 1 commit into
apache:masterfrom
liutang123:fix-topn-lazy-non-lsc-table

Conversation

@liutang123

@liutang123 liutang123 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

A top-N query that emits a non-order-by column fails on a non-light-schema-change OLAP table (a table created/upgraded with light_schema_change = false, where every column's uniqueId is -1):

select id, name from tbl order by createdate desc limit 10;
ERROR 1105 (HY000): errCode = 2, detailMessage =
[INTERNAL_ERROR]field name is invalid. field=__DORIS_GLOBAL_ROWID_COL__tbl,
field_name_to_index=[...], col_unique_id=2147483647

Fix: disable topn lazy materialization for non-light-schema-change OLAP tables in MaterializeProbeVisitor and fall back to normal topn (the working two-phase read path). A new helper supportOlapTopnLazyMaterialize() consolidates the existing AGG_KEYS exclusion with the new light_schema_change requirement, applied at visitPhysicalOlapScan, visitPhysicalCatalogRelation and visitPhysicalFilter.

Add regression test topn_lazy_light_schema_change verifying:

  • light_schema_change=false: no lazy materialization in the plan, correct results.
  • light_schema_change=true: lazy materialization still applies, correct results.

Behavior after the fix

Table Plan Result
light_schema_change = false plain PhysicalOlapScan (no lazy) → safe two-phase read correct values
light_schema_change = true PhysicalLazyMaterialize / PhysicalLazyMaterializeOlapScan (unchanged) correct values

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

… non-light-schema-change tables

TopN lazy materialization appends a synthetic global row-id column
(__DORIS_GLOBAL_ROWID_COL__<table>) to the OLAP scan. The BE only rebuilds the
tablet schema from FE's columns_desc when columns_desc[0].col_unique_id >= 0,
which only holds for light_schema_change tables. On a non-light-schema-change
table every column's uniqueId is -1, so the synthetic row-id column never enters
the BE tablet schema and the scan fails with:

  field name is invalid. field=__DORIS_GLOBAL_ROWID_COL__<table>

(forcing the column into the scan schema instead makes the second-phase fetch
silently return NULL for the lazily-materialized columns, which is worse).

Fix: disable topn lazy materialization for non-light-schema-change OLAP tables in
MaterializeProbeVisitor and fall back to normal topn (the working two-phase read
path). A new helper supportOlapTopnLazyMaterialize() consolidates the existing
AGG_KEYS exclusion with the new light_schema_change requirement, applied at
visitPhysicalOlapScan, visitPhysicalCatalogRelation and visitPhysicalFilter.

Add regression test topn_lazy_light_schema_change verifying:
- light_schema_change=false: no lazy materialization in the plan, correct results.
- light_schema_change=true: lazy materialization still applies, correct results.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liutang123

Copy link
Copy Markdown
Contributor Author

run buildall

@liutang123

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR 65415 for TopN lazy materialization gating on non-light-schema-change OLAP tables.

Critical checkpoint conclusions:

  • Goal and coverage: The PR disables OLAP TopN lazy materialization before source selection when the table is AGG_KEYS or has light_schema_change=false, and adds a regression covering explicit false and true table properties.
  • Scope: The change is small and focused on MaterializeProbeVisitor support checks plus one targeted regression suite.
  • Concurrency/lifecycle: No new concurrency, locks, async lifecycle, static initialization, persistence, or transaction paths are introduced.
  • Config/session compatibility: No new config is added. Existing topn_lazy_materialization_threshold and topn_lazy_materialization_using_index paths remain unchanged; the new table-property gate is applied in the normal scan, catalog-relation, and using-index filter probe paths.
  • Parallel paths: The relevant OLAP probe entry points are covered; non-OLAP lazy materialization paths are unchanged.
  • Test coverage: The regression checks both plan shape and query result for light_schema_change=false fallback and light_schema_change=true lazy behavior. I did not run the regression locally in this shallow Actions checkout because thirdparty/installed and thirdparty/installed/bin/protoc are missing; GitHub CI currently has style/license checks passing, with compile/FE UT/performance still pending.
  • User focus: No additional user-provided focus points were supplied.

Subagent conclusions: optimizer-rewrite reported NO_NEW_VALUABLE_FINDINGS; tests-session-config reported NO_NEW_VALUABLE_FINDINGS. No subagent candidate became an inline comment, and none were dismissed or merged as duplicates. Final convergence round 1 ended with both live subagents reporting NO_NEW_VALUABLE_FINDINGS for the same empty inline-comment set and this comment-only review body.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 6.85% (5/73) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants