Skip to content

GH-51638: [CI][C++][Parquet] Update expected JSON output in parquet-reader-test for simdjson 5.0 - #51661

Merged
kou merged 2 commits into
apache:mainfrom
tadeja:51638-parquet-JSONOutput
Oct 1, 2026
Merged

kou merged 2 commits into
apache:mainfrom
tadeja:51638-parquet-JSONOutput

Conversation

@tadeja

@tadeja tadeja commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Rationale for this change

See #51638 - fractured_json formatting changes with the new simdjson version 5.0 (whitespace, line breaks, key order in tables),
cause parquet-reader-test to fail on four of its tests TestJSONWithLocalFile.JSONOutput...

What changes are included in this PR?

Check simdjson::SIMDJSON_VERSION_MAJOR, and for >= 5 use the new expected formatting (JSON content hasn't changed).
Keep existing expected output formatting for simdjson 4.x

Are these changes tested?

Yes, CI runs pass.

Are there any user-facing changes?

No, only test changes.

Was AI used for this PR?

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@tadeja
tadeja marked this pull request as ready for review September 30, 2026 17:23
Copilot AI balanced review requested due to automatic review settings September 30, 2026 17:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The version-specific expectations match the generated field order, and relevant C++ CI checks pass.

Review effort: Balanced
Findings: None

What changed in this PR

Updates Parquet reader test expectations for simdjson 5.0 formatting while preserving compatibility with 4.x.

Changes:

  • Selects expected JSON formatting by simdjson major version.
  • Updates four affected JSON-output tests.
File Description
cpp/​src/​parquet/​reader_test.cc Adds simdjson 5.x expected output variants.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@kou

kou commented Sep 30, 2026

Copy link
Copy Markdown
Member

I think that we can skip these tests with simdjson < 5 to reduce maintenance costs.
I think that testing only with simdjson > 5.0 is enough.

@pitrou

pitrou commented Oct 1, 2026

Copy link
Copy Markdown
Member

Or perhaps we can make those tests whitespace-insensitive?

Copilot AI balanced review requested due to automatic review settings October 1, 2026 07:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Bundled simdjson 4.6.4 builds now skip four tests instead of retaining their existing expectations.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)

Comment on lines +1128 to +1130
if constexpr (simdjson::SIMDJSON_VERSION_MAJOR < 5) {
GTEST_SKIP() << "Test requires simdjson >= 5";
}
@tadeja

tadeja commented Oct 1, 2026 •

Copy link
Copy Markdown
Member Author

Alright, I've reworked this to skip four JSONOutput tests when simdjson < 5 per @kou's request as I agree that might make most sense later (but might seem premature now that bundled simdjson is still 4.6.4)

@pitrou as mentioned on the issue, I have a demo by 🤖 on my fork tadeja#2 (that checks only the content... but might be radical :))

@raulcd

raulcd commented Oct 1, 2026

Copy link
Copy Markdown
Member

pitrou as mentioned on the issue, I have a demo by 🤖 on my fork tadeja#2 (that checks only the content... but might be radical :))

To be honest the diff on that one is smaller than the one here 🤣

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am happy with this fix (and improving the validation to skip formatting differences on a different PR) to solve the current failures for the release.

@github-actions github-actions Bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Oct 1, 2026

@kou kou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@kou
kou merged commit ba6c5fe into apache:main Oct 1, 2026
59 checks passed
@kou kou removed the awaiting merge Awaiting merge label Oct 1, 2026
@github-actions github-actions Bot added the awaiting merge Awaiting merge label Oct 1, 2026
raulcd pushed a commit that referenced this pull request Oct 2, 2026
…eader-test for simdjson 5.0 (#51661)

### Rationale for this change
See #51638 - `fractured_json` formatting changes with the new simdjson version 5.0 (whitespace, line breaks, key order in tables),
cause `parquet-reader-test` to fail on four of its tests `TestJSONWithLocalFile.JSONOutput...`

### What changes are included in this PR?
Check `simdjson::SIMDJSON_VERSION_MAJOR`, and for `>= 5` use the new expected formatting (JSON content hasn't changed).
Keep existing expected output formatting for simdjson 4.x

### Are these changes tested?
Yes, CI runs pass.

### Are there any user-facing changes?
No, only test changes.

### Was AI used for this PR?

**PR code and description written by:**

- [x] Human
- [x] AI

**Reviewed before submission by:**

- [x] Human
- [ ] AI
- [ ] Not reviewed

* GitHub Issue: #51638

Authored-by: Tadeja Kadunc <tadeja.kadunc@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants