Skip to content

fix CI for datasets v4.7.0#1004

Closed
snimu wants to merge 2 commits intomainfrom
sebastian/fix-datasets-nonsense-2026-03-10
Closed

fix CI for datasets v4.7.0#1004
snimu wants to merge 2 commits intomainfrom
sebastian/fix-datasets-nonsense-2026-03-10

Conversation

@snimu
Copy link
Copy Markdown
Contributor

@snimu snimu commented Mar 10, 2026

Description

Starting with v4.7.0, HuggingFace datasets serialize heterogeneous content-part dicts (e.g. text + image_url) to JSON strings when storing in Arrow tables. Recover these by attempting json.loads on string content parts during message normalization.

This is a quick and dirty fix for now. #1002 could be an opportunity for a more general fix.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Test improvement

Testing

  • All existing tests pass when running uv run pytest locally.
  • New tests have been added to cover the changes

Checklist

  • My code follows the style guidelines of this project as outlined in AGENTS.md
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Note

Low Risk
Small, localized normalization change with best-effort JSON parsing; main risk is subtle behavior changes if arbitrary string content happens to be valid JSON.

Overview
Fixes message normalization for HuggingFace datasets v4.7+ where heterogeneous content part dicts may be persisted as JSON strings.

During _normalize_raw_message_content, string parts are now best-effort json.loads-decoded and only dict parts with a type field are converted via from_raw_content_part, improving robustness when reading dataset-backed messages.

Written by Cursor Bugbot for commit 5817f8c. This will update automatically on new commits. Configure here.

snimu and others added 2 commits March 10, 2026 14:52
HuggingFace datasets may serialize heterogeneous content-part dicts
(e.g. text + image_url) to JSON strings when storing in Arrow tables.
Recover these by attempting json.loads on string content parts during
message normalization.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@snimu snimu changed the title Sebastian/fix datasets nonsense 2026 03 10 fix CI for datasets v4.7.0 Mar 10, 2026
Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

try:
part = json.loads(part)
except (json.JSONDecodeError, TypeError):
pass
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

json.loads silently converts strings to non-dict types

Medium Severity

json.loads on a string content part can return non-dict types (e.g., "null"None, "123"123, "true"True, "[1,2]"list). These silently replace the original string in part, bypass the isinstance(part, dict) guard, and get appended as invalid ContentPart types. The parsed result is only useful when it's a dict, so the assignment to part needs to be guarded — e.g., only replacing part when isinstance(parsed, dict).

Fix in Cursor Fix in Web

@snimu snimu closed this Mar 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant