add fineweb-edu dataset as calibration bakeup

[WeiweiZhang1]

Description

Please briefly describe your main changes, the motivation.

Type of Change

Bug fix

Related Issues

Fixes or relates to #

Checklist Before Submitting

• My code has been tested locally.
• Documentation has been updated as needed.
• New or updated tests are included where applicable.
• The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

[AutoRoundBot]

CI Failure Analysis Report (Unit-Test-XPU-AutoRound)

• Commit: 84aa488
• Run: https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=75593
• Totals: 1 failed test(s), 1 cluster(s) (0 known / 1 new)

New Issues (1)

No.1 — 1 occurrence(s)

🔍 importlib.metadata.PackageNotFoundError: No package metadata was found for modelscope

📝 Basic info

• Affected tests (1): TestFineWebEduDataset.test_alias_loads_modelscope_sample_when_enabled
• Logs (1): unittest_common_utils.log

🖥️ Log excerpt

return Distribution.from_name(distribution_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/home/hostuser/.local/share/uv/python/cpython-3.14.7-linux-x86_64-gnu/lib/python3.14/importlib/metadata/__init__.py:409: in from_name
    raise PackageNotFoundError(name)
E   importlib.metadata.PackageNotFoundError: No package metadata was found for modelscope
During handling of the above exception, another exception occurred:
unit/common/utils/test_calib_dataset_helpers.py:123: in test_alias_loads_modelscope_sample_when_enabled
    result = calib_dataset.get_fineweb_edu_dataset(MagicMock(), 128)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../auto_round/calib_dataset.py:295: in get_fineweb_edu_dataset
    require_version(
/home/hostuser/.venv/lib/python3.14/site-packages/transformers/utils/versions.py:103: in require_version
    raise importlib.metadata.PackageNotFoundError(
E   importlib.metadata.PackageNotFoundError: No package metadata was found for The 'modelscope' distribution was not found and is required by this application.
E   Loading FineWeb-Edu from ModelScope requires `modelscope`; install it with `pip install modelscope`.

✨ AI analysis

• Category: Test Case Issue

• Confidence: high

• Root cause: The test injects an in-memory modelscope module, but the changed implementation first calls transformers.utils.versions.require_version, which checks installed distribution metadata. Since the optional modelscope distribution is absent in this test environment, execution fails before the mocked MsDataset is used.

• Suggested fix: Mock transformers.utils.versions.require_version in this isolated routing test so it validates ModelScope dataset selection without requiring the optional package installation.

• Patch:

  --- a/test/unit/common/utils/test_calib_dataset_helpers.py
  +++ b/test/unit/common/utils/test_calib_dataset_helpers.py
  @@ -118,6 +118,7 @@ class TestFineWebEduDataset:
           modelscope.MsDataset = types.SimpleNamespace(load=load)
           monkeypatch.setitem(sys.modules, "modelscope", modelscope)
           monkeypatch.setattr(calib_dataset.envs, "AR_USE_MODELSCOPE", True)
  +        monkeypatch.setattr("transformers.utils.versions.require_version", MagicMock())
   
           result = calib_dataset.get_fineweb_edu_dataset(MagicMock(), 128)
   

Notes

• Only top-3 issues receive AI analysis.
• To request an additional fix from Copilot, use "Quote reply" on the PR comment and @mention Copilot.

[AutoRoundBot]

/azp run Unit-Test-CUDA-AutoRound

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).