[wip] switch to transformers main again. #12976
Draft
+370
−190
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sayakpaul
commented
Jan 14, 2026
| - "tests/pipelines/test_pipelines_common.py" | ||
| - "tests/models/test_modeling_common.py" | ||
| - "examples/**/*.py" | ||
| - ".github/**.yml" |
Member
Author
There was a problem hiding this comment.
Temporary. For this PR.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
sayakpaul
changed the title
sayakpaul
changed the title
sayakpaul
commented
Jan 16, 2026
| logger.addHandler(stream_handler) | ||
|
|
||
|
|
||
| @unittest.skipIf(is_transformers_version(">=", "4.57.5"), "Size mismatch") |
Member
Author
There was a problem hiding this comment.
Internal discussion: https://huggingface.slack.com/archives/C014N4749J9/p1768474502541349
sayakpaul
commented
Jan 16, 2026
| torch.nn.ConvTranspose2d, | ||
| torch.nn.ConvTranspose3d, | ||
| torch.nn.Linear, | ||
| torch.nn.Embedding, |
Member
Author
There was a problem hiding this comment.
Happening because of the way weight loading is done in v5.
sayakpaul
commented
Jan 16, 2026
Comment on lines
+23
to
+25
| model = AutoModel.from_pretrained( | ||
| "hf-internal-testing/tiny-stable-diffusion-torch", subfolder="text_encoder", use_safetensors=False | ||
| ) |
Member
Author
There was a problem hiding this comment.
Internal discussion: https://huggingface.slack.com/archives/C014N4749J9/p1768462040821759
sayakpaul
commented
Jan 16, 2026
Comment on lines
+281
to
+283
| input_ids = ( | ||
| input_ids["input_ids"] if not isinstance(input_ids, list) and "input_ids" in input_ids else input_ids | ||
| ) |
Member
Author
There was a problem hiding this comment.
Internal discussion https://huggingface.slack.com/archives/C014N4749J9/p1768537424692669
sayakpaul
commented
Jan 16, 2026
| inputs = { | ||
| "prompt": "dance monkey", | ||
| "negative_prompt": "", | ||
| "negative_prompt": "bad", |
Member
Author
There was a problem hiding this comment.
Otherwise, the corresponding tokenizer outputs:
negative_prompt=[' ']
prompt=[' ']
text_input_ids=tensor([], size=(1, 0), dtype=torch.int64)which leads to:
E RuntimeError: cannot reshape tensor of 0 elements into shape [1, 0, -1, 8] because the unspecified dimension size -1 can be any value and is ambiguous
sayakpaul
commented
Jan 16, 2026
Comment on lines
+569
to
+580
| if is_transformers_version("<=", "4.58.0"): | ||
| token = tokenizer._added_tokens_decoder[token_id] | ||
| tokenizer._added_tokens_decoder[last_special_token_id + key_id] = token | ||
| del tokenizer._added_tokens_decoder[token_id] | ||
| elif is_transformers_version(">", "4.58.0"): | ||
| token = tokenizer.added_tokens_decoder[token_id] | ||
| tokenizer.added_tokens_decoder[last_special_token_id + key_id] = token | ||
| del tokenizer.added_tokens_decoder[token_id] | ||
| if is_transformers_version("<=", "4.58.0"): | ||
| tokenizer._added_tokens_encoder[token.content] = last_special_token_id + key_id | ||
| elif is_transformers_version(">", "4.58.0"): | ||
| tokenizer.added_tokens_encoder[token.content] = last_special_token_id + key_id |
Member
Author
There was a problem hiding this comment.
Still doesn't solve the following issue:
FAILED tests/pipelines/test_pipelines.py::DownloadTests::test_textual_inversion_unload - AttributeError: CLIPTokenizer has no attribute _added_tokens_decoder. Did you mean: 'added_tokens_decoder'?Internal thread: https://huggingface.slack.com/archives/C014N4749J9/p1768536480412119
Member
Author
|
@DN6 I have fixed a majority of the issues that were being caused by v5 / |
sayakpaul
commented
Jan 19, 2026
Comment on lines
+94
to
+95
| config = AutoConfig.from_pretrained("hf-internal-testing/tiny-random-t5") | ||
| text_encoder_2 = T5EncoderModel(config) |
Member
Author
There was a problem hiding this comment.
Member
Author
|
A bunch of tests actually fail on Diffusers UnravelFAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_cfg - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_attention_slicing_forward_pass - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_callback_inputs - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_dict_tuple_outputs_equivalent - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_inference_batch_single_identical - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_pipeline_with_accelerator_device_map - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_save_load_float16 - AssertionError: False is not true : `safety_checker.dtype` switched from `float16` to torch.float32 after loading.
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_sequential_cpu_offload_forward_pass - AssertionError: False is not true : Not offloaded: ['safety_checker']
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_sequential_offload_forward_pass_twice - AssertionError: False is not true : Not offloaded: ['safety_checker']
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_cfg - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_attention_slicing_forward_pass - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_callback_inputs - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_dict_tuple_outputs_equivalent - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_inference_batch_single_identical - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_pipeline_with_accelerator_device_map - RuntimeError: Expected all tensors to be on the same device, but got index is on cpu, different from other tensors on cuda:0 (when che...
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_save_load_float16 - AssertionError: False is not true : `safety_checker.dtype` switched from `float16` to torch.float32 after loading.
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_sequential_cpu_offload_forward_pass - AssertionError: False is not true : Not offloaded: ['safety_checker']
FAILED tests/pipelines/cosmos/test_cosmos_video2world.py::CosmosVideoToWorldPipelineFastTests::test_sequential_offload_forward_pass_twice - AssertionError: False is not true : Not offloaded: ['safety_checker'] |
Member
Author
|
As we can see here and here, some tests fail across both To summarize things a bit, With 4.7.3, we have a total of 132 failures and 157 with main. 93 new test failures. Common: 64 Unfold to see the failures with main
Little utility to obtain these reports: Unravelimport re
import argparse
def extract_failed_tests(log_file_path):
"""
Parses a pytest log file and returns a list of fully qualified
paths for all failed tests.
"""
failed_tests = []
# This regex looks for the 'FAILED ' prefix used in pytest's
# 'short test summary info' section.
failure_pattern = re.compile(r'^FAILED\s+(.*)$')
with open(log_file_path, 'r') as file:
for line in file:
line = line.strip()
match = failure_pattern.match(line)
if match:
# The first group contains the full path:
# script_path.py::ClassName::test_name
failed_tests.append(match.group(1))
return failed_tests
if __name__ == "__main__":
path1 = "/Users/sayakpaul/Downloads/pr_pytorch_pipelines_torch_cpu_pipelines_transformers_4.57.3_test_reports/tests_torch_cpu_pipelines_summary_short_4_57_3.txt"
path2 = "/Users/sayakpaul/Downloads/pr_pytorch_pipelines_torch_cpu_pipelines_transformers_main_test_reports/tests_torch_cpu_pipelines_summary_short_main.txt"
failures_one = set(extract_failed_tests(path1))
failures_two = set(extract_failed_tests(path2))
print(f"Old: {len(failures_one)}, New: {len(failures_two)}")
difference = failures_two.difference(failures_one)
intersection = failures_two.intersection(failures_one)
print(f"{len(difference)} new test failures. Common: {len(intersection)}")
diff_list = "\n".join(f"- `{diff}`" for diff in difference)
print(diff_list)
Will work on fixing the new ones where possible. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR is to assess if we can move to
transformersmain again for our CI. This will also help us migrate totransformersv5 successfully.Notes
For the
FAILED tests/pipelines/test_pipelines.py::DownloadTests::test_textual_inversion_unload - AttributeError: CLIPTokenizer has no attribute _added_tokens_decoder. Did you mean: 'added_tokens_decoder'?error, see this internal discussion.