[BUG] Trinity-RFT v0.6.0在单卡colocate模式卡在Ray资源调度

[wuzhenqing]

Bug Description

Please provide a detailed description of the issue you encountered.
我在单张NVIDIA A100 80GB上使用Trinity-RFT v0.6.0运行GSM8K GRPO样例，配置为mode: colocate，但任务会卡在初始化阶段，始终没有rollout数据或训练step。

Environment Information

• Operating System: Ubuntu 22.04
• Python Version: 3.12.13
• GPU: 1 * NVIDIA A100-80G
• CUDA Version: 13.0
• Installation Method: pypi
• Trinity-RFT Version: 0.6.0
• TRL 1.6.0
• Model：Qwen2.5-1.5B-Instruct

Steps to Reproduce

当前配置文件为

project: "Trinity-RFT-gsm8k"
name: "qwen2.5-1.5B-gsm8k"
checkpoint_root_dir: /vepfs-mlp2/c20250501/240806010/Learn/checkpoints
mode: colocate
algorithm:
  algorithm_type: grpo
  repeat_times: 8
  optimizer:
    lr: 1e-5
model:
  model_path: /vepfs-mlp2/c20250501/240806010/models/Qwen/Qwen2.5-1.5B-Instruct
  max_response_tokens: 1024
  max_model_len: 2048
cluster:
  node_num: 1
  gpu_per_node: 1
buffer:
  total_epochs: 1
  batch_size: 64
  explorer_input:
    taskset:
      name: gsm8k
      storage_type: file
      path: /vepfs-mlp2/c20250501/240806010/datasets/gsm8k
      subset_name: 'main'
      split: 'train'
      format:
        prompt_key: 'question'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
    eval_tasksets:
    - name: gsm8k-eval
      storage_type: file
      path: /vepfs-mlp2/c20250501/240806010/datasets/gsm8k
      subset_name: 'main'
      split: 'test'
      format:
        prompt_key: 'question'
        response_key: 'answer'
    default_workflow_type: 'math_workflow'
  trainer_input:
    experience_buffer:
      name: gsm8k_buffer
      storage_type: queue
      path: 'sqlite:///gsm8k.db'
explorer:
  eval_interval: 50
  runner_per_model: 8
  rollout_model:
    engine_num: 1
    tensor_parallel_size: 1
    enable_prefix_caching: false
    enforce_eager: true
    dtype: bfloat16
    seed: 42
synchronizer:
  sync_method: 'memory'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  trainer_type: 'verl'
  save_interval: 100
  grad_clip: 1.0
  use_dynamic_bsz: true
  max_token_len_per_gpu: 16384
  ulysses_sequence_parallel_size: 1
# stages:  # Uncomment to add a SFT warmup stage before RFT
#   - stage_name: sft_warmup
#     mode: train
#     algorithm:
#       algorithm_type: sft
#     buffer:
#       train_batch_size: 128
#       total_steps: 10
#       trainer_input:
#         experience_buffer:
#           name: sft_warmup_dataset
#           storage_type: file
#           path: ${oc.env:TRINITY_SFT_DATASET_PATH}
#           format:
#             prompt_type: messages
#             messages_key: 'messages'
#   - stage_name: rft  # leave empty to use the original configs for RFT

Expected Behavior

期望可以在单卡完成样例的训练

Actual Behavior

Trainer已显示Trainer is ready，但Ray一直存在等待中的GPU请求，经验队列始终为0，未创建rollout进程，也没有训练指标或checkpoint。

Total Usage:
  0.3333/1.0 GPU (0.3333 used of 1.0 reserved in placement groups)

Pending Demands:
  {'CPU': 1.0, 'GPU': 1.0} * 1 (PACK): 1+ pending placement groups

初步对比发现，v0.5.0的colocate路径会将rollout actor设为num_gpus=0并通过CUDA_VISIBLE_DEVICES=0与Trainer共享GPU；而v0.6.0的Allocator看起来会为rollout额外创建GPU:1的placement group，导致单卡上永久等待。请问这是v0.6.0的已知回归吗？单卡colocate模式是否有推荐的修复方案？

Log Information

If applicable, include any relevant log output here.

Are You Willing to Fix This Issue?

• Yes, I am willing to fix this issue!

[pan-x-c]

感谢反馈，可尝试使用 #603 中的代码看看问题是否仍然存在