Fix transformers_modeling_backend for pretrained dense models

[HosseinKaviani-H]

Fix for #3775

Problem

Loading a real pretrained dense model (e.g. Qwen3-0.6B) into the HF backend produced near-random loss, and on recent PyTorch the backend crashed before training. The HF safetensors load correctly (verified byte-for-byte): the bugs are in how the backend builds/initializes the model, not in loading.

With the same Qwen3-0.6B checkpoint + data, step-1 loss was 11.08 (≈ ln(vocab), i.e. random) vs native TorchTitan qwen3's 3.63.

Root causes fixed

1. RoPE inv_freq left uninitialized after meta-device init. HF rotary modules compute inv_freq in init; meta init + to_empty()
   zeros it and init_weights only re-inits parameters, silently disabling RoPE. Recompute it from each rotary module's rope_init_fn in
   init_states.
2. TitanDenseModelConfig overrode the HF config. Non-None defaults (dim=4096, n_layers=32, n_heads=32, norm_eps=1e-5,
   rope_theta=10000) were injected over AutoConfig values, forcing the wrong architecture (e.g. rope_theta 1e6 → 10000 for Qwen3).
   Default these to None; set explicitly only to intentionally override (e.g. debugmodel).
3. Tied embeddings crashed FSDP2. tok_embeddings/lm_head share a parameter under tie_word_embeddings but were in separate fully_shard
   groups (forbidden by recent FSDP2). Group tied tok_embeddings/norm/lm_head into one unit.
4. head_dim/intermediate_size derived instead of read from the HF config. update_from_config set head_dim = hidden_size // num_heads,
   overriding Qwen3's explicit head_dim=128 with 64 and mismatching checkpoint shapes. Only derive when dim is explicitly overridden;
   otherwise keep the HF config's values.
5. No state-dict adapter. model_registry set state_dict_adapter=None, so HF weight names were never mapped and initial_load_in_hf
   loaded nothing. Add HFTransformerStateDictAdapter (model. prefix + tied-embedding handling).

Verification

Loading Qwen3-0.6B into the full config, same checkpoint/data/seed as native (--debug.seed 42 --debug.deterministic):

• Forward parity (step 1, on loaded weights): native 3.631 vs HF 3.623 — identical within bf16 tolerance (was 11.08 before).
• Training parity (matched LR 3e-4): the curves track step-for-step with no instability:

step	native	HF
1	3.63	3.62
2	4.06	4.14
5	4.08	4.28

• (Residual drift is expected — two different model implementations diverge as updates accumulate.)
• The backend previously crashed at the FSDP tied-param check on any tied model (incl. debugmodel); it now trains.

[pytorch-bot]

Workflows were awaiting approval. CI has now been triggered for the ciflow labels on this PR.

[acisseJZhong]

Forward parity (step 1, on loaded weights): native 3.631 vs HF 3.623 — identical within bf16 tolerance (was 11.08 before).

can you also compare the kl divergence between titan logits and HF logits after the first step? previously @shuhuayu compared titan model vs HF model and the logits difference is within 1e-7.

[shuhuayu]

Forward parity (step 1, on loaded weights): native 3.631 vs HF 3.623 — identical within bf16 tolerance (was 11.08 before).

can you also compare the kl divergence between titan logits and HF logits after the first step? previously @shuhuayu compared titan model vs HF model and the logits difference is within 1e-7.

yeah, you can try a numerical test similar to scripts/checkpoint_conversion/numerical_tests_qwen3_5.py to check the logits divergence to have more confidence in the numerics.

[HosseinKaviani-H]

@acisseJZhong @shuhuayu Ran a logit-divergence test (let me know if you want me to push the script as well) -backend vs plain HF from_pretrained on Qwen3-0.6B:

• Full sequence: KL ≈ 9e-7, cosine = 1.0, top-1 = 100%
• Last token: KL ≈ 9e-8, max_diff ≈ 2.6e-5

So the backend logits match HF within ~1e-7, in line with @shuhuayu's earlier result.

[shuhuayu]

@acisseJZhong @shuhuayu Ran a logit-divergence test (let me know if you want me to push the script as well) -backend vs plain HF from_pretrained on Qwen3-0.6B:

• Full sequence: KL ≈ 9e-7, cosine = 1.0, top-1 = 100%
• Last token: KL ≈ 9e-8, max_diff ≈ 2.6e-5

So the backend logits match HF within ~1e-7, in line with @shuhuayu's earlier result.

thanks. i think the script will be very similar to the existing example so it's fine to not include it.