Swap the Stage 2 backbone to DINOv3-ConvNeXt (there is no ConvNeXt V3) - size-matched pretraining comparison, licence-gated

[jonfroehlich]

Split out of #20, which reserved a slot for "newer ViT/DINO-family backbones" but never named a candidate. Same pattern as #39. Full reasoning in #20 (comment); this issue is the schedulable version.

First, the negative result: there is no ConvNeXt V3

Checked 2026-08-04. ConvNeXt V2 (CVPR 2023) is still the last entry in the line — what has appeared since are task-specific variants (E-ConvNeXt and friends), not a successor backbone. Recorded here so the search isn't repeated.

The candidate

DINOv3 released ConvNeXt-architecture backbones alongside its ViTs (tiny/small/base/large), distilled from DINOv3 ViT-7B on LVD-1689M. Available in timm (DINOv3 support from 1.0.20; ConvNeXt distillation configs released 2025-11-20).

	current	candidate
timm id	convnextv2_base.fcmae_ft_in22k_in1k_384	convnext_base.dinov3_lvd1689m
params	~88 M	87.6 M
architecture	ConvNeXt V2 (GRN)	ConvNeXt V1 (no GRN)
pretraining	FCMAE → IN-22k → IN-1k supervised ft	DINOv3 SSL on 1.69 B web images, distilled from ViT-7B
licence	permissive	⚠️ DINOv3 Licence (custom Meta)

Why this backbone, and why not the DINOv3 ViTs

1. Size parity — 87.6 M vs ~88 M. Same head, same budget, ~same capacity, so the swap changes pretraining and little else. That isolation is precisely what Attribute any RampNet-vs-YOLO gap: backbone, output representation, and training budget are confounded in #51 #95 says is missing from the RampNet-vs-YOLO comparison, where backbone / output representation / training budget are all confounded at once.
2. Dense features are DINOv3's actual claim. Its headline is frozen-feature SOTA on segmentation, depth, and detection. KeypointModel consumes the feature map (backbone.children()[:-2]), not a pooled embedding, so the advertised strength is the one we would be buying rather than an incidental benchmark number.
3. At our input size it has to be a ConvNet. Stage 2 feeds 2048×4096 equirectangular panos into a backbone pretrained at 384. Only fully-convolutional backbones tolerate that gracefully — the DINOv3 ViTs are not a realistic drop-in here. This is the specific reason to take the ConvNeXt member of the suite rather than the headline model.

⚠️ Licence gate — resolve before spending any compute

DINOv3 code and weights ship under a custom "DINOv3 License", not Apache/MIT. We publish projectsidewalk/rampnet-model on HF, so a derived checkpoint inherits whatever that licence permits for redistribution and downstream use.

Read LICENSE.md in facebookresearch/dinov3 first. If it forbids what we need, this issue closes without a GPU-hour spent — and that is the result to write down here, because it applies to every future DINOv3-derived candidate too, not just this one.

Experiment

Design is a 2×2 (backbone × frozen), because the frozen arm is only interpretable against a frozen control:

arm	backbone	backbone weights	what it answers
A0 control	convnextv2 (current)	finetuned	the honest baseline at the chosen budget — see "ordering"
A1	dinov3-convnext	finetuned	does DINOv3 pretraining beat IN-22k/IN-1k for this task?
A2	dinov3-convnext	frozen, head only	does DINOv3's frozen-feature claim hold for curb ramps?
A3	convnextv2 (current)	frozen, head only	separates "frozen features work" from "DINOv3 features work"

A0/A1 are the core comparison; A2/A3 are the cheap pair and can run second. Evaluate every arm at the #54 operating point (peak threshold 0.30, single-pass — no flip-TTA, per #78) on manual_gold plus the benchmark splits, reported per-split, not pooled — an in-domain-only gain is the interesting shape and pooling hides it.

Cost, and one thing not to assume

Per #84's measurements: 3.49 h/epoch at 16 GPUs (~56 GPU-h), ×1.67 preemption overhead on ckpt-all ⇒ roughly 28 h compute / 47 h calendar per arm at 8 epochs.

Do not expect the frozen arms (A2/A3) to be much cheaper in wall-clock. #84 measured the run as I/O-bound, not compute-bound — ~3% MFU, p25–p75 step-time spread of 6 ms over 119,902 samples. Dropping backbone gradients removes work that was not the bottleneck. Budget them as full-cost runs.

Ordering: downstream of #84

Same constraint that moved #82 behind it. At 1 epoch every pano is seen exactly once, and #84's rescued curve shows auto-label val loss still improving through epoch 5. Comparing backbones at a budget where neither has converged under-reads both — and differently-pretrained backbones are exactly the case where the benefit shows up as convergence speed, so a short schedule doesn't just add noise, it biases the comparison. Run at whatever epoch count #84 settles on.

This is also why A0 is not optional and cannot be the released checkpoint: per #84's 2026-08-04 note, the released model is a hand-picked epoch 1 that neither selection rule chose. It is a valid budget-matched control at 1 epoch and nothing more; at any other budget, comparing to it reads schedule as method.

Implementation notes

BACKBONE_NAME at rampnet/model.py:4 is a module constant, so the nominal change is one line. Three things to verify rather than assume:

• Module layout. The docstring at rampnet/model.py:19-27 warns that feature_extractor = nn.Sequential(*list(backbone.children())[:-2]) is the layout every released checkpoint was trained and saved with. timm builds V1 and V2 ConvNeXts from the same ConvNeXt class (V2 = V1 + GRN), so children() should be structurally identical and num_features the same 1024 — confirm it, don't trust a strict load to catch it.
• Preprocessing stats. rampnet/model.py:14-15 hardcodes IMAGENET_MEAN/IMAGENET_STD. Pull the mean/std from the candidate's own timm pretrained config instead of assuming they carry over — a mismatch here degrades quietly and looks like "the backbone is worse."
• LR. PRESET_LR['scratch'] = 1e-5 was chosen for finetuning this backbone. A differently-pretrained backbone may want a different LR, so an A1 loss at 1e-5 is confounded with LR choice. If A1 underperforms, that is the first thing to rule out before concluding anything about pretraining.

Also: a new backbone starts a new checkpoint lineage — existing weights will not load, and scripts/hf_package/modeling_rampnet.py (the verbatim copy synced by the exporter) must stay in step or published-model loading breaks.

Definition of done

• Licence verdict written into this issue, whichever way it goes.
• If cleared: A0/A1 per-split tables in docs/, with the epoch budget and the LR caveat stated next to the numbers.
• If A1 wins, a note in Attribute any RampNet-vs-YOLO gap: backbone, output representation, and training budget are confounded in #51 #95 that one of its three confounds now has a measurement.

🤖 Generated with Claude Code (claude-opus-5[1m])