Skip to content

Calibrate detector defaults using benchmark sweep data #559

Closed
Closed
Calibrate detector defaults using benchmark sweep data#559
Labels
improvement
Milestone
0.7.2

Description

@Breakthrough
Breakthrough
opened on Jul 18, 2026

It might be worth revisiting detector defaults on a more systematic basis. A full parameter sweep of all detectors across BBC Planet Earth, AutoShot, and ClipShots is now complete (see benchmarks page):

Detector v0.7 default Swept best Mean F1 (tol=1) Proposal
HistogramDetector threshold=0.05, bins=256 threshold=0.20, bins=128 54.3 -> 66.3 change
HashDetector threshold=0.395, size=16 threshold=0.35, size=8 65.8 -> 69.8 change
ContentDetector threshold=27 threshold=31 72.2 -> 73.4 keep
AdaptiveDetector threshold=3.0, window=2 threshold=3.5, window=3 74.7 -> 76.3 keep

ThresholdDetector is out of scope for these datasets, because it detects fades, not shot changes. The ordering comes from benchmark/analyze_sweep.py, which requires more than just the best average: a candidate must sit on a plateau (not a spike), beat the current default even with any one dataset removed, not trade away precision (over-splitting is the most common complaint), and clear a minimum +2.0 F1 gain to justify changing user-visible output. Full output is collapsed below.

We also may want to come up with a strategy for dealing with changing the defaults if the addition of more datasets warrants it. At minimum this can help guide on what reasonable values might be.

Rollout

  • Land in a minor release (target 0.7.2), not a patch release.
  • Changelog callouts with the old values and how to restore them (e.g. detect-hist -t 0.05 -b 256).
  • Regenerate the README / website benchmark tables afterwards.

Tasks

  • HistogramDetector default threshold=0.20, bins=128
  • HashDetector default size=8, threshold=0.35 (or 0.375)
  • docstrs / CLI help updated
  • Guidance for tuning content/adaptive
  • scripts/benchmark_defaults.sh rerun; README + website tables regenerated
  • Changelog entries with restore-old-behavior instructions

Default Recommendations from Benchmark Datasets

Generated by benchmark/analyze_sweep.py.

detect-content

Cell Mean F1@1 BBC AutoShot ClipShots Params
default 72.15 87.72 72.24 56.50 min_scene_len=0.6, threshold=27
best 73.39 85.95 71.51 62.73 min_scene_len=0.6, threshold=31

Candidates (mean F1@1 within 1 of best):

Mean Micro NbrDrop Robust WeightStable PrecFloor Material Verdict Params
73.39 72.27 0.41 yes NO yes NO - min_scene_len=0.6, threshold=31
73.37 72.76 0.50 yes NO yes NO - min_scene_len=0.6, threshold=33
73.32 72.10 1.46 yes NO yes NO - min_scene_len=0.4, threshold=33
73.15 71.36 1.00 yes NO yes NO - min_scene_len=0.6, threshold=29
73.09 71.23 1.85 yes NO yes NO - min_scene_len=0.4, threshold=31
73.04 72.19 1.15 yes NO yes NO - min_scene_len=0.4, threshold=35
72.99 72.46 0.22 yes NO yes NO - min_scene_len=0.8, threshold=31
72.92 72.65 0.61 yes NO yes NO - min_scene_len=0.6, threshold=35
72.87 72.81 0.56 yes NO yes NO - min_scene_len=0.8, threshold=33
72.77 71.71 0.80 yes NO yes NO - min_scene_len=0.8, threshold=29
72.72 70.06 2.44 NO yes yes NO - min_scene_len=0.4, threshold=29

With min_scene_len fixed at the default-equivalent 0.6s:

Mean F1@1 BBC AutoShot ClipShots Params
73.39 85.95 71.51 62.73 min_scene_len=0.6, threshold=31
73.37 84.27 70.95 64.89 min_scene_len=0.6, threshold=33
73.15 87.28 72.05 60.13 min_scene_len=0.6, threshold=29

Recommendation: KEEP current default (no candidate passes all criteria; see verdict column for which gate fails).

detect-adaptive

Cell Mean F1@1 BBC AutoShot ClipShots Params
default 74.71 92.52 75.48 56.14 adaptive_threshold=3, min_scene_len=0.6, window_width=2
best 76.34 90.41 76.27 62.32 adaptive_threshold=3.5, min_scene_len=0.6, window_width=3

Candidates (mean F1@1 within 1 of best):

Mean Micro NbrDrop Robust WeightStable PrecFloor Material Verdict Params
76.34 74.10 1.15 yes NO yes NO - adaptive_threshold=3.5, min_scene_len=0.6, window_width=3
76.21 74.56 0.75 yes NO yes NO - adaptive_threshold=4, min_scene_len=0.6, window_width=3
76.18 73.36 1.44 yes NO yes NO - adaptive_threshold=3.5, min_scene_len=0.4, window_width=3
76.16 74.06 0.72 yes NO yes NO - adaptive_threshold=4, min_scene_len=0.4, window_width=3
75.45 74.09 1.21 yes NO yes NO - adaptive_threshold=4.5, min_scene_len=0.6, window_width=3
75.44 73.73 1.16 yes NO yes NO - adaptive_threshold=4.5, min_scene_len=0.4, window_width=3

With min_scene_len fixed at the default-equivalent 0.6s:

Mean F1@1 BBC AutoShot ClipShots Params
76.34 90.41 76.27 62.32 adaptive_threshold=3.5, min_scene_len=0.6, window_width=3
76.21 87.87 76.17 64.58 adaptive_threshold=4, min_scene_len=0.6, window_width=3
75.45 85.28 75.71 65.37 adaptive_threshold=4.5, min_scene_len=0.6, window_width=3

Recommendation: KEEP current default (no candidate passes all criteria; see verdict column for which gate fails).

detect-hash

Cell Mean F1@1 BBC AutoShot ClipShots Params
default 65.78 85.01 66.82 45.50 size=16, threshold=0.4
best 69.83 86.38 68.98 54.12 size=8, threshold=0.35

Candidates (mean F1@1 within 1 of best):

Mean Micro NbrDrop Robust WeightStable PrecFloor Material Verdict Params
69.83 67.60 1.64 yes yes yes yes PASS size=8, threshold=0.35
69.63 68.78 2.62 NO yes yes yes - size=8, threshold=0.4
69.63 68.77 -0.00 yes yes yes yes PASS size=8, threshold=0.375

Recommendation: CHANGE to size=8, threshold=0.35 (+4.05 mean F1@1 vs default).

detect-hist

Cell Mean F1@1 BBC AutoShot ClipShots Params
default 54.31 81.21 61.29 20.43 bins=256, threshold=0.05
best 66.27 82.10 68.99 47.72 bins=128, threshold=0.2

Candidates (mean F1@1 within 1 of best):

Mean Micro NbrDrop Robust WeightStable PrecFloor Material Verdict Params
66.27 61.87 0.07 yes yes yes yes PASS bins=128, threshold=0.2
66.23 62.08 0.04 yes yes yes yes PASS bins=128, threshold=0.21
66.20 61.58 0.18 yes yes yes yes PASS bins=128, threshold=0.19
66.19 62.25 0.05 yes yes yes yes PASS bins=128, threshold=0.22
66.17 62.63 0.08 yes yes yes NO - bins=128, threshold=0.24
66.14 62.42 -0.02 yes yes yes NO - bins=128, threshold=0.23
66.09 62.70 0.07 yes yes yes NO - bins=128, threshold=0.25
66.02 61.11 0.03 yes yes yes yes PASS bins=128, threshold=0.18
66.01 62.73 0.13 yes yes yes NO - bins=128, threshold=0.26
65.99 60.87 0.17 yes yes yes yes PASS bins=128, threshold=0.17
65.88 62.77 0.23 yes yes yes NO - bins=128, threshold=0.27
65.83 60.46 0.05 yes yes yes yes PASS bins=128, threshold=0.16
65.77 59.99 0.13 yes yes yes yes PASS bins=128, threshold=0.15
65.65 62.69 0.23 yes yes yes NO - bins=128, threshold=0.28
65.64 59.37 0.25 yes yes yes yes PASS bins=128, threshold=0.14
65.42 62.60 0.25 yes yes yes NO - bins=128, threshold=0.29
65.39 58.63 0.42 yes yes yes yes PASS bins=128, threshold=0.13
65.28 60.49 0.08 yes yes yes yes PASS bins=256, threshold=0.21

Recommendation: CHANGE to bins=128, threshold=0.2 (+11.97 mean F1@1 vs default).

Activity

  1. Breakthrough
    added
    improvement
    on Jul 18, 2026
  2. Breakthrough
    added this to the 0.7.2 milestone on Jul 18, 2026

  3. MuhammadBilalKhan267 commented on Aug 10, 2026

    @MuhammadBilalKhan267
    MuhammadBilalKhan267
    on Aug 10, 2026
    Collaborator

    It looks like the recommended HashDetector and HistogramDetector defaults from this issue are already present on main (0.35/8 and 0.20/128 respectively), and the changelog also includes #559 entries.

    The issue checklist should to be updated to reflect what has already been completed and what work is still outstanding.

  4. Breakthrough
    closed this as completedon Oct 12, 2026
  5. Breakthrough
    added a commit that references this issue on Oct 12, 2026

    Release v0.7.2 (#579)

    7cd1005
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvement

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions