I was testing Quantile Skill Score in the reports and came across some results that initially confused me. Table below shows metrics for 4 pairs of forecasts
| Forecast |
BS |
QS |
BSS |
QSS |
| Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 0.0% |
0.46 |
46.3 |
-0.113 |
0.653 |
| Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 40.0% |
0.282 |
39.6 |
-0.161 |
0.688 |
| Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 50.0% |
0.25 |
44.7 |
0.00e+00 |
0.643 |
| Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 100.0% |
0.264 |
93.4 |
0.55 |
0.201 |
| Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 0.0% |
0.415 |
129 |
nan |
nan |
| Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 40.0% |
0.243 |
122 |
nan |
nan |
| Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 50.0% |
0.25 |
121 |
nan |
nan |
| Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 100.0% |
0.585 |
112 |
nan |
nan |
I came away from this wanting a few things in the brier score documentation
- a statement about how the Arbiter computes o for non-event forecasts (is f <= x?).
- some discussion about brier score vs. quantile score for quantile forecasts (like all of our reference forecasts as of today).
I was testing Quantile Skill Score in the reports and came across some results that initially confused me. Table below shows metrics for 4 pairs of forecasts
I came away from this wanting a few things in the brier score documentation