It would be useful if test runs that encountered statistical anomalies were re-run automatically. As the test suite count increases, the probability that any one suite encounters an inconclusive result grows asymptotically to 1. It's therefore insufficient to simply run the entire suite over again until no results are inconclusive. It's also extremely time consuming, given the way bench-node is choosing to apply t-test. It could be minutes per run and doing 5 runs could make someone a bit hostile, or worse, neglectful.
If you don't want to do this here, and soon, I may try to implement this myself in faceoff. Because I suspect the right thing to do is to analyze between suites rather than collecting the entire results at the end before looking at them. I don't know how hard that will be to plumb into bench-node. It'll be annoying in faceoff but I think there's a spot that makes sense.
It would be useful if test runs that encountered statistical anomalies were re-run automatically. As the test suite count increases, the probability that any one suite encounters an inconclusive result grows asymptotically to 1. It's therefore insufficient to simply run the entire suite over again until no results are inconclusive. It's also extremely time consuming, given the way bench-node is choosing to apply t-test. It could be minutes per run and doing 5 runs could make someone a bit hostile, or worse, neglectful.
If you don't want to do this here, and soon, I may try to implement this myself in
faceoff. Because I suspect the right thing to do is to analyze between suites rather than collecting the entire results at the end before looking at them. I don't know how hard that will be to plumb into bench-node. It'll be annoying infaceoffbut I think there's a spot that makes sense.