The manner in which I utilize bench-node particularly amplifies this effect,
actual
Summary (vs. baseline):
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ latest (baseline)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ current (1.04x faster)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ trunk (1.04x faster)
registry ⇒ metrics() 6 x 2 ⇒ latest x 2,530 ops/sec (41 runs sampled) min..max=(204.50us...429.13us)
registry ⇒ metrics() 6 x 2 ⇒ trunk x 2,630 ops/sec (41 runs sampled) min..max=(199.25us...400.58us)
registry ⇒ metrics() 6 x 2 ⇒ current x 2,598 ops/sec (41 runs sampled) min..max=(207.04us...428.31us)
These tests were scheduled in [latest, trunk, current] order. As you can see, one of the sets of output doesn't preserve this order, making reading comprehension a challenge, particularly during a CI run where scrolling is happening. This behavior is built into the text reporter, which sorts the results by runtime. From the git history I suspect intentionally, but I believe this is an ergonomics flaw in the code.
When you're running one test this isn't much of a problem. When you're running (k x n) tests and comparing each k set of results to each other instead of the other more than the other n results (a little of both happens of course), then it becomes very difficult to scan. Misreading leads to taking the wrong corrective actions, or no action at all. And in a CI context that is problematic.
Preferred:
Summary (vs. baseline):
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ latest (baseline)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ trunk (1.04x faster)
registry ⇒ metrics() 2 x 8 and openMetrics ⇒ current (1.04x faster)
The simple examples don't bear this out because the number of results in any single one is 2-3. But when you're looking at 180 batches of 3-5 tests each, then operator error becomes a substantial problem, and any features of the code that hamper reading comprehension begin to look more like bugs instead.
The manner in which I utilize bench-node particularly amplifies this effect,
actual
These tests were scheduled in [latest, trunk, current] order. As you can see, one of the sets of output doesn't preserve this order, making reading comprehension a challenge, particularly during a CI run where scrolling is happening. This behavior is built into the text reporter, which sorts the results by runtime. From the git history I suspect intentionally, but I believe this is an ergonomics flaw in the code.
When you're running one test this isn't much of a problem. When you're running (k x n) tests and comparing each k set of results to each other instead of the other more than the other n results (a little of both happens of course), then it becomes very difficult to scan. Misreading leads to taking the wrong corrective actions, or no action at all. And in a CI context that is problematic.
Preferred:
The simple examples don't bear this out because the number of results in any single one is 2-3. But when you're looking at 180 batches of 3-5 tests each, then operator error becomes a substantial problem, and any features of the code that hamper reading comprehension begin to look more like bugs instead.