More benchmark

[dalonsoa]

Description

Proof of concept of conditional compilation to use unstable features during benchmarking. For reference, check:

• https://doc.rust-lang.org/reference/conditional-compilation.html#the-cfg_attr-attribute
• https://doc.rust-lang.org/1.12.1/book/benchmark-tests.html

Ignore the whole diff - I do not know why it is not picking it right. Just check 71fa007

When running cargo test or any other normal command, it will drop any code related to the feature = bench, and hence it will work fine.

When running cargo bench --features bench, the code will not be dropped, and compilation will fail, because the stable version of rust does not support feature(test) and all the bench stuff.

To make it run, we need to indicate we are using the nightly version of the compiler, using cargo +nightly bench --features bench. With this, it will run the benchmarks we had before, within the bench directory and using all the features of Criterion, as well as the much simpler #[bench] benchmarks. For the example I've added, it will just print:

test finance::bench::bench_add_two ... bench:           0.21 ns/iter (+/- 0.01)

Among all the other output. No graphs or comparison with previous runs, but might be enough for many cases.

Note that the benchmark code is isolated in its own module, not within mod tests like in the example, such that it can be conditionally excluded/included during compilation.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change to fix an issue)
• New feature (non-breaking change to add functionality)
• Refactoring (non-breaking, non-functional change to improve maintainability)
• Optimization (non-breaking change to speed up the code)
• Breaking change (whatever its nature)
• Documentation (improve or add documentation)

Key checklist

• All tests pass: $ cargo test
• The documentation builds and looks OK: $ cargo doc
• Update release notes for the latest release if this PR adds a new feature or fixes a bug
  present in the previous release

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[tsmbland]

This looks like it could be useful (at least for some things). Two comments:

• I get some warnings about unused imports (even though the imports are clearly used)
• I think lack of graphs isn't ideal in this particular case, as it's all about scaling, which we want a clear way to visualise. I liked the comparison feature with Criterion as well. It might be worth just biting the bullet on this and sticking with the previous approach - if there's no perfect solution them making a few things public for the sake of useful benchmarking is probably a worthwhile trade-off, so long as we don't get carried away with it.

[dalonsoa]

This approach is called micro-benchmarking for a reason, so clearly it won't offer all the features of Criterion. It might be useful for some things, clearly, but I do agree that in this case, the comparison with previous rounds is a strong reason for not using it.

I've no idea why the warnings, to be honest. I also noticed it.