criterion performance measurements

overview

want to understand this report?

pyth/ndet/mtl

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 3.1296677394245935e-4 3.131659393506705e-4 3.138248689171451e-4
Standard deviation 4.4905954592977045e-7 1.082045292054602e-6 2.394427081755867e-6

Outlying measurements have slight (1.0525124490719781e-2%) effect on estimated standard deviation.

pyth/ndet/eff

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 5.765862115382986e-2 5.7724624861021166e-2 5.7831047269477104e-2
Standard deviation 6.674830132101631e-5 1.4695388253674097e-4 1.9874819288508882e-4

Outlying measurements have slight (7.638888888888888e-2%) effect on estimated standard deviation.

pyth/ndet : st/mtl

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.16786697004760126 0.17411292817007443 0.1791173460714693
Standard deviation 6.200043968794704e-3 8.226276133458112e-3 1.1341173659211225e-2

Outlying measurements have moderate (0.12245003885656061%) effect on estimated standard deviation.

pyth/ndet : st/eff

lower bound estimate upper bound
OLS regression xxx xxx xxx
R² goodness-of-fit xxx xxx xxx
Mean execution time 0.16963141483944053 0.16983360530204153 0.17039271496191327
Standard deviation 7.932014476160957e-5 4.731140387192827e-4 7.079808580216453e-4

Outlying measurements have moderate (0.12244897959183669%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.