webgear

	lower bound	estimate	upper bound
OLS regression	5.03 ms	5.12 ms	5.23 ms
R² goodness-of-fit	0.997	0.998	0.999
Mean execution time	5.04 ms	5.08 ms	5.13 ms
Standard deviation	116 μs	153 μs	198 μs

Outlying measurements have a moderate (17.4%) effect on estimated standard deviation.

servant

	lower bound	estimate	upper bound
OLS regression	11.0 ms	11.2 ms	11.3 ms
R² goodness-of-fit	0.998	0.999	0.999
Mean execution time	11.1 ms	11.2 ms	11.3 ms
Standard deviation	218 μs	382 μs	653 μs

Outlying measurements have a moderate (16.5%) effect on estimated standard deviation.

scotty

	lower bound	estimate	upper bound
OLS regression	14.0 ms	14.2 ms	14.5 ms
R² goodness-of-fit	0.994	0.997	0.999
Mean execution time	13.9 ms	14.1 ms	14.2 ms
Standard deviation	346 μs	452 μs	579 μs

Outlying measurements have a moderate (13.6%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.

The chart on the right is the raw data from which the kernel density estimate is built. The x-axis indicates the number of loop iterations, while the y-axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression estimate of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line. The transparent area behind it shows the confidence interval for the execution time estimate.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.

R²; goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R²; should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.

Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.

criterion performance measurements

overview

webgear

servant

scotty