One challenge we face in summarizing end of release performance findings is how do you effectively illustrate release to release performance in a concise visual manner? One method I have been experimenting with is the use of line plots like the one to the right. This plot uses the following visualization techniques:

• x-axis represents workload type – a group of experiments that use the same API call(s),
• y-axis represents the throughput rate for a given experiment using a log scale,
• the y-axis also represents a frequency plot of the occurrence of throughput values for the release,
• each vertical stack of experiments represents a thread walk starting from the bottom up representing 1, 4, 8 and 16 threads,
• each bar connects the throughput rate for the previous release (left data point) to the throughput rate of the current release under study (right data point)
• gray bars indicate a zero or greater slope indicating no delta or an improvement in throughput,
• red bars indicate a negative slope or decrease in throughput,
• the black vertical bars are 95th percentile confidence intervals for each experiment set.

This particular plot represents the results from 160 distinct 3 hour experiments. The methods used in the plots allow you to quickly gauge the results of the entire study set, focusing on potential problem areas with the release (in red). The confidence intervals identified with the black vertical bars provide additional context to illustrate the variability of each workload type’s results for the 8 thread count study. This visual tool replaced 16 tables with all 160 values spread across a wiki page and multiple slides from an end of release summary presentation.