How Much Does Memory Layout Impact Performance? A Wide Study

Memory layout variations are often cited as a threat to the reproducibility of computer performance experiments. Previous research has shown worst-case effects as large as 300% of execution time, among other surprising cases. While these are worrying results, how frequent and widespread are these large effects?  To answer this question, this paper presents a wide-scale evaluation of memory layout effects on the performance of a large subset of the SPECCPU2006 benchmark suite on a wide array of diverse machines. We find that on average, these benchmarks are not as susceptible to memory layout effects as the worst-case analysis found in the literature suggests. Finally, we re-execute our own experiment and demonstrate why the reproducibility of an experiment's result should not depend on its data being perfectly reproducible.