DataMill: Rigorous Performance Evaluation Made Easy

DataMill is a community-based easy-to-use services-oriented open benchmarking infrastructure for performance evaluation.

Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate this dilemma, especially with the given time pressure of producing results. So how can one trust any reported empirical analysis of a new idea or concept in computer science?

DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors.

Go an give it a try: DataMill.

Project members: 
Last updated: July 18, 2014

Related Publications

Oliveira, A., S. Fischmeister, A. Diwan, M. Hauswirth, and P. Sweeney, "Perphecy: Performance Regression Test Selection Made Simple but Effective", Proc. of the 10th IEEE International Conference on Software Testing, Verification and Validation (ICST), Tokyo, Japan, 2017. PDF icon [pdf] (252.6 KB)
Valov, P., J-C. Petkovich, J. Guo, S. Fischmeister, and K. Czarnecki, "Transferring Performance Prediction Models Across Different Hardware Platforms", International Conference on Performance Engineering (ICPE), L'Aquila, Italy, 2017. PDF icon [pdf] (821.37 KB)
Oliveira, A., J-C. Petkovich, T. Reidemeister, and S. Fischmeister, "DataMill: Rigorous Performance Evaluation Made Easy", Proc. of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE), Prague, Czech Republic, pp. 137--149, April, 2013. PDF icon [paper] (758.5 KB)
Oliveira, A., S. Fischmeister, A. Diwan, M. Hauswirth, and P. Sweeney, "Why You Should Care About Quantile Regression", Proceedings of the Eighteenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Houston, USA, March, 2013. PDF icon [paper] (692.97 KB)
Petkovich, J-C., A. Oliveira, Y. Zhang, T. Reidemeister, and S. Fischmeister, "DataMill: a distributed heterogeneous infrastructure for robust experimentation", Software Practice and Experience, vol. 46, issue 10, pp. 29, 2016.
Blackburn, S. M., A. Diwan, M. Hauswirth, P. Sweeney, J. Nelson Amaral, T. Brecht, L. Bulej, C. Click, L. Eeckhout, S. Fischmeister, et al., "The Truth, The Whole Truth, and Nothing But the Truth: A Pragmatic Guide to Assessing Empirical Evaluations", ACM Transactions on Programming Languages and Systems (TOPLAS), vol. 38, issue 4, pp. 1-15, 2016.