DataMill: Rigorous Performance Evaluation Made Easy

TitleDataMill: Rigorous Performance Evaluation Made Easy
Publication TypeConference Paper
Year of Publication2013
AuthorsOliveira, A., J-C. Petkovich, T. Reidemeister, and S. Fischmeister
Conference NameProc. of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE)
Date PublishedApril
Conference LocationPrague, Czech Republic
Keywordsbenchmarking, empirical evaluation, systems

Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate this dilemma, especially with the given time pressure of producing results. So how can one trust any reported empirical analysis of a new idea or concept in computer science?

This paper introduces DataMill, a community-based easy-to-use services-oriented open benchmarking infrastructure for performance evaluation. DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors. Multiple research groups already participate in DataMill.

DataMill is also of interest for research on performance evaluation. The infrastructure supports identifying new hidden factors, disseminating the research results beyond mere reporting. It provides a platform for investigating interactions and composition of hidden factors.

Refereed DesignationRefereed
Related files: 


Looking for motivated students (undergrads and grads) interested in working on embedded software and systems research. Mail Sebastian Fischmeister for further information.