Performance-testing research earns ACM Distinguished Paper Award

A paper co-authored by electrical and computer engineering professor Weiyi (Ian) Shang has received an ACM Distinguished Paper Award at the 2026 IEEE/ACM International Conference on Automated Software Engineering (ASE), a flagship conference in software engineering.

Titled “Stop When It Matters: Detectability-Guided Microbenchmarking for Performance Regression Testing,” the paper resulted from a collaboration among institutions in Canada, Europe and China.

The research tackles a practical challenge in performance testing. Developers run microbenchmarks to catch software slowdowns, but they often do not know how long those benchmarks need to run. Existing approaches usually stop when results appear stable or after reaching a fixed time limit. The paper demonstrates that stability alone is not enough: a benchmark can appear stable while still being unable to reliably detect a real performance regression.

The researchers found that benchmark execution should be guided by whether a meaningful slowdown can actually be detected, rather than simply by whether measurements appear stable. Their proposed method, TT-MDE, uses minimum detectable effect reasoning to determine whether a benchmark should stop, continue running or end because the target slowdown cannot be reliably detected within the available time.

The approach could make performance regression testing less costly and more trustworthy in continuous integration (CI) environments. Teams could spend less time and computing power on benchmarks that have already gathered enough evidence, avoid wasting resources on benchmarks that cannot achieve the required precision and receive clearer signals about whether a slowdown is meaningful.

The method also provides teams with a more explicit engineering control. Developers can define which slowdowns are negligible and which require action, allowing the benchmark system to adapt around those thresholds.