Please note: This master’s thesis presentation will take place in DC 2314.
Jiale (Amber) Wang, Master’s candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Pengyu Nie
Software tests must evolve with the code they check: when developers change production code, they add tests for new behavior, update tests that the change breaks, and refine tests that no longer exercise what changed. Coding agents are increasingly asked to do this work, and continuous integration collects code coverage on every revision. Yet tests are still measured one revision at a time. An evolved test is judged by whether it compiles and passes on the new code, which cannot show whether it captures the change, and coverage is recomputed from scratch even when a change affects little of the code. This thesis argues that measuring tests in evolving software should be change-aware: anchored to the pair of revisions, grounded in test execution, and scoped to the code that a change affects. We apply this principle in two independent contributions.
The main contribution is TestEvo-Bench, a benchmark of 3,020 test generation, update, and refinement tasks from 196 Java Maven repositories, each packaged with both revisions in an executable environment. Generated and updated tests must pass on the new revision but not on the old one, refined tests must gain coverage of the changed code, and passing tests are scored by coverage and mutation score against the developer’s tests. On a 584-task subset, six coding-agent configurations solve 98.3–99.1% of the tasks and reach overall scores, which combine task success with coverage and mutation scores, of 79.6–84.5%, on par with the developer’s tests (82.1%), whereas two prior test-update tools reach 35.8–39.0% on update tasks. The agents’ success depends on execution feedback and on how well the task is specified: their first patches solve only 81.7–91.6% of the tasks, and their overall score drops by up to 9.2 percentage points when the task instructions and test context are under-specified.
The second contribution, iJaCoCo, is an efficient incremental code coverage analysis that reruns only the tests needed to recompute the coverage a change may affect and reuses the rest from the previous revision. We prove it correct whenever the underlying regression test selection is safe. On 1,122 revisions of 22 open-source Java projects, iJaCoCo is 1.86× faster than JaCoCo on average and up to 8.20× faster, with no statistically significant difference in coverage.
Together, beyond a practical benchmark and technique for measuring evolving tests, this thesis develops a change-aware methodology for collecting, executing, and analyzing tests across revisions. Future work can build on this methodology to support ever faster software evolution in the era of agentic coding, where testing remains an essential guard of software quality.