Candidate: Haonan Zhang
Date: November 17, 2026
Time: 11:30 AM
Location: Online
Supervisor: Weiyi Shang
All are welcome!
Abstract:
Web GUI testing validates web applications from the user’s perspective, but scripted tests built with tools such as Selenium and Playwright are notoriously brittle. LLM-based web agents offer a more flexible alternative: they execute natural-language test scenarios and adapt their actions to the observed GUI. Yet they are typically evaluated on task completion or bug detection in a single run under stable conditions. This seminar presents two studies that look beyond a single clean run.
The first study introduces BrittleBench, a mutation-based benchmark that injects 16 configurable runtime disturbances, such as popups, cookie banners, layout shifts, loading delays, new tabs, and viewport changes, while preserving task intent and application logic. On 57 test scenarios, we compare Playwright tests generated from human trajectories with a ReAct-style web agent. Scripted tests tolerate most non-blocking changes but fail when a disturbance blocks their fixed path or changes the browser context. The agent has a lower overall failure rate (27.3% vs. 42.2%) and often recovers by taking a different path, but it is not always better: it remains sensitive to layout and page-readiness changes and sometimes claims success on tasks it did not complete.
The second study asks whether web testing agents detect the same bug consistently. Evaluating two agents on 100 bug-injected scenarios (4,000 executions in total), we find that aggregate metrics change little across repeated runs, yet for 22–28% of scenarios, depending on the agent, the same bug is detected in some runs and missed in others. More than half of the missed detections happen after the faulty state has already been reached, due to how the agent generates or executes its test oracle. Benign runtime changes further reduce consistency and can sharply inflate false positives. Handling popups and loading overlays with a lightweight Runtime Change Handling Agent restores both agents to near their clean-baseline performance at modest overhead.
Together, these findings suggest that web testing agents should be evaluated across repeated runs and realistic runtime conditions, and they motivate hybrid tools that combine the precision and repeatability of scripted tests with the adaptability of agents.