Cheriton School of Computer Science Professor Dan Berry and his former PhD student Sri Fatimah Tjong have received the most influential paper award at REFSQ 2023. Also known as the 29th International Working Conference on Requirements Engineering: Foundation for Software Quality, the annual meeting took place this year from April 17–20 in Barcelona, Spain.
“The Design of SREE — A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned,” a paper presented a decade earlier at REFSQ 2013, was selected for the award because it has strongly impacted the field of natural language processing tools for requirements engineering, in particular, for its highlighting of the role and importance of recall and precision in natural language requirements specifications. The paper has also influenced industrial practices and it continues to be cited in a variety of journals, conferences and workshops spanning the fields of software engineering, requirements engineering, and software and systems modelling.
Daniel M. Berry is a Professor at the Cheriton School of Computer Science. His research interests are in software engineering generally and in requirements engineering and electronic publishing specifically. Last year, he celebrated 50 years as a professor of computer science, a vibrant career that has spanned three countries and multiple institutions.
Sri Fatimah Tjong (shown inset in a 2007 photo in article) is the Executive Director and IT Manager at Capital Dynamics Ltd., a Singapore fund management company.
About this award-winning research and how Dan Berry became a co-supervisor of a doctoral student half a world away
In 2004, Sri Fatimah Tjong received a PhD scholarship to attend the University of Nottingham Malaysia, a branch campus of the United Kingdom’s University of Nottingham that had been established four years earlier. Although she was an ambitious student with full funding, Sri’s academic future was at risk. She had been assigned three consecutive advisors, none of whom had a background in requirements engineering and all of whom had returned to their home university shortly after joining the University of Nottingham Malaysia. She approached her graduate dean, concerned that her doctoral degree was in jeopardy, and insisted that she be co-supervised by a local supervisor in Malaysia and a requirements engineering expert elsewhere, a search for a potential co-supervisor she instigated by scanning the conference and journal literature in her field.
A few experts had replied to her plea, but only Waterloo’s Dan Berry responded enthusiastically.
“She sent me an e-mail explaining the situation and that she needed an external supervisor. She told me precisely what she wanted to work on,” Dan recalled. “I love this kind of chutzpah in a student because it practically guarantees the student will finish without my having to crack a whip or rescue the student. So, I accepted.”
In 2004, ambiguity in requirements specifications was a hot topic, and several tools to find ambiguity known as abstraction finders had and were being built. Sri wanted to improve on them. At the time, intelligent abstraction finder tools used in natural language processing were achieving recall no better than about 85 per cent. Sri’s supervisors suggested that she look into building a parser for natural language processing, since all requirements are written in natural language.
When Sri implemented a parser-based tool, the recall was low — less than 70 per cent — partly because of the very ambiguity it was seeking to detect.
“I knew that all the natural language tools built to date, which had no better than 85 per cent recall, were based on parsers,” Dan explained. “I told Sri that I was not surprised at the low recall, and that some in the research community were thinking that a dumb clerical tool was the way to go.”
Sri explored such a tool for the abstraction finding task. The key turned out to be decomposing the abstraction finder task into two parts — a tool part and a human part. She identified a decomposition of the abstraction finder task that was worth exploring — i.e., using a dumb clerical tool that finds with 100 per cent recall every sentence that has at least one instance of any algorithmic ambiguity, and leaving any other, non-algorithmic ambiguity to be found manually in a search focused on the non-algorithmic ambiguities.
To this end, Sri designed and built SREE, a prototype abstraction finding tool, and tested whether it would work on real-world requirements specification documents in English, a natural language. The results, however, were not good as the tool barely met its goals. Nevertheless, Sri’s PhD thesis was defended successfully as it was an exemplar of good science — a rigorous empirical test of a reasonable hypothesis that unfortunately yielded negative results.
Publishing a summary of her research, however, proved to be a much more daunting task, as negative results are notoriously difficult to publish.
A few years later, Sri and Dan joined Ricardo Gacitua and Pete Sawyer, then of Lancaster University, to publish a paper in REFSQ 2012, titled “The Case for Dumb Requirements Engineering Tools.”
Soon after that publication, Dan spent a week visiting Sri and her husband in Singapore to find a way to frame Sri’s thesis work and to write a draft of the paper that they submitted to REFSQ 2013. Dan recalled, “We decided to frame the work as though were a research idea that did not work quite the way it was planned, and as an instance of a kind of dumb tool.”
Because the tool did not work as hoped, Dan and Sri presented the research as promising ideas for future work. The framing worked, and the paper was accepted for publication at REFSQ 2013.
“It drives home the point that as authors we have no control over a paper’s impact,” Dan said. “But this paper, which I thought was mediocre because of its negative results, won a most-influential paper award a decade later.”
To learn more about the research on which this article is based, please see Sri Fatimah Tjong, Daniel M. Berry. The Design of SREE — A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned. In J. Doerr, A.L. Opdahl (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2013. Lecture Notes in Computer Science, vol. 7830. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37422-7_6 (journal subscription or institutional access required)
