Cheriton School of Computer Science Professor Dan Berry and his former PhD student Sri Fatimah Tjong have received the most influential paper award at REFSQ 2023. Also known as the 29th International Working Conference on Requirements Engineering: Foundation for Software Quality, the annual meeting took place this year from April 17–20 in Barcelona, Spain.
“The Design of SREE — A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned,” a paper presented a decade earlier at REFSQ 2013, was selected for the award because it has strongly impacted the field of natural language processing tools for requirements engineering, in particular, for its highlighting of the role and importance of recall and precision in natural language requirements specifications. The paper has also influenced industrial practices and it continues to be cited in a variety of journals, conferences and workshops spanning the fields of software engineering, requirements engineering, and software and systems modelling.
Daniel
M.
Berry
is
a
Professor
at
the
Cheriton
School
of
Computer
Science.
His
research
interests
are
in
software
engineering
generally
and
in
requirements
engineering
and
electronic
publishing
specifically.
Last
year,
he
celebrated
50
years
as
a
professor
of
computer
science,
a
vibrant
career
that
has
spanned
three
countries
and
multiple
institutions.
Sri
Fatimah
Tjong
(shown
inset
in
a
2007
photo
in
article)
is
the
Executive
Director
and
IT
Manager
at
Capital
Dynamics
Ltd.,
a
Singapore
fund
management
company.
About this award-winning research and how Dan Berry became a co-supervisor of a doctoral student half a world away
A few experts had replied to her plea, but only Waterloo’s Dan Berry responded enthusiastically.
“She sent me an e-mail explaining the situation and that she needed an external supervisor. She told me precisely what she wanted to work on,” Dan recalled. “I love this kind of chutzpah in a student because it practically guarantees the student will finish without my having to crack a whip or rescue the student. So, I accepted.”
In 2004, ambiguity in requirements specifications was a hot topic, and several tools to find ambiguity known as abstraction finders had and were being built. Sri wanted to improve on them. At the time, intelligent abstraction finder tools used in natural language processing were achieving recall no better than about 85 per cent. Sri’s supervisors suggested that she look into building a parser for natural language processing, since all requirements are written in natural language.
When Sri implemented a parser-based tool, the recall was low — less than 70 per cent — partly because of the very ambiguity it was seeking to detect.
“I knew that all the natural language tools built to date, which had no better than 85 per cent recall, were based on parsers,” Dan explained. “I told Sri that I was not surprised at the low recall, and that some in the research community were thinking that a dumb clerical tool was the way to go.”
Sri explored such a tool for the abstraction finding task. The key turned out to be decomposing the abstraction finder task into two parts — a tool part and a human part. She identified a decomposition of the abstraction finder task that was worth exploring — i.e., using a dumb clerical tool that finds with 100 per cent recall every sentence that has at least one instance of any algorithmic ambiguity, and leaving any other, non-algorithmic ambiguity to be found manually in a search focused on the non-algorithmic ambiguities.
To this end, Sri designed and built SREE, a prototype abstraction finding tool, and tested whether it would work on real-world requirements specification documents in English, a natural language. The results, however, were not good as the tool barely met its goals. Nevertheless, Sri’s PhD thesis was defended successfully as it was an exemplar of good science — a rigorous empirical test of a reasonable hypothesis that unfortunately yielded negative results.
Publishing a summary of her research, however, proved to be a much more daunting task, as negative results are notoriously difficult to publish.
A few years later, Sri and Dan joined Ricardo Gacitua and Pete Sawyer, then of Lancaster University, to publish a paper in REFSQ 2012, titled “The Case for Dumb Requirements Engineering Tools.”
Soon after that publication, Dan spent a week visiting Sri and her husband in Singapore to find a way to frame Sri’s thesis work and to write a draft of the paper that they submitted to REFSQ 2013. Dan recalled, “We decided to frame the work as though were a research idea that did not work quite the way it was planned, and as an instance of a kind of dumb tool.”
Because the tool did not work as hoped, Dan and Sri presented the research as promising ideas for future work. The framing worked, and the paper was accepted for publication at REFSQ 2013.
“It drives home the point that as authors we have no control over a paper’s impact,” Dan said. “But this paper, which I thought was mediocre because of its negative results, won a most-influential paper award a decade later.”
To learn more about the research on which this article is based, please see Sri Fatimah Tjong, Daniel M. Berry. The Design of SREE — A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned. In J. Doerr, A.L. Opdahl (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2013. Lecture Notes in Computer Science, vol. 7830. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37422-7_6 (journal subscription or institutional access required)