Recent PhD graduate Mike Schaekermann has received the 2020 Distinguished Dissertation Award from CS-Can|Info-Can, the non-profit professional society that represents computer scientists and the interests of their discipline across the nation.
The Distinguished Dissertation Award was established in 2018 to recognize excellence in research and writing by a computer science doctoral student. A nominee’s thesis is assessed for originality, potential impact, technical depth and significance, and quality of presentation by a panel of computer scientists from a variety of subdisciplines in the field. This prestigious national award is conferred annually to one recent PhD graduate.
L to R: Professor Kate Larson, recent PhD graduate Mike Schaekermann and Professor Edith Law deliberate to resolve a disagreement. One aspect of Dr. Schaekermann’s thesis examined whether disagreements among crowdsourced workers can be resolved in the context of both objective and subjective classification.
Dr. Schaekermann’s dissertation titled, “Human-AI Interaction in the Presence of Ambiguity: From Deliberation-based Labeling to Ambiguity-aware AI,” demonstrated how humans and AI can be partners on ambiguous problems. To address this question, he studied group deliberation as a tool to detect and analyze ambiguous cases in data labelling. He presented three case studies that investigate group deliberation in the context of different labelling tasks, data modalities and types of human labelling expertise.
Dr. Schaekermann was co-supervised by Cheriton School of Computer Science Professors Edith Law and Kate Larson.
“Mike is creative, ambitious, organized and articulate,” said Professor Edith Law. “He has without question produced a coherent and beautifully written thesis with substantial real-world impact. His work fundamentally changes the way we think about how human users should work with classification labels and machine-generated predictions. As a result of his thesis work, the CrowdEEG system, which is now open-sourced, has been used by neurologists and epileptologists all over the world. It continues to support neurology research at Sunnybrook Hospital in Toronto.”
“Mike was a joy to work with,” adds Professor Kate Larson. “I am delighted that his incredible work is being recognized by a CS-Can|Info-Can Distinguished Dissertation Award.”
He is currently an Applied Scientist at Amazon Web Services.
More about Dr. Schaekermann’s research
Dr. Schaekermann’s research focused on developing new interfaces and algorithms to handle ambiguous classification problems. His thesis presented the argument that although some classification problems are irresolvable, the mere act of deliberation — facilitated by appropriate tools and interfaces — can improve decision-making and ultimately the classification labels. His dissertation resulted in three significant contributions.
First, Dr. Schaekermann’s research changed the way that people who use machine learning approach classification labels and machine-generated predictions. In machine learning, it is often assumed that labelled data are correct and that every object has one and only one classification label. This assumption often is false — whether because of conflicting evidence or poorly specified categories — yet the standard practice has been to simply ignore ambiguity. Dr. Schaekermann challenged this assumption and showed how ambiguity can be resolved through deliberation. Moreover, he demonstrated how deliberation can be critical in medical domains and how ambiguity can guide AI-assisted medical decision-making. His work on classifying ambiguous crowdsourced data was presented at CSCW 2018, the 21st ACM Conference on Computer Supported Cooperative Work and Social Computing, where it received a best paper award.
Second, Dr. Schaekermann produced a deliberation-based annotation system, which is technically challenging to build because the design of interfaces needs to match the expectations of medical experts as well as allow them to appropriately visualize the data — in this case, retinal images and brain EEG data. He singlehandedly developed a complex crowdsourcing system called CrowdEEG — a web-based, collaborative annotation tool for medical time series data — that facilitated multiple research studies that explore the work processes of doctors and technicians. He also created a mobile version in collaboration with researchers from Harvard Medical School on the Guinea Epilepsy Project to allow epilepsy diagnosis and treatment in resource-poor clinics. Similarly, at Google he contributed to and studied a system that allows ophthalmologists to collaboratively annotate and deliberate on ambiguous images of eye disease. His thesis not only answered fundamentally interesting research questions, but it also produced immediate real-world benefits.
Third, Dr. Schaekermann made a tangible impact on medical data analysis practices. To date, 225 neurologists and epileptologists across the globe have read and annotated EEG recordings using the CrowdEEG web interface he created. The studies have collected more than 145,000 annotations on 50 sleep recordings, producing a unique open-source dataset for clinicians and researchers. At Google Research, he explored the question of expert disagreement in diagnosing vision diseases. This research was presented not only at leading human-computer interaction conferences, but also published in medical journals, including Seizure — European Journal of Neurology, Translational Vision Science & Technology and Ophthalmology.
