WICI Speaker Series: Research Funding as a Complex System

Research Funding as a Complex System: Can AI Predict What Peer Reviewers Will Flag as Weaknesses in Grant Applications?

Monday, March 17th, 2025

3 - 4:30 p.m. | DC 1302

WICI external core member James Shelley returns to Waterloo to talk about how complexity theory has informed his exciting new project, now a university-wide pilot at Western, on AI use in the research funding competition space. 

Please register in advance to attend this talk in person or online via Zoom!

We consider the peer-review process as a complex system that appears to defy predictive modelling. Noisy and inconsistent reviewer judgments create a well-documented “black box” problem for applicants, as wide variability in human evaluation makes it difficult for researchers to preemptively identify the weaknesses—both real and perceived—that might reduce an application’s competitiveness. Assuming that the peer review process to be a highly permeable, abstruse and unpredictable system, we explore the potential efficacy of intervention strategies. We investigate the potential of leveraging LLMs to “simulate” potential reviewer critiques and hypothesize that exposing applicants to a comprehensive analysis of potential application critiques ahead of submission might mitigate some of the effects of weak inter-rater reliability. In this presentation, we report on the development of Automated Grant Feedback (AGF), a bespoke AI application that leverages historical reviewer critiques from past competitions in order to anticipate areas that adjudicators might flag as critical weaknesses. The result is a simulation of the “randomness dynamics” of peer review, specific to each unique application. As the model incorporates more historical data, its ability for identifying the same weaknesses as human reviewers increases. AGF is both a unique application of complexity theory in research administration and a demonstration of the potential benefit of using frontier AI models to provide rapid, scalable, and in-depth feedback on grant applications, offering researchers a new layer of novel feedback for refining their proposals prior to submission.

James Shelley heads Western University’s Complex Adaptive Systems (CAS) Lab. This network, established in early 2020, serves as an interdisciplinary hub for the study of complexity and system scholarship at Western University, bridging every faculty on the campus. James is also a Knowledge Mobilization Specialist in the Faculty of Health Sciences at Western University. His role involves piloting strategic initiatives, designing experimental ways to share new research, and delivering professional development training.

James especially focuses on novel approaches to using data, automation and storytelling in knowledge mobilization and science communication.  An autodidact at heart, he has a special interest in the overlap of complexity theory, information, narrative, and communications.

Please register in advance to attend this talk in person or online via Zoom!