Candidate: Cameron Hadfield
Date: April 20, 2026
Time: 10:30 AM
Location: Hybrid (E5-4047 and Microsoft Teams)
Supervisor: Sebastian Fischmeister
All are welcome!
Abstract:
Modern generative LMs present as black boxes, requiring significant trust in their capabilities and making it difficult to understand the reasoning behind their decisions. As these LMs are increasingly used for code and test-case generation, testers must trust them without knowing what drives the model's outputs. To improve accuracy, modern LMs rely on supplementary documentation, such as RAG, or other content directly provided in their prompts to enhance background knowledge. When testers use LM-generated test cases for other purposes, such as fuzz testing, they must place greater trust in their quality, as seed cases can significantly affect fuzzer coverage performance.
We adapt existing methods to build an analysis pipeline that explains document retrieval when the LM relies on documentation to generate test cases. We achieve this with only black-box access to the \glspl{lm} under test. We use RFC 959 (the FTP protocol) and two synthetic protocols to isolate the LM's reliance on data in its RAG system. Statistical analysis shows that the explanations from our pipeline capture real phenomena rather than random data.
To aid integration with automated security testing, we present a formal definition of protocol communication. This formalism helps map our pipeline's features to the protocol domain and lays a foundation for future work with fuzzers.
The explanations our pipeline generates yield unexpected, non-interpretable results, suggesting the need for further development.