PhD Seminar • Software Engineering | Machine Learning • Improving the Reliability of Deep Learning Software Systems at Three Stages of Development

Thursday, July 28, 2022 10:00 am - 11:00 am EDT (GMT -04:00)

Please note: This PhD seminar will be given online.

Hung Pham, PhD candidate
David R. Cheriton School of Computer Science

Supervisors: Professors Lin Tan, Yaoliang Yu

For the last decade, Deep learning (DL) has emerged as a new effective machine learning approach that is capable of solving difficult challenges. Due to their increasing effectiveness, DL approaches have been applied widely in commercial products such as social media platforms and self-driving cars. Such widespread application in critical areas means that mistakes caused by bugs in such DL systems would lead to serious consequences. Our research focuses on improving the reliability of such DL systems.

On a high level, the DL systems development process starts with labeled data. This data is then used to train the DL model with some training methods. Once the model is trained, it can be used to create predictions for some unlabeled data in the inference stage. We developed testing and analysis techniques that help improve the DL system reliability for all stages.

In this talk, I will first present our work in improving the reliability of the DL system inference by applying differential testing to find bugs in DL libraries. We leverage different equivalence concepts such as equivalence DL libraries or equivalence DL models to detect bugs in DL software systems.

In the second part of the talk, I will present our analysis of how the software implementation of DL libraries alone causes a large variance in training in terms of both model accuracy and fairness. Our large DL developers survey also shows that a majority of DL developers are not aware of or underestimate such training variance caused by software implementation.

I will also present our work in detecting blindspots in training data. Our approach leverages N models to detect weakspot samples that can be added to improve the models with retraining.

Finally, I will present my research vision for applying software engineering techniques to improve the reliability of DL software systems. My future research would focus on the synergy between the DL model and its implementation to improve the reliability of the DL software system from both the software engineering and artificial intelligent perspectives.