Please note: This PhD seminar will take place in DC 2310 and online.
Mohammad Hasan Payandeh, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Jian Zhao
Today’s graphical user interfaces are programmed as code that the CPU executes to issue drawing commands, which the GPU turns into pixels. We introduce Pixel-Generated Graphical User Interfaces (PGUIs), which replace this pipeline with a large language model. Running on the GPU, the model generates each frame of pixels directly from the user’s input, allowing users to create and interact with personalized interfaces without writing code. A PGUI works in two phases. In the generation phase, a UI Generator turns the user’s natural-language description into a persistent interface configuration and its first frame. In the runtime phase, each user interaction triggers a Frame Generator to produce the next frame. When needed, it can issue operating-system commands and show their results. An ongoing work is evaluating whether users can create personalized interfaces and accomplish their daily tasks with PGUIs. This work has limitations: generating every frame through model inference is slower and more costly than the conventional pipeline. We expect these limitations to ease as models become more capable, faster, and smaller, alongside optimizations spanning AI, computer architecture, and graphics research.
To attend this PhD seminar in person, please go to DC 2310. You can also attend virtually on Zoom.