Computer science and software engineering students made up the dynamic team that created PearPiano, a winning entry at the recent Hack the North hackathon.
PearPiano is an augmented reality (AR) program for playing, composing and recording piano music. The AR runs on Unity and Oculus Quest goggles and incorporates hand tracking to allow users to play the virtual instrument and interact with a composition. The team also integrated OpenAI's Whisper API in the form of a chatbot that helps users with any questions.
“The setup for it was difficult at first,” says Sarah Wilson, an undergraduate computer science student and Schulich Leader at Waterloo. “We couldn’t detect our hands initially and we couldn’t touch the piano keys. And we couldn’t change the colours of the keys to give the program a better visual aspect. But after a very late night working, we got it.”
The student team that created the winning PearPiano app, presenting at the finals of Hack the North. L-R: Sarah Wilson, Justin Lin, Sophia Sun and Emily Wang.
The creative team behind PearPiano also included software engineering students Justin Lin, Sophia Sun and Emily Wang. The team members all have an interest in music and composition, and of course also a shared interest in hacking and innovation.
Hack the North runs over the course of an entire weekend and is hosted at Waterloo. Undergraduates and some high school students from across Canada and internationally travel to take part in the hackathon. Participants have a time limit on their work and then submit their projects for judges to decide the winners.
“One of the features we were building is an edit mode, which allows users to drag and drop notes,” Wilson continues. “When you’re composing, you make a lot of mistakes as part of a creative process. So if you can drag and drop keys and section out that part, it will really help with composition.”
Read more about PearPiano and all that went into the winning app.