Translating music into light and motion with robots

A system developed by researchers at the University of Waterloo lets people collaborate with groups of robots to create works of art inspired by music. 

The new technology features multiple wheeled robots about the size of soccer balls that trail coloured light as they move within a fixed area on the floor in response to key features of music including tempo and chord progression. 

A camera records the co-ordinated light trails as they snake within that area, which serves as the canvas for the creation of a “painting,” or visual representation of the emotional content of a particular piece of music. 

“Basically, we programmed a swarm of robots to paint based on musical input,” said Dr. Gennaro Notomista, a professor of electrical and computer engineering at Waterloo. 

“The result is a cohesive system that not only processes musical input, but also co-ordinates multiple painting robots to create adaptive, expressive art that reflects the emotional essence of the music being played.” 

The robots represent emotion as they “listen” to music via the colours, intensity and width of their lights trails, as well as their position on the canvas and the speed with which they move within it. 

People can simultaneously influence a painting in progress using controls to change the width of light trails and their location on the virtual canvas. 

“We included the human control input to allow people and robots to work together,” said Notomista, whose interests include the intersection of art and technology. “The human painter should complement and be complemented by what the robots do.” 

The first challenge for researchers was developing an algorithm to control multiple robots within a given area. They tested the system with up to 12 robots, but it can be scaled to handle any number. 

Step two involved creating technology to extract and analyze musical features that express emotion so they can then be translated into light trails that appropriately represent them. 

Lessons learned during the project have potential applications in other areas requiring the control and co-ordination of multiple robots working in unison, such as environmental monitoring, precision agriculture, search and rescue missions, and planetary exploration. 

The research also reflects the University of Waterloo’s Global Futures initiative, which advances interdisciplinary work that considers how emerging technologies can shape society, culture and the human experience. 

Later, Notomista plans to enlist professional painters and musicians to explore the possibilities of the new tool in user studies and stage public exhibitions. 

A paper on the system, Music-driven Robot Swarm Painting, by Notomista and Jingde Cheng, a former Waterloo graduate student, was presented at the 2025 IEEE International Conference on Advanced Robotics and its Social Impacts.