How to have a more natural conversation with a robot

Talking to a robot often feels stilted or delayed, thanks to computer software trying to keep up with the conversation. However, new research from the University of Waterloo has improved the ability for humans to communicate naturally with humanoid robots.

Waterloo researchers were able to solve how a humanoid robot can identify the direction where human speech was coming from, re-orient itself to track that voice to create a more realistic conversation and speed up its reaction time to communicate back with its human counterpart.

“Creating more natural conversations with humanoid robots is an interesting challenge because speech is so key to our social interactions,” said Ewen MacDonald, a professor in Waterloo’s Department of Systems Design Engineering and a member of the research team.

The research team built a hearing system for a humanoid robot using two microphones where a human’s ears would typically sit to help generate an estimate of the direction in which audio sounds were coming from. Sounds produced by humans and other acoustic sources are reflected by surfaces such as objects or walls.

A signal processing pipeline is needed to account for these reflections as they could incorrectly fool the humanoid robots on where the sound is coming from. For a robot to react as quickly as a human would sonically, the robot’s computer needs to generate that estimated location extremely quickly.

Waterloo’s research develops a framework that can optimize the robot’s processing speed and characterize different sounds based on overall performance and latency. Testing on the humanoid robot used the framework with recordings in a variety of acoustic environments.

Pranav Barot, a graduate student in Systems Design who worked on the research paper, said the team was motivated by the desire to test the capabilities of humanoid robots being able to listen and interact with humans in real-time. One major challenge of the research was testing how the robot reorients itself to hear humans in large, loud, or crowded spaces. 

“The implications of this research are important in any scenario or environment where humanoid robots will work together with human beings, both in social robotic situations or where humans and robots are working together,” said Barot. 

More information about this work can be found in a research paper published recently by PLOS One.