Some chairs may look futuristic, but a particular chair at the Cheriton School of Computer Science is futuristic. It has been custom fitted with sensors, servos, computers and a small projector by a team of human-computer interaction researchers to create an office chair as a platform for personal spatial augmented reality.
“Much research has shown that the movements we make in office chairs have implicit meaning, and because these movements have meaning they can be enhanced by spatial augmented reality,” explains project lead Nikhita Joshi, a PhD student advised by Professor Daniel Vogel. “For example, leaning back in an office chair suggests a person is relaxing. We wondered if we could enhance and support such movements using spatial augmented reality.”
L to R: PhD candidates Nikhita Joshi and Antony Albert Raj Irudayaraj, and Professor Daniel Vogel from the Cheriton School of Computer Science’s Human-Computer Interaction Lab. PhD graduate Jeremy Hartmann was unavailable for the photo.
Nikhita’s research focuses on understanding how significant interface constraints, such as time or character limits, can be leveraged to enable users. She is also interested in augmented reality, novel interaction techniques, and software learning.
Antony’s research interests span interactive displays, wearable systems, and haptics.
Professor Vogel’s research focuses on fundamental characteristics of human input and novel forms of interaction for current and future computing form factors like touch, tangibles, mid-air gestures, and whole-body input, for everything from on-body wearable devices and mobile phones to large displays and virtual reality.
Jeremy graduated from the Cheriton School of Computer Science with a PhD in 2022. His research explores merging the real with the virtual, which his company MTION is putting into practice as a virtual reality streaming service and digital clubhouse.
Spatial augmented reality is a technology that merges the real and virtual worlds by super-imposing computer-generated content onto surfaces using one or more digital projectors.
“Say you’re leaning forward in your office chair,” Nikhita said. “Leaning forward suggests you’re engaged in a task or meeting with a co-worker. A spatial augmented reality system could detect your sitting posture and the location of the chair in the office, then use these inputs to project digital content you’re discussing with a co-worker on the wall, perhaps in a zoomed in way.”
But before a standard office chair can be used to project computer-generated images on surfaces, the chair itself needs to be able to sense its environment. It needs to be instrumented.
Chair sensors and projector actuation
(a) Force-sensitive resistors on the back and seat of the chair sense the user’s posture
(b) Capacitive copper strips on the armrests sense touch input
(c) A circuit board and Arduino computer under the seat processes the chair inputs
(d) A Raspberry Pi computer under the seat collects the data and images and streams them to a local server
(e) A custom mount under the right armrest holds a depth camera to sense chair location and a servo pan/tilt mechanism to steer the projector
(f) An accelerometer on the chair’s back tracks tilt and rotation
(g) A battery pack powers the devices
“For touch input we placed a series of capacitive sensors on the chair’s armrests,” explains Antony Albert Raj Irudayaraj, a PhD student advised by Professor Vogel who collaborated with Nikhita on the project. “The back of the chair has an inertial measurement unit — an accelerometer — to detect tilt and rotation, and the seat and backrest have force-sensitive resistors to detect seated postures and leg position. A depth camera tracks the chair’s position and surfaces in the office where content can be projected. Underneath the chair are small battery-powered computers to process these inputs and determine the projector’s output. On the right side is a servo-actuated pan/tilt head that determines where the projector displays digital content in the office.”
It’s important to note that an instrumented office chair does not use the various inputs from a user to explicitly control the projector’s output, Professor Vogel notes. “The system we developed processes implicit input to support some action we want to do. The system then finds the best way to support that action. When you move the chair, the projector has to move to compensate for that movement. It’s a complex relationship that combines what the user is doing, how they’re sitting, what the contextual cues mean, and where the chair moves. The system processes these inputs and brings them together using a projector to place digital content on surfaces in the office that’s useful to the user.”
With the chair now instrumented, the team explored 11 demonstration applications in a typical office to evaluate their proof-of-concept system.
“We made videos of these demonstration applications across three categories — notifications, supporting work-related tasks, and encouraging breaks and relaxation,” Nikhita said. “We then conducted an online survey to evaluate how the chair’s technical components work together, by asking questions that examined the participants’ understanding of the system, and its perceived usefulness and comfort. We wanted feedback on our proof-of-concept because an instrumented chair is futuristic.”
| Notifications | |
|---|---|
| Day at a glance | A summary of the user’s day is projected on a wall above the computer |
| Ambient notifications | Notifications are projected on nearby objects to serve as reminders |
| Be-back-soon message | When a user stands up from the chair, the person’s name, photo, and a be-back-soon message are projected on a nearby wall |
| Notification tray | Notifications of the user’s emails, social media feed, calendar events, and weather alerts, are projected on a nearby door |
| Supporting work-related tasks | |
| Enhancing a meeting | During a meeting with a co-worker, the projector rotates to display a meeting agenda or presentation slideshow on a nearby cabinet |
| Augmenting physical drawing surfaces | Brainstorming sessions with co-workers often involve drawing on a whiteboard; to augment illustration, a grid, geometric pattern, or circuit diagram, is projected on the whiteboard |
| Supporting work-related tasks | If the chair is moved toward a specialized work area, the projector displays related content such as a video tutorial |
| Encouraging work breaks and relaxation | |
| Reducing eye strain | The chair applies the 20-20-20 guideline — i.e., every 20 minutes, look away from your monitor and focus on an item about 20 feet away for 20 seconds to reduce eye strain; the projector periodically moves content from a desktop to surfaces further away |
| Deep-breathing exercises | If the user leans back in the chair, the projector displays a deep-breathing exercise to follow on the ceiling |
| Ambient lighting to augment video games | Ambient lights are projected above the user’s monitor as a video game is played to enhance the game |
| Games | Computer games are expanded to larger surfaces, such as the floor or a nearby wall, for individual and multiplayer participation |
The survey found that respondents understood the purpose of the 11 applications clearly, but were split on their perceived usefulness. Spatial augmented reality that supported tasks, reduced eye strain, served as reminders, and prompted deep-breathing exercises were seen as particularly useful.
“The survey also revealed some interesting differences,” Nikhita said. “For example, within notifications we looked at different types — abstract vs detailed notifications. With a projector everyone can see the content, not just the user, so preserving privacy becomes important. Respondents were not as comfortable showing the content of an email but liked an ambient notification that a message had been received. This is an important finding, and future work might be on ways to occlude parts of a notification to preserve privacy.”
The survey also provided insight on certain applications, such as working with tools at a desk, where the instrumented chair projects a video tutorial on the wall to aid the user, Nikhita said. “These findings could be used to dive deeper into specific uses and augment them further to make the applications more compelling.”
“We focused on tasks people do in a typical day in an office, but an instrumented chair could have many other applications,” said Professor Vogel when asked about possible future directions. “A medical exam room also has chairs that could be instrumented. A doctor’s chair could be instrumented to project a patient’s medical records or the results from diagnostic tests on the wall when the physician makes certain movements in the chair. Similarly, a teacher’s chair could be instrumented to support teaching activities in a classroom.”
This research received the Best Paper Award at SUI 2022, the ACM Spatial User Interaction Symposium, where it was presented originally in December 2022.
To learn more about the research on which this feature is based, please see Nikhita Joshi, Antony Albert Raj Irudayaraj, Jeremy Hartmann, Daniel Vogel. An Instrumented Office Chair with a Steerable Projector for Personal Spatial Augmented Reality. SUI 2022: Proceedings of the 2022 ACM Symposium on Spatial User Interaction, December 2022, article no.: 1, pp. 1–12.
Please also see the demonstration video for example of the applications in use.
