PhD candidate Ludwig Wilhelm Wall and two recent Waterloo graduates have built an interactive installation at the intersection of art and technology called The Light Within. Their installation is part of Lumière: The Art of Light, a free outdoor light experience on exhibit at Ontario Place’s Trillium Park from March 10 to May 7.
The Light Within is an interactive outdoor exhibit that brings together the artistic talents of Nathan Fischer (BSc Chemical Physics, 2022) and Diana Tran (BFA, 2021) with Ludwig’s expertise in constructing large-scale pixel displays.
“We are excited to welcome visitors to Trillium Park to experience the beauty and vibrance of Lumière,” said Janet Gates, general manager and CEO of Ontario Place, in a media release. “As the park transitions from winter to spring, it’s the perfect location for this outdoor public art exhibition. We are also thrilled to provide a forum to showcase and celebrate the work of Ontario artists, providing them with an accessible and beautiful space to exhibit their art.”
When a person stands in front of The Light Within exhibit, a camera captures their image and projects it back toward them using a wall of large-scale pixels. The exhibit is like a mirror but instead of a reflection the person becomes part of the art installation itself in the form of an illuminated matrix of pixel elements, each about the size of a ping-pong ball.
A person in front of the Light Within Exhibit, a pixel-based art installation that allows the viewer to become part of the installation.
“Constructing a public art display comes with many challenges,” Ludwig explains. “All the interactive elements need to be constructed like a lock and expose no screws or fittings. All the external surfaces need to be resistant to the elements, as well as to kids and adults — even to dogs in the park accidentally bumping into them. The exhibit is interactive and can be manipulated by the viewer. With one dial a viewer can control how noisy or clear the image is, while another alters the image from black and white to vibrant colours.”
Ontario Place recently hosted a call for Ontario-based artists for submissions, with 60 proposals received and reviewed through a juried process. In total, 16 projects were selected, among them The Light Within, to be showcased at Lumière: The Art of Light. Open seven nights a week, patrons are invited to view and experience the exhibits from dusk to 11 p.m.
Technical details of The Light Within installation
Data transmission
A Raspberry Pi 4 computer controls a webcam, reads the interactive input from the user-adjustable dials, and transforms the camera image according to the artist’s vision and the patron’s input. However, the Raspberry Pi cannot refresh the nearly 2,000 LEDs of the display through its GPIO or PWM pins quickly enough, so an ESP32 microcontroller running an instance of the WLED webserver provides the required GPIO speeds.
As signal strength is another limiting factor, the display was divided into four quadrants to keep the length of the physical data cable short. As a result, four GPIO pins are used to control each quadrant of the overall display in parallel.
Adding specialized hardware addresses the need for the wall of LEDs to operate interactively, but it also creates a new communication channel that must run similarly fast. The serial connection between the Raspberry Pi and ESP32 microcontroller does not have sufficient speed so a shielded wireless connection was the solution implemented.
Although WLED offers a comprehensive JSON HTTP API, using a stateless communication protocol like UDP is preferrable for interactive display applications. UDP offers a graceful way to manage packet loss by simply displaying the last frame on missed packets. This avoids cascading issues, and the worst-case scenario is that the display skips refreshing for a frame or two.
However, the UDP-based protocols that WLED supports have a downside. The maximum number of LEDs that can be updated with one UDP packet is 490, so multiple packets are required to update the entire display. The DNRGB protocol can be used to select a range of LEDs to be updated, though the LEDs must be in order, and only raw pixel colour data can be transmitted. For this display, four packets need to be sent to transmit one frame of display data.
Updating a single LED in a line of LEDs requires updating the entire line. Rectangular LED displays are typically wired in a serpentine fashion that snakes through the display from one corner to the opposite corner. Since this display was constructed in quadrants by hand, it does not match the default serpentine order of LEDs.
A key step to achieve interactive display frame rates is to ensure that any LED is updated only once per frame. In practice, this means that any UDP packet may contain only data affecting LEDs that are connected to the same GPIO data pin.
Since DNRGB UDP packets address a single continuous range of LEDs, the default serpentine LED layout cannot be used. Instead, the permanent digital representation of the order of LEDs has to be adjusted to match the physical wiring. This representation can be segmented to simplify the following steps. Additionally, the webcam pixel data — a 3D array of colour values — must be split into segments and shaped to match the LED order. This can be done sufficiently quickly by making use of NumPy’s efficient matrix manipulations. Once the pixel order of a flattened sub-array matches the order of LEDs, data can be transmitted to the display by crafting a custom UDP packet header and appending the sub-array in byte form directly.
Resilient execution
The first step in making the output more resilient is to ensure that separate parts of the system can run independently. If real-time display data from the Raspberry Pi is not available, the WLED instance running on the ESP switches to a default animated LED pattern that resembles an animated loading screen.
The ESP microcontroller and WLED library are designed to be resilient by default, and restart automatically after temporary outages. Although the Raspberry Pi restarts automatically after losing power, the script that generates the display output must be made resilient by hand. The script itself is designed to exit in case an issue is detected. By running the script as a service, and using systemd to restart the script on failure, resilience is achieved with minimal code as Linux itself ensures that one instance of the script is always running.
Any temporary issue, such as a power outage, is addressed by restarting parts of the system. Unexpected ongoing issues are circumvented by restoring the system to its default state. And, as a last resort, the code is designed to run with limited functionality even while parts of the system are unavailable.
