How we brought virtual reality and dome visuals to a live performance
event at OMSI’s Kendall Planetarium Theater.
Imagine a show with live dance and musical performances that are timed perfectly to accompany visuals projected inside of a 52-foot dome theater… that is also synchronized with a custom built XR experience. Sounds pretty intricate, right? It sure was! This multimedia event was called When We Were Ocean, and the story of how it all came together is about as complex as the show itself.
Hungry Mantis filmed original VR content, converted VR and traditional video footage to project correctly in the dome theater, and facilitated the live VR component of When We Were Ocean for every night of the show’s run.
Creating the VR Content
We first started working with Laura Cannon and her company ProLab Dance in 2022 to create a VR film called Origins. Laura’s vision was to create a dance film in VR that would take place inside of an old wind turbine blade. Capturing the footage for Origins required us to build a custom rig out of our VR/180 camera, a skateboard, gimbal and motorized monopod.
As creative as the filming process was, the editing was one-step further into experimental territory. Once we had a rough edit of Origins completed, we dove into nine months of heavy post-production experimentation. This included adding 3D elements that were tracked to the moving camera, as well as artificially flipping the perspective of the camera’s lenses.
These editing techniques pushed the traditional rules of VR films, and sometimes even went a bit too far. When that happened, we dialed it back just enough to prevent head splitting stereoscopic footage.
The results were beyond our own expectations. By opening our minds to creative experimentation with Laura, we achieved results in VR that we never thought were technically possible. Origins is a truly surreal and beautiful film that could only be achieved through creative VR storytelling.
Making a Full XR Experience
Pleased with the VR visuals, Laura then wanted to add an interactive dance layer. Thus began experimenting with adding touch to what was becoming more than just a VR/180 film. Let’s just say that the first attempts… brought a lot of laughter!
As it turns out- people don’t like to be closely touched by strangers… who knew!
Subtle touch combined with the VR/180 video was the goal, so we had to create a barrier between the dancer and viewer that would make the experience more comfortable.
Laura then began construction of a specialized XR viewing seat. She upholstered a wheelchair with soft and flexible material. The idea was to connect a dancer to the chair while a viewer was seated inside. The foam material around the chair would allow the dancers to gently embrace the viewer, while occasionally moving them forward and backwards over a short distance. The movement and touch were timed to specific camera movements in Origins. This “seed pod chair,” as we began to call it, looked like a living organism. The combination of VR, dance, movement, and touch was fascinating to behold.
Testing Origins with an Audience
Origins was accepted into the FIVARS 2023 Film Festival in Toronto. Laura and Origins vocalist and performer Lynne Piper traveled internationally (with the seed pod chair in tow!) to facilitate the international premiere. The unique combination of fancy VR tech and “analog” human performance was the talk of the festival. The experience motivated all of us to bring this version of Origins to even larger audiences.
Synchronized VR with Showtime
A few months later, Origins debuted as a VIP experience within Laura’s Break to Build show at Zidell Yards. Three users were seated in the seed pod chairs, each with their own dancer. Hungry Mantis used Showtime VR and a monitor to help synchronize VR playback while allowing dancers to move in-time with the events of the film. Thanks to Showtime VR, the dancers knew exactly when to push, pull and embrace the viewer in Origins.
Creating Dome Content
After the success of Break to Build, Laura set about crafting a new immersive experience called When We Were Ocean. This show would take place inside OMSI’s planetarium. Access to this special venue was made possible in part due to our prior experience editing and converting footage into dome projection for Outside the Frame. OMSI knew our work and therefore trusted our team to produce high quality results for their planetarium.
We continued filming more content for Laura at Zidell Yards, only this time with dome presentation in mind. We opted to use a traditional cinematic camera from Z Cam with a fish-eye lens for much of the dome-specific content. Our footage featured aerialists suspended from the ceiling of an old barge building, dancers wielding spark sticks and playing hand-made instruments, and a very creative interpretation of a coral reef inhabited with dancing “fish.”
In addition to filming, we also helped artist Nanda D’Agastino upres her dome video footage. The result was a 30 minute full dome format film that would play alongside Laura’s live dance and musical performances for When We Were Ocean inside the planetarium.
Bringing a VR Screening into the Dome
Another key live element of the show was, of course, screening Origins.
Our plan for this VR component was ambitious. We wanted the VR viewers to actually become part of the show themselves. To do this, two random audience members would be selected to experience Origins. The selection process was choreographed as part of the overall show, and happened about midway through the event.
Dancers guided the chosen viewers to follow them backstage. They were then seated into the seed pod chairs. Laura had expanded the chair design from Break to Build so that now each dancer could essentially “wear” the seed pod by climbing into a special fabric component stitched into the back. This made the dancer and VR viewer look like a new hybrid creature, a perfect compliment to the themes of When We Were Ocean.
Once situated, the dancers would slowly push the chairs out into the middle of the planetarium, where everyone could watch the VR viewers as they watched Origins.
Showtime VR and Synchronization
In order to do all of this in sync with the VR film, we created signals between the dancers and the control booth to begin playing Origins in the Quest 2 headsets. Using Showtime VR, we quickly calibrated the “front” perspective in the headset, so the viewers would mostly be looking straight ahead. This was key for a comfortable experience, because in order to avoid motion sickness, the forward movement in the chairs needed to match the in-headset experience as closely as possible.
We also used Showtime to trigger Origins to play at the same time on both headsets. This timing mattered, as the dome visuals were created specifically to accompany what the viewer was seeing in VR. Gliding across the room, the viewers and dancers moved under the backdrop of beautiful dome projections by Fernanda D'Agostino.
It was also important for the dancers to be aware of what the user was looking at inside VR. They needed to know when to move forward, backward, and embrace the viewer. But they were performing in front of a live audience, so they couldn’t break character to stare up at the dome or look at a monitor off-stage for reference.
Using a headphone splitter, the dancers learned audio queues from the music of Origins and were able to coordinate their movements to precise sounds in the film. After about 10 minutes, the VR viewers had crossed the entire length of the planetarium and arrived on the opposite side of where they had started. Backstage, they were slowly brought back into reality and gently guided to their seats in the planetarium. The show continued on seamlessly from there.
Creating a smooth viewing experience in VR for all levels of VR users can be very difficult to achieve. Showtime VR was integral in streamlining that experience for us. Our viewers didn’t have to press “play” or use a remote to begin Origins. We gave them a truly magical experience that was powered by the capabilities of Showtime VR software.
Even though it felt effortless for our viewers, opening night jitters and the added pressure of running multiple headsets during a theatrical performance timed with dome video, musicians, dancers, and singers was daunting for us… at first. But when everything worked as we had planned and practiced, our worries quickly went away. All the different technical and artistic aspects worked together and our implementation of VR during When We Were Ocean was a success!
The journey of Origins XR and Break to Build will continue to evolve and expand as ProLab Dance prepares for another show this July. We are currently in production with Laura for the next original XR experience, so stay tuned and mark your calendars!
We want thank our collaborators for making these creations possible:
All the incredible dancers and performers
Comments