Converting videos to spatial videos

Spatial videos are videos that incorporate 3D content allowing users to interact with the environment and provide a much more immersive viewing experience. These videos look more real and make you feel like you were at the original scene when viewed on devices like the Apple Vision Pro. Earlier, we managed to convert 2d photos into spatial photos as shown in the previous blog post and we have now extended this to videos.

We found a few spatial converter implementations online but didn't initially find any free ones. We were also pretty curious about how challenging it would be to make one of our own. So...

You take a regular 2d video and transform it into a 3d video format that has been around for a while. More specifically, spatial videos conform to the video compression codec format MV-HEVC on the Apple Vision Pro. For us, that means you get the depth map of each frame in a video using an off the shelf solution for monocular depth estimation and generate two rotated images from that to represent the left and right eye. Then, you stack the photos on top of each other to create a new video that looks like a video with stacked frames. With the new video, we leverage Mark Swanson's Spatial converter to finish the conversion of the the stacked frames into the earlier format that the device understands. The device does the actual hard lifting of partitioning the frames and sending the right frames to each eye.

We thought that creating a video would be super difficult, especially since we were unfamiliar with the MV-HEVC video format and we didn't know which tools would be necessary to concatenate all the frames together. We also anticipated a lot of compute problems since it already takes a while to generate a single spatial photo and expected a video to take much longer. However, the open source community is strong and a lot of our work was offloaded to existing libraries.

Leave a Comment