Last week, I sat around a table with fellow journalists as Greg Sullivan, Microsoft’s head of Mixed Reality, detailed the company’s vision for the future of virtual collaboration. Nobody was wearing masks or standing apart. We weren’t worried about getting sick. Instead, we were all wearing HoloLens 2 headsets and sitting in different parts of the world. The holographic table was right beside my actual desk, and my media pals were floating around my office as we chatted with our cartoonish avatars. For a second, it felt like mingling in real life during the Before Times.
We were experiencing one of the first apps powered by Microsoft Mesh, the company’s ambitious new attempt at unifying holographic virtual collaboration across multiple devices, be they VR headsets, AR (like HoloLens), laptops or smartphones. Powered by Microsoft’s Azure cloud, Mesh isn’t just an app, it’s a platform that other developers can use to bring remote collaboration to their own software. If remote work is here to stay — and by most accounts, it is — Microsoft wants to be the company taking us beyond Zoom video chats, and towards holographic experiences that everyone can join.
“Not only are we going to be able to share holograms, but we’ll be able to do so in a way that gives us agency and presence,” Sullivan said during our virtual meeting. “We can create these experiences, where even though we’re physically separated, it feels like we’re in the same room, sharing in an experience and collaborating on a project.”
While we’ve seen a solid stab at virtual collaboration from Spatial, Microsoft is attempting something even more complex. Sullivan likens Mesh to the launch of Xbox Live in 2002, a service that dramatically simplified online multiplayer gaming for consoles. It made it easier for developers to connect their games to the internet, and led to a boom in online multiplayer titles for the Xbox and Xbox 360. That gave Microsoft a strong leg up on Sony and Nintendo, both of which took years to catch up.
Microsoft is using today’s Ignite conference keynote to show off the capabilities of Mesh. Alex Kipman, the company’s Technical Fellow behind the HoloLens and Kinect, will hit the stage as a real-time hologram (something Microsoft calls “holoportation”). Think of it a bit like the holographic messages we’ve seen in Star Wars and other science fiction stories. It’s not photo realistic, but if you’re wearing a VR headset, it’s almost as if he’s in the room with you. On a standard monitor or phone screen, it may just come off as a hokey special effect. But it’s not hard to imagine eventually slipping on an AR headset like the HoloLens 2 and watching a hologram presentation right in your living room, as if you were sitting in the front row during a show.
My Microsoft Mesh demo, to be clear, was nowhere near as impressive. Our avatars were simplistic, with detached arms and limited facial movement. It was like being surrounded by a bunch of Nintendo Miis. But there was still a decent sense of immersion: I could tell exactly where everyone was even when I had my eyes closed, thanks to realistic audio processing. And we were able to collaborate with 3D models, passing them around the table and resizing them to our heart’s content.
While we were looking at fairly basic 3D figures, Sullivan pointed out that Mesh can also stream high-quality models from the cloud (it’s powered by Azure, after all). That would allow designers and engineers to collaborate with the same assets they’re using on their workstations from anywhere in the world. That’s what film director and producer James Cameron is aiming to do with his upcoming series, OceanXplorers. The non-profit behind that show, OceanX, plans to create a Mesh-enabled “holographic laboratory” on its advanced ship, allowing scientists on-board and remotely to collaborate around 3D models.
“The idea is to take all this amazing scientific data we’re collecting and bring it into a holographic setting and use it as a way to guide scientific missions in real time,” Vincent Pieribone, vice chairman of OceanX, said in a statement. It would let researchers huddle around data and chat as they would in real life, no matter how far apart they actually are.
While existing VR collaboration apps offer some semblance of that functionality today — like Microsoft’s own AltSpaceVR — what’s truly interesting about Microsoft Mesh is its cross-device compatibility. Ideally, you’d be able to jump into a Mesh-powered experience no matter what device you’re using. If you’re calling in from a computer or phone, you’d appear in a floating webcam window. And it’s not hard to imagine being able to navigate 3D environments from those devices too, perhaps by pointing and clicking like in adventure game. Phones could potentially be AR windows that let you walk around 3D models projected into your living room or office.
On-stage at the Ignite conference, Niantic also demoed what a Mesh-enabled Pokemon Go experience could look like on HoloLens 2. John Hanke, CEO of Niantic, was able to walk around a park and feed a Pikachu before encountering a colleague, who challenged him to a battle. The demo was clearly a CG proof of concept, and not something actually running on the HoloLens. Still, the move into true AR seems like a natural step forward for Pokemon Go, which became popular for being on of the first accessible mobile AR experiences. Hanke was quick to point out that the demo wasn’t a sign of anything actually coming to consumers (yet). It’s more like a taste of things to come, perhaps when Apple and other companies finally deliver AR glasses.
Pokemon Go is a particularly interesting example, since it relies on a planet-scale rendering of Earth. Perhaps Microsoft could eventually bring back Minecraft Earth after it shuts down in June — Mesh could finally make those early promo videos of collaborative block-building a reality. Microsoft’s new platform could also be a boon for virtual events. Cirque du Soleil co-founder Guy Laliberté is exploring how Mesh could power things like concerts, virtual performance or remote family events with the Hanai World Project. He wants to capture high-fidelity 3D experiences and venues so it actually feels like you’re there while wearing Mixed Reality gear — consider it a leap beyond grainy 360-degree VR videos.
Microsoft’s first two Mesh apps aren’t too surprising: It’s going to upgrade its AltSpaceVR apps with support for the new platform, as well as launch a Mesh preview app on HoloLens. The company expects to bring Mesh support to Teams, Dynamics 365 and its other products eventually. But the most intriguing aspect of Mesh is how third-party developers will use it. Microsoft says it’ll offer developers AI powered tools in the coming months to help them deal with things like spatial rendering, session management and holoportation.
Sullivan admits there’s a lot Microsoft doesn’t know yet. Just like the invention of the internet and smartphones, it’s hard to predict where exactly transformative new tech can take us. But after years of false starts, it seems like the future of virtual collaboration is finally coming into focus.
Be the first to comment