The Wonderful World of Windows Mixed Reality
So over the past few weeks I've been working with HP's Windows Mixed Reality Headset to try and make nifty future type games that kind of look like this:
Over the time that I've been working with the Mixed Reality Headset I've learned a few things that might be helpful to someone else, so I'm going to put them here.
The first thing that you'll probably notice is that this is a "Mixed Reality" headset as opposed to your normal Virtual Reality (VR) headsets and you might be thinking that its some sort of combination of Augmented Reality (AR) and VR (which is just more AR if you think about it.).
In reality though the HP Windows Mixed Reality Headset is just a normal VR headset, much like the Oculus Rift and HTC Vive. It works with your VR games, and does all the VR things that normal VR does. The reality of Windows Mixed Reality is that Windows Mixed Reality is a product line that covers a wide range of wearable technology from VR headsets to the HoloLens. This particular headset is a VR headset under the Windows Mixed Reality brand.
But what about those camera looking things on the headset? Can't I use those to turn it into a nifty AR headset like a HoloLens? Well those camera looking things on the front of the headset are actually sensors for the controllers, much like the sensors that come with the Rift and Vive. While you can't use them as outward facing cameras (at least to my knowledge) they do make setting up the headset a lot simpler than a Rift or a Vive since all you have to do is plug in the headset and your good to go. No worrying about needing extra plugs for more sensors or trying to figure out where to fit sensors on your desk so that they'll calibrate properly.
SO if the Mixed Reality Headset is actually just a VR headset, how do we do all the fancy real world jazz hand stuff? Well, the first thing that we're going to need is a camera.
Enter the ZEDm camera, a camera specifically designed to work with VR headsets to give them AR capabilities. What's neat about the ZEDm is that it can sense depth, which gives it a leg up compared to using a normal camera for AR as it can simulate virtual objects being obscured by real world objects. The ZEDm is also quite developer friendly and has plugins for both Unity and Unreal, and supports both Rift and the Vive.
Wait, Rift and Vive? What about our HP Windows Mixed Reality Headset (or HPMR for short)? Well, as it turns out the camera itself works just fine with the HPMR, but the issue is in mounting it as the HPMR has a more curved visor compared to the Rift and Vive. That's not to say that it can't be done, but we had to resort to a more... creative solution.
So now we can see, but still need to use the HPMR controllers to be able to interact with things. To get around we add yet another piece of tech, the Leap Motion Controller.
Leap Motion Controller is a small sensor that detects and maps your real life hands to virtual ones as seen above. Normally you would be in front of a screen and see the virtual hands that you're controlling, or use it in VR as an alternate to using controllers, but because we have the ZEDm we can instead use the Leap Motion Controller to convert our hand movements in AR to input in game.
Wow, that's a lot of stuff strapped to a headset. With all these different cameras and sensors does this stuff actually work? Yes. Yes it does.
The Leap Motion Controller also has plugins for both Unity and Unreal, and from what we've tested it works with the ZEDm fairly well.
So does this mean that in the future that everyone's going to be strapping a bunch of gadgets to their headsets to be able to poke virtual anime girls? Probably not, this is just how we did it. The gif at the start of this post isn't just some mock up, that's Leap Motion's latest project, North Star, an AR headset that's built to do essentially what we're doing. At the time of writing this these headsets aren't commercially available, but the project is open source so if you wanted to you could theoretically put your own headset together if you had the know-how.
For us though it looks like it's going to be duck tape and elastic bands.
 |
| GIF by Keiichi Matsuda, showing off Leap Motion's Project North Star |
Over the time that I've been working with the Mixed Reality Headset I've learned a few things that might be helpful to someone else, so I'm going to put them here.
The first thing that you'll probably notice is that this is a "Mixed Reality" headset as opposed to your normal Virtual Reality (VR) headsets and you might be thinking that its some sort of combination of Augmented Reality (AR) and VR (which is just more AR if you think about it.).
In reality though the HP Windows Mixed Reality Headset is just a normal VR headset, much like the Oculus Rift and HTC Vive. It works with your VR games, and does all the VR things that normal VR does. The reality of Windows Mixed Reality is that Windows Mixed Reality is a product line that covers a wide range of wearable technology from VR headsets to the HoloLens. This particular headset is a VR headset under the Windows Mixed Reality brand. But what about those camera looking things on the headset? Can't I use those to turn it into a nifty AR headset like a HoloLens? Well those camera looking things on the front of the headset are actually sensors for the controllers, much like the sensors that come with the Rift and Vive. While you can't use them as outward facing cameras (at least to my knowledge) they do make setting up the headset a lot simpler than a Rift or a Vive since all you have to do is plug in the headset and your good to go. No worrying about needing extra plugs for more sensors or trying to figure out where to fit sensors on your desk so that they'll calibrate properly.
SO if the Mixed Reality Headset is actually just a VR headset, how do we do all the fancy real world jazz hand stuff? Well, the first thing that we're going to need is a camera.
Enter the ZEDm camera, a camera specifically designed to work with VR headsets to give them AR capabilities. What's neat about the ZEDm is that it can sense depth, which gives it a leg up compared to using a normal camera for AR as it can simulate virtual objects being obscured by real world objects. The ZEDm is also quite developer friendly and has plugins for both Unity and Unreal, and supports both Rift and the Vive.
Wait, Rift and Vive? What about our HP Windows Mixed Reality Headset (or HPMR for short)? Well, as it turns out the camera itself works just fine with the HPMR, but the issue is in mounting it as the HPMR has a more curved visor compared to the Rift and Vive. That's not to say that it can't be done, but we had to resort to a more... creative solution.
So now we can see, but still need to use the HPMR controllers to be able to interact with things. To get around we add yet another piece of tech, the Leap Motion Controller.
Leap Motion Controller is a small sensor that detects and maps your real life hands to virtual ones as seen above. Normally you would be in front of a screen and see the virtual hands that you're controlling, or use it in VR as an alternate to using controllers, but because we have the ZEDm we can instead use the Leap Motion Controller to convert our hand movements in AR to input in game.
Wow, that's a lot of stuff strapped to a headset. With all these different cameras and sensors does this stuff actually work? Yes. Yes it does.
The Leap Motion Controller also has plugins for both Unity and Unreal, and from what we've tested it works with the ZEDm fairly well.
So does this mean that in the future that everyone's going to be strapping a bunch of gadgets to their headsets to be able to poke virtual anime girls? Probably not, this is just how we did it. The gif at the start of this post isn't just some mock up, that's Leap Motion's latest project, North Star, an AR headset that's built to do essentially what we're doing. At the time of writing this these headsets aren't commercially available, but the project is open source so if you wanted to you could theoretically put your own headset together if you had the know-how.
For us though it looks like it's going to be duck tape and elastic bands.
