Windows Mixed Reality Portal and ZEDm Camera Troubleshooting

Having a bunch of things elastic'd to a headset may look neat, but working with 3 pieces of technology gave us no end of issues. In particular the ZEDm camera presented us with a number of obstacles. For example, while tracking is usually pretty good sometimes it would get slightly displaced. This would happen most frequently when someone walked in front of the camera, displacing whatever was on screen and effectively kidnapping them. 

This is because of the ZED's own motion tracking. When an there's a large object in the cameras frame, the camera things that the object is actually the world and moves the things in the scene to where it thinks they should be in space. If the object is moving, then so does everything else. The first thing we did to try to get around this was simply disable the ZED's tracking.

The ZED's motion tracking is actually optional, and if you disable it the camera won't do any tracking on it's own, but rather use the headsets tracking. Seemed like a good idea and our tests showed that it would help us get around the kidnapping so we decided to try it out.

Of course nothing is simple with Windows Mixed Reality. While we had solved the kidnapping issue our spatial tracking wasn't working working any more, meaning even if you moved through the world the in game camera stayed in the same spot.

This resulted in everything being just slightly out of reach and and as you frustrated players to no end. After a lot of scratching our heads we found something of an answer in the Windows Mixed Reality set up. When you plug in your HP Windows Mixed Reality (HPMR) headset in for the first time you're given a series of steps to set things up. One of these steps is setting up a boundary for your play space. 

Previously when we did this we set up for seated experiences because we didn't have a whole lot of room in our workspace and it seemed to work fine without setting up a boundary so we didn't think too much of it. 

I don't think we ever managed to find a clear answer online anywhere, but from our tests it seemed like in order for the HPMR headset's spatial tracking to work we needed to have a boundary set up for standing experiences. If I had to guess I would say that this is due to the headset having inside out tracking without having any base stations compared to how the how Oculus or Vive have both external sensors.

Switching over to standing experiences sort of worked. It worked sometimes, but we found that more often than not it wouldn't. This was usually due to the boundary we had set up being erased or deleted, which seemed happen whenever we turned off the system. It was also a headache to troubleshoot the thing because a lot of the time you would find answers for the HoloLens, which is also Windows Mixed Reality and to my understanding uses some of the same stuff as the headsets that we have.

It didn't help that the Mixed Reality Portal would crash sometimes and give us an error code that didn't mean anything. We eventually found out that this was because of the version of Windows that we were on, and most of the issues we ran into were resolved when we upgraded from Windows 10 version 1803 to version 1809.

Even after we updated Windows versions we were still having issues with the headset's spatial tracking, and our efforts to find answers hadn't gone anywhere. At this point I figured that if relying on the HPMR headset's tracking was going to give us just as much trouble as relying on the ZED's tracking we might as well try looking into the ZED again since we were getting nowhere with Windows. 

This brought us back to the initial issue of kidnapping. My first idea to get around this was simply to give players the ability to reset the scene so that they could deal with kidnapping and drifting once it became an issue for them. I had tried this previously by just reloading the scene but many of the assets we were using use DontDestroyOnLoad(), which meant that reloading the scene would create duplicates which caused no end of errors. 

This time instead of resetting the scene I figured I would simply set everything back to its default state. This worked for the most part except for one tiny thing:

Resetting everything caused the in game camera to rotate about 90 degrees. With no idea why this was happening I sent a message to Stereolabs, the creators of the ZED camera who had been helpful so far. What we ended up doing was making the camera rig a child of another object then, after we reset the camera by enabling it and disabling it using the built in ZED methods, we would set the parent transform to have the inverse of the camera rig's position and rotation. 

This worked out, but as always solving one problem only leads to another and now our scene was doing this:

This isn't normally the kind of thing you want to see, let alone in a game with a headset where you have to worry about potentially making players sick. We're still dealing with this issue, and from my back and forth's with the guys at Stereolabs it seems that it might have something to do with the camera's IMU calibration.