Messing with Leap Motion and it's Interaction Engine
When we started using Leap Motion we knew we wanted players to be able to grab objects and throw them around with their hands. As it turns out the Leap Motion struggles a bit with this, but the developers know this and thus we have the Interaction Engine:
The Interaction Engine allows you to more easily pick up and grab objects using your virtual hands, and before we even started development we knew we were going to need this. However as we began playing around with it we found that people often had trouble actually grabbing things.
The problem with this was two-fold. First, because you could see your real hands in addition to the Leap Motion hands there was a disconnect between grabbing objects with your Leap hands and not your real hands.
The reason for this is that there's a distance between the Leap Motion and the camera itself. If you remember we had to MacGyver together our setup with elastic bands because the ZEDm doesn't support the HP Windows Mixed Reality Headset with its slightly-more-curved-than-normal front.
In this case the Leap Motion is placed directly below the camera, however that distance between the camera lenses and the Leap Motion is enough so that where you see your hands and where the Leap Motion sees your hands is different. We did account for this however and added an offset to the Leap hands position, but this causes its own problems.
When we rotate around the Leap hands slowly drift away from our hands and get out of sync. It's probably just the way that we're adding the offset, but we're running out of time till our next milestone so for the time being we're probably going to just have the Leap hands a farther distance away from your real hands. Kind of like mage hands.
For reference, our offset is just modifying the position of the Leap hands in LeapXRServiceProvider.cs on line 426. I'm sure we could make it work if we spent more time with it, but alas that's only one part of the problem.
The other part of the problem is that from what we've seen it's a little difficult to judge the distance of virtual objects in a real world space. Is it really far away or is it just really small? Figuring out where objects in 3D is hard. When Mario first made the jump to 3D they solved the same sort of problem by adding a drop shadow below him, but we can't really do that.
To help mitigate the problem we added a second, larger, collider that players can grab but won't collide with things. This works but it looks a little weird.
To fix this we simply have to specify the objects position when its grabbed to be set to the grasp point of the hand that grabbed it. This is done in InteractionBehaviour.cs itself around line 1323ish where the grasp movement is set using a new pose.
We still need to work on this to make it look smoother, but for now it'll do.
