Reducing the barrier to VR gesture control

High-end Virtual Reality (VR) headsets such as the HTC Vive, PSVR and now the Oculus Rift, can all be used with a gesture controller. The big problem with delivering VR solutions to OU students is that they cannot be expected to own expensive equipment, and the kinds of advanced interactions that gesture controllers enable are therefore unavailable for us to use.

Google Cardboard HMDs do a great job of creating a very low barrier into the world of VR, but the amount of control they give is very limited; at best they contain a single button that, when used in combination with a gaze, can form the VR equivalent of a mouse pointer.  This works well in lots of situations but doesn’t really enable any sort of manipulation of virtual objects.

This led me to experiment with using the image tracking concepts used in augmented reality (AR) and combining them with a VR world where the AR trigger images form a collection of tools that could be used in the virtual world. This opened up a whole new area of possible interactions within VR experiences using a combination of Google Cardboard and held controllers that could be printed out on a student’s printer.

The tracking image was printed out on A4 paper and folded until it was about 6″ x 4″ so that it eventually looked like this.

I then built a couple of working examples in Unity showing how this could be used in VR. This example shows a user manipulating objects using a tool that can pick up an object (the one that looks like a matchstick) and another tool that can knock them off (the hammer).  This enables any kind of 3D object to be held and positioned in a virtual space and moved around.  Applications for this could include using scientific apparatus or design/modelling.  The tools can be picked up and put down by simply changing pieces of paper so that a whole suite of tools could be available to students.

A very simple boxing game where the tracking images become boxing gloves

The video below shows Nathan trying out the boxing game.

Using pieces of paper as controllers obviously has some strong limitations. For example, the controller can be moved too close to the user’s face so that the camera loses tracking.  You also need to be looking directly at the tracking image at all times.  The quality of the tracking could probably be improved by using card instead of paper or by using a cube instead of a flat piece of paper, which should also help the camera keep sight of the image if the user rotates the controller in their hand.

In general, this approach to VR control had varying results. Situations where the user needs to pick up objects and have very fine control over them worked particularly well.  Objects can be moved and rotated in a very natural way, and the tendency to move slowly helped the camera track the image more consistently.  In the boxing example, the user often feels compelled to try and take quick jabs at their opponent.  While this is amusing to watch for those not in the VR world, it did cause loss of the tracking image in quite a few instances.