VR Blink Detection

In December 2015 I was invited to Granada Gaming, a Video-games festival held in my home town, to talk about VR and my interaction experiments. Very exciting times!

I had to give two talks: the first one was oriented to all the professionals (coders, artists, journalists) where I explained some of the decisions I took while creating Apnea (my always in-progress  videogame). The second talk was for a general public and for this one I wanted to talk about something that seems to concern a lot of people: VR limitation and why FPS won't work very well at the beginning.

I won't cover the whole talk here as many of the interaction experiments showcased can be found already in the "VR Wireless" post and my github page, but I created something I thing is a cool hack to solve one of the main trends in VR movement: the blink transition.

Blink transition

Moving the FPS way in VR with a gamepad causes nausea to pretty much 45% of the players (nDreams CEO dixit) so companies are coming up with a lot of creative solutions to workaround this problem.
One of the main solutions is called blink transition, and it has been popularised by great experiences such as Epic's Bullet Train and GearVR game Land's end.

With this solution you basically look at the point where you want to move and just press a button that will teleport you there. This transition sometimes happen by closing and opening virtual eyes in front of the user or simply lerping him there really fast.

While it's true that this technique usually does not cause any nausea it is a big presence breaker for me: looking at a point and pressing a button to suddenly be there is not a very natural way of moving... how could I improve it?

Blink detection

What if I could detect if the user is truly blinking? Instead of closing some fake eyes in front of the player and having to press any button it will feel like a super power, think about NightCrawler from the X-men. This will suddenly make the movement system way more natural and amazing, improving the presence factor a lot while being more enjoyable and comfortable.

Some experimental HMDs are starting to support eye tracking, the most notorious one being the FOVE HMD: if you can track the user's eye it has to be trivial to detect if it's closed. But all I had in my hands was a GearVR for Galaxy Note 4, some hacking was needed.

I realised that the GearVR has some covers in the front to prevent scratching the phone when attached. If you remove any of this plastic-covers.. voilá! there is a screw!

Front facing you, the top right screw just happens to be exactly inlined with the Note 4's front camera! I inspected the Galay S6 GearVR and seems to be the same case. -What a lucky coincidence!- I thought - if only I could use this camera to track the user's eyes...

Then a new idea came to my mind: I don't need to track the eye, I just want to know if it is opened. When you look at someone's eyes they happen to reflect a lot of light, but when you close them your skin is not near as reflective.

If I create a very bright scene my eye will reflect a lot of light and maybe some will reach the front facing camera. But that was not the case, the camera (even thought I removed the screw right in front of it) was still too far away and angled to read this subtle amount of light. A visit to the store will solve this: 10 cm (2£) of optical-fibre: If I put one end in the screw's hole facing directly the camera and the other end facing the eye I can redirect the light from the screen -> to the eye -> to the cable -> to the camera!

The camera end of the cable

The eye end

All that rested was to create an ultra-simple script that will read the amount of light received: just add the value of all pixels and check if it's higher than a threshold. Thanks to Unity and Android I can control the resolution and discovered than using a 20x20 texture was good enough so the calculation was lightning fast... even detecting the fastest blink. For obvious reasons I set that the eye had to be closed for around 500ms or the user will be travelling surely more than desired.


Here is a video. The black square is the camera input, you can see the amount of light received (multiplied by a very very big factor) and how it becomes pretty much black when I close my eyes:

My last problem was with Android 4.4.4 and the camera sensitivity. It seems that they added some enhancements in Android 5, but with my version I could not control the shutter, so sometimes it was not sensitive enough to read the light. The solution i quite lame, just sightly remove the phone from the HMD (while keeping the USB connected) so a lot of light comes inside the camera this will adjust the shutter from you and everything will be fine!

This was so simple/cheap to do  (and so useful!!) that I would love future HMDs to come at least with some sort of blinking detection. VR movement sure is an interesting problem and when deciding between presence or not-sickening every little can help.