Digital technology, media and intellectual property
Random header image at GB Media

The built-in microphone and speakers of an iPhone can be used for far more than just talking.

November 21st, 2016 |  Published in Apple  |  2 Comments


Eric De Grasse
Chief Technology Officer

21 November 2016 – I was at a MIT Media Lab workshop last week and learned a very funky thing: controlling an app via sonar, generated by the phone’s original hardware.

This is made possible by software built by Wei Wang and Alex X. Liu from the Department of Computer Science and Technology at Nanjing University in China. It uses the phone’s onboard speakers to emit sound at frequencies between 17 and 23 kilohertz—toward the upper end of human hearing, and just barely audible to younger ears. By analyzing the reflected signals detected by built-in microphones, it’s possible to measure the proximity of an object, such as your hand, to within four millimeters.

Controlling a device without having to directly touch it isn’t a cheap gimmick. It’s useful anywhere there’s a mess, from a kitchen to an operating room. Efforts to create low-cost gesture control for computers have been proposed in the past (see “Leaping into the Gesture-Control Era”). But Wang and Liu think their feature could be cheaply baked into any modern smartphone without the need for any additional hardware.

All it takes is a wave of your hand

It’s not the first time that a smartphone has been used to perform echolocation. Shyam Gollakota from the University of Washington has used a similar approach as part of his FingerIO project, which investigated whether it was feasible to locate human hands using high-frequency sound and on-board hardware. But Gollakota is impressed by the work of Wang and Liu, because they’re able to use it for direct control of the phone. It’s “very cool,” he says, that the technique has been implemented to work in real-time.

There is a little lag in its response to human gestures. Wang says that the phone takes 15 milliseconds to detect and process movement. In practice that’s barely discernable, though, and the control itself is impressive. At first you feel compelled to move your whole hand, but with a little practice it becomes possible to achieve the same results by moving a single finger.

The app is currently a research project and not available in any app store. But the pair plans to turn it into an API that can be used by other developers to bake the echolocation system into apps on iPhones and Android devices. They reckon that it could end up being used to scroll Web pages, say, or turn pages of an e-book.

As I wrote a few months ago after we attended DLD Tel Aviv, those are much like the features being proposed as part of Google’s Project Soli. But the approach being taken by Google researchers is to build a dedicated radar chip that could be added to a device. While that provides more accuracy than the sonar technique, it also means integrating yet another technology into a smartphone.

Obviously hardware manufacturers could instead simply optimize the positioning of microphones and speakers about existing devices. They could even increase the upper frequency at which devices transmit and receive sound to achieve submillimeter resolutions—the hardware would still likely be cheaper than that required for radar.

Meanwhile, MIT tells me that Wang and Liu continue to improve their software in the lab. They hope to develop versions for smart watches and VR headsets, where interacting with a screen is difficult or impossible, as well as building new algorithms to detect individual fingers. They already claim that they’re able to track the motion of a hand accurately enough to identify characters being written with more than 90 percent accuracy.

About the author

Email | All posts by

"The mind that lies fallow but a single day sprouts up follies that are only to be killed by a constant and assiduous culture."
Latest Videos

Un aperçu de la FIC 2017 / A quick look at FIC 2017 (Lille, France)

Cybersecurity: a chat with John Frank, Vice President EU Government Affairs for Microsoft

From Legaltech NYC 2017: a chat with Andy Wilson of Logikcull

5G is coming ... and it's going to blow you away. Yes. Really.

The Internet of Things ... or the cybernetic consortia? (Part 1)

From the Mobile World Congress 2016: an introduction