A wearable device that uses sonar to build facial expressions | technology

People may not realize how smart their wearables are, what information they receive about you, and what companies can do with that information.

Cornell University researchers have developed a headset that receives sound echoes from the cheeks, transforms the echo into an avatar of a fully moving face, and transmits the speaker’s facial expressions as he or she speaks.

The team described what can be achieved from the system called “EarIO” – a “low-power acoustic sensor for continuous tracking of detailed facial movements” – in an article published in the “Proceedings of the ACM” the ACM) on interactive, mobile , and ubiquitous wearable technologies, in this July issue.

Like a ship sending out sonar pulses

The team, led by Assistant Professor of Information Sciences Cheng Chang and Professor of Information Sciences Francois Gembertier, designed an Airio system that transmits facial movements in real time to a smartphone and is compatible with commercially available headsets for hands-free wireless video conferencing. .

In the press release published on Cornell University’s website on July 19, Zhang says that the devices that track facial movements using the camera are “large and heavy and require a lot of power, which is a big problem for wearable devices”. “It is also important that it captures a lot of private information,” he added, stressing that face tracking through voice technology can offer better privacy, affordability, convenience and better battery life.

The newly invented Air IO works like a ship that emits sonar pulses. A speaker on each side of the earpiece sends audio signals to either side of the face, and the microphone picks up the echo. When the wearer speaks, smiles or raises an eyebrow, the skin moves and stretches, changing the echo. The researchers’ deep learning algorithm uses artificial intelligence to continuously process data, translating changing echoes into full facial expressions.

The innovative earpiece receives the sound and turns the echo into an avatar (communication sites)

The power of artificial intelligence

“With the power of artificial intelligence, the algorithm finds complex links between muscle movement and facial expressions that the human eye cannot identify,” says researcher Kee Lee, PhD student in the field of information science. “We can use it to derive complex information. which is difficult to capture, the entire facial interface”.

Previous attempts by Zhang’s lab to detect facial movements with camera headsets reconstructed the entire face based on cheek movements as seen from the ear.

By collecting audio instead of data-filled images, smart devices can communicate with the smartphone through a wireless “Bluetooth” connection, while maintaining the privacy of user information. The photos require the device to connect to a Wi-Fi network and send data back and forth to cloud storage, which can leave it vulnerable to hackers.

“People may not realize how smart wearables are. What this information says about you, and what companies can do with that information,” Guimpretiere says.

The face is recognized and reconstructed by the mobile phone camera (network sites)

All information is always under your control

With facial images, a person can also infer feelings and actions. “The aim of this project is to make sure that all information that is considered so valuable for your privacy will always be under your control,” explained Gambretier.

Using audio signals also consumes less energy than recording images, and the AIRIO system uses 1/25 the energy of another camera-based system previously developed in Zhang’s lab. Currently, the headset lasts about 3 hours with a wireless earphone battery, but future research will focus on extending the usage time.

The researchers tested the device on 16 participants and used a smartphone camera to verify the accuracy of facial simulation performance. Initial experiments show that the device works while users are sitting and walking around, and that wind noise, road noise and background noise do not interfere with its audio signals.

In future versions, the researchers hope to improve the device’s ability to adjust for nearby noise and other disturbances. “The acoustic sensing method we use is very sensitive,” said information science doctoral student Ruidong Zhang. “It’s good, because it can track very precise movements, but it’s also bad, because when something changes in the environment, or when your head moves a little, we record that too.”

The Air Io device works like a ship sending sonar pulses (communication sites).

One limitation of this technology is that before first use, an EarIO 32 device must collect a full minute of facial data to train the algorithm. Zhang said, “In Ultimately, we hope to plug and play this device.”

Leave a Comment