One aspect of video calling that many of us take for granted is how they can switch between feeds to highlight who's talking. Great, if talking is the way you communicate. Silent speech like sign language doesn't trigger those algorithms, unfortunately, but this Google search could change that.
It's a real-time sign language detection engine that can tell when someone is signing (instead of moving) and when they're done. Obviously it's trivial for humans to tell this sort of thing, but it's harder for a video calling system that's used to just pushing pixels.
A new paper from Google researchers, presented (virtually, of course) at the ECCV, shows how it can be done efficiently and with very little latency. The point would be nullified if sign language detection worked, but it resulted in delayed or degraded video, so their goal was to make sure the model was light and reliable.
The system first runs the video through a model called PoseNet, which estimates the positions of the body and limbs in each frame. This simplified visual information (essentially a stick figure) is sent to a model trained on pose data from videos of people using German sign language, and he compares the live image to what he thinks the signature is.
This simple process already produces 80% accuracy in predicting whether a person is signing or not, and with some further optimization it achieves up to 91.5% accuracy. Considering how "active speaker" detection on most calls is just so-so in telling if a person is talking or coughing, those numbers are pretty respectable.
In order to work without adding a new "person is signing" signal to existing calls, the system does a clever little trick. It uses a virtual audio source to generate a 20 kHz tone, which is beyond the reach of human hearing, but noticed by computer audio systems. This signal is generated whenever the person is signing, making speech detection algorithms to think who are talking loudly.
Right now it's just a demo, which you can try here, but there doesn't seem to be any reason why it can't be integrated directly into existing video calling systems or even as an app that supports them. You can read the entire document here.
