Google’s Project Relate app is a new machine learning-based research app to make communication easier for people with speech impairments. The Project Relate app will be made available for beta-testing in Australia, Canada, New Zealand, and the United States. Google plans to test this out with native English speakers who are willing to participate.
In a blog post, Google said it is looking to help those with speech impairments, caused by conditions such as stroke, ALS, Parkinson’s Disease, or even a traumatic brain injury.
“We realised back in 2018 that speech recognition could be improved to be more accessible for people whose speech has been impacted by a condition. But regular speech recognition technology doesn’t always work as well for those who have atypical speech simply because we don’t have a lot of training data to train the algorithms on the examples,” Julie Cattiau, Product Manager at Google Research, explained over a call with select media.
In order to get to Project Relate, Google started a crowdsource data program back in 2019 called Euphonia, which gathered examples of how people with different speech disabilities sound like. Cattiau revealed that Google worked with “multiple partner organisations like ALS TDI, the Canadian Down Syndrome Society, and Team Gleason in the United States” to help them identify individuals who would participate in the project. Google eventually relied on over a million speech samples to create the app.
According to the video demonstrated by Google, a user with a speech impairment is able to talk to the app, which is then able to speak out her request to another user. The user is also able to talk to Google Assistant from the app, which can carry out the user’s request.
Those joining the early testing for Project Relate will be asked to record a set of phrases. Cattiau revealed that the app requires 500 examples from the user in order to work accurately and this can take anywhere between 30 and 90 minutes to create a personalised model.
“If after 250 phrases, we realise that the accuracy of the model is good enough, we release the model early, but it might not be the case for every user. It depends on the severity of their speech impairment,” she added.
The app will then use these phrases to automatically learn how to better understand the user’s unique speech patterns and give them access to the app’s three main features, which are Listen, Repeat, and Assistant.
With the Listen feature, the app transcribes the user’s speech to text in real-time, so they can copy-paste the text into other apps, or let people read what they wish to tell them. Repeat is a feature where the user talks to the app, and then the app repeats the same thing in a “clear, synthesised voice,” to those around them.
“With the Listen feature, which we demoed in the app, we start with the baseline speech recognition model that is also being used in G board. It is an on-device speech recognition model, so it doesn’t require any access to the internet in order to run,” Cattiau revealed, adding that Relate was using a lot of the technology that has already been developed at Google and personalising it for each individual.
With Relate’s speech models, Google is trying to cover certain domains in order to ensure that the model is accurate. “There are a number of phrases in the Relate app that are queries for the Google assistants and the goal behind it is to make the model more robust to queries that are supposed to be said to the Google Assistant. We also have everyday conversational phrases,” she said. Users of the app will also be able to create custom phrases that they would like to say to the Assistant. This Google says will help the app model become more robust and personal for each user.
The Assistant feature, of course, lets the speaker talk directly to the Google Assistant from within the Relate app to carry out different tasks. Google says it also worked with Aubrie Lee, a brand manager at the company, whose speech is affected by muscular dystrophy. “Project Relate can make out the difference between a look of confusion and a friendly laugh of recognition,” Aubrie said in the video.
When asked whether Google plans to extend this for other languages, Cattiau said that they are starting with English first and hoped to add support for other languages later. Japanese is one language they are studying actively at the moment for the app.