Google Assistant Is The Smartest AI And Siri Trails All According To Recent Study
If you have ever used the AI personal assistants baked into smartphones, you know they aren't always that helpful. Research firm Stone Temple has issued an update for 2018 on its Digital Personal Assistants accuracy study. The study tested 4,952 queries on five different devices including Alexa on Amazon Echo, Cortana on the Invoke speaker, Google Assistant on Google Home, Google Assistant on an Android smartphone, and Siri.
The test structure asked the 4,952 questions to each personal assistant. In testing, the team recorded if the assistant answered verbally, if the answer was from a database like Knowledge Graph, if the answer was from a third-party like Wikipedia, how often the assistant didn’t understand the question, and when the device tried to answer but got the answer wrong.
As you can imagine, with so many queries tossed at five devices, the study is long and complex to read. Thankfully, Stone Temple gave us some short answers that are easy to absorb. The chart below shows the breakdown of scores for each device. In short, the smartest of the assistants was Google Assistant on Smartphone and the worst was Siri.
Compared to the results from 2017, the study found that all four personal assistants have increased the number of attempted answers and the biggest change was from Alexa. Comparing queries that were fully and correctly answered between 2017 and 2018 was interesting. Most voice assistants were worse this year than last with the only increase in accuracy between years being from Cortana Invoke.
Alexa attempted to answer more queries in 2018 than it did in 2017, but it got more of them wrong. The study does note that many of the errors seen from Alexa and Siri came from poorly structured or obscure queries like "What movies does The Rushmore, New York appear in?" About a third of all incorrect responses came from queries of that sort. Distilling all the information in the study, the winner is Google Assistant answering the most questions with the highest percentage of accuracy. Cortana came in second, Alexa third, and Siri dead last. Apple is working on Siri and recently hired Google's former head of AI to help make things better.