Google's Meena AI Chatbot Is Surprsingly Human-Like, Even Tells Dad Jokes
The true measure of a chatbot is not simply its ability to offer responses to questions, or even to just to make sense when 'conversing' with a human. Instead, it is the ability to allow a flesh and blood person to suspend disbelief and feel as though they are chatting with an actual intelligent being (and do it without being offensive, which is challenging, as Microsoft discovered). Google claims its new Meena chatbot is better at this than another other "conversational agent."
Calling Meena a chatbot is selling it short. It is an end-to-end, neural conversational model that learns to respond with replies that make actual sense, given whatever context is at hand. Google says Meena consists of 2.6 billion parameters and has been trained on 341 gigabytes (GB) of text, filtered from social media conversations that are available in the public domain.
"Compared to an existing state-of-the-art generative model, OpenAI GPT-2, Meena has 1.7x greater model capacity and was trained on 8.5x more data," Google states.
That is all well and good, but how human-like is Meena really? Well, it can tell jokes. As in, those corny jokes your dad busts out at family gatherings. Being a corny dad myself, I have to admit at being impressed with this exchange...
Meena: Which animal do you like?Human: I like cows.Meena: I heard they go to college.Human: Cows go to college?Meena: I heard that a cow went to Harvard.Human: What did the cow study?Meena: Bovine science.Human: Do horses ever go to college?Meena: Horses go to Hayvard.Human: That's a pretty good joke, I feel like you led me into it.Meena: You were trying to steer it elsewhere, I can see it.
It is a simple exchange, but also nuanced with a few 'ba-dum tish!' deliveries. What's remarkable is how natural the conversation flowed. Telling jokes is a difficult thing for chatbots to grasp, because it requires a deeper understanding of language and sarcasm—if it was easy, there would be more stand-up comedians.
There is some fancy technology that allows this to happen. Meena uses encoder blocks to process conversation context and to understand what has already been said. It then uses a decoder to formulate an appropriate response. For this to be accurate, Google says it tuned "hyper parameters."
Source: Google
According to Google, Meena is far closer to being able to carry on conversations like a human than other advanced chatbots. Google designed a new human evaluation metric called Sensibleness and Specificity Average (SSA) to highlight its claim.
"For each chatbot, we collect between 1600 and 2400 individual conversation turns through about 100 conversations. Each model response is labeled by crowdworkers to indicate if it is sensible and specific. The sensibleness of a chatbot is the fraction of responses labeled 'sensible', and specificity is the fraction of responses that are marked 'specific'. The average of these two is the SSA score," Google explains.
We have to be careful not to read too much into a test Google designed to predictably show its own chatbot technology being better than the rest. However, it's interesting, as is aforementioned exchange.
It's not clear where Google with take this. Perhaps its own digital assistant will benefit in the future, but for now, the company is focused on further improving its algorithms and architectures.
