NVIDIA's Choreographer AI Model Automatically Kicks Up Thriller Dance Moves Set To Music
Is there anything that artificial intelligence does not do these days? Sure there is, but the list is growing smaller by the day. In case you were wondering, AI even has dance moves that it came with by itself. That's right, through the power of deep learning, researchers at NVIDIA collaborated with the University of California, Merced, to develop a deep learning-based model that can automatically compose new dance moves.
"This is a challenging but interesting generative task with the potential to assist and expand content creations in arts and sports, such as a theatrical performance, rhythmic gymnastics, and figure skating," NVIDIA stated in a paper on the subject.
The team collected dance videos showcasing three categories of dance, including ballet, Zumba,and hip-hop in order to train their generative adversarial network (GAN). The accumulation dance videos resulted in 71 hours of dancing footage, spread out across 361,000 clips. So, there was quite a bit of dance data to feed into the GAN.
This deep learning method is based on a decomposition-to-composition framework. That's to say, the GAN learns how to move from the data its fed, and then how to compose dance moves on its own. It may sound simple, but there is a lot involved here—the resulting dance moves the GAN comes up have to match the type of music being played, stay in beat, and look natural.
"Extensive qualitative and quantitative evaluations demonstrate that the synthesized dances by the proposed method are not only realistic and diverse but also style-consistent and beat-matching," the researchers stated in their paper.
Naturally, NVIDIA tapped its Tesla V100 GPUs for this task, which worked in conjunction with the PyTorch deep learning framework. And for the pose processing, the researchers leveraged OpenPose, an open-source, real-time multi-person system developed by Carnegie Mellon University.
It's quite the impressive feat, and could help professional dance teams come up with new moves and choreography, among other uses. If you want to dive into all the nitty-gritty details, you can check out the detailed whitepaper (PDF).
