Learning Neural Character Controllers from Videos and Motion Capture Data

Abstract

Computer games and Virtual Reality (VR) are not only entertainment, but becoming a novel way of communication, especially in the post-Covid world where reduced physical communication may remain as a new-normal. Character motion is one of the important factors in increasing the immersion of the users into the virtual world. Using neural networks for character controllers can significantly increase the scalability of the system - the controller can be trained with a large amount of motion capture data while the run-time memory can be kept low. The main challenge is in designing an architecture that can produce movements in production-quality and also manage a wide variation of motion classes and styles. In this talk, I will cover our recent development of neural network based character controllers that can be trained from both videos and motion capture data. By training our neural controller from both videos and motion capture data, the system can learn motion styles of a wider population, increasing its capabilities to produce various motion types and styles for real-time animation purposes. Using our system, a wide variation of movements such as walking, running, side stepping and backward walking can be synthesized for a wide variation of characters in real time. In the end of the talk, I will discuss the open problems and future directions of character animation.