UW researchers develop realistic, lip-synced video software


SEATTLE -- University of Washington researchers have developed a computer program that is both amazing and scary.

They can put someone’s words back into their mouth in a realistic, lip-synced video.

U.W.'s Paul G. Allen School of Computer Science and Engineering took hours of video of Barack Obama of giving speeches, mapped his facial movements into the program, and are now able to produce a video using only an audio clip of the former president talking.

The video is real to the point that it’s Obama delivering a real speech, but the words he says are entirely different, and may have come from a TV interview he did 25 years ago. The end result is a near perfect lip sync that really didn’t happen but looks as if it did.

It’s not so much the life-like quality in the video, it’s how fast the video can be made without animators.

“That’s the technological advancement, that we can do it automatically without any help from people” says Ira Kemelmacher-Shlizerman, an assistant professor at the school and faculty adviser to the project.

Hollywood and it’s digital animators could possibly do the same thing but it would take hours of manual work. The software tool the researchers develop does it without human involvement in a matter of minutes.

It learns the facial moments tied to sounds a person makes. A neural network first converts the sounds from an audio file into basic mouth shapes. Then the system grafts and blends those mouth shapes onto an existing target video and adjusts the timing to create a new realistic, lip-synced video.

“We’ve gotten two types of reactions,” says Kemelmacher-Shlizerman. One type of reaction is ‘oh my God this is so cool, this is a technology break through’ and so on. The other reaction is ‘oh my gosh how we going to believe anything’, ‘we are not going to be able to trust videos now’.

The team chose Obama because the tool needs hours of videos to learn from and there is plenty of the former President in the public domain. The tool is not being developed for commercial use says Kemelmacher-Sclizerman. She says the team wants to show how people need be wary of what they see nowadays.

“This is part of the reason why we show it to the public, to show that technology can enable this kind of thing,” says Kemelmacher-Shlizerman. “It's important to know that algorithms can be developed that can edit videos like this.”

It can seem like an ready-made formula to produce fake news in the software is used in a malicious manner.

“We just want people know what’s possible.”

close video ad
Unmutetoggle ad audio on off