Google plans universal translator with lip synchronization

Google presented a universal translator with lip-syncing at I/O 2023. The project is not yet fully developed and quite problematic. Google wants to prevent misuse by watermarking.

Transcription, translation, dubbing

The Universal Translator, which is still in the experimental stage, is a real all-rounder. It transcribes audio from videos, translates the transcript and dubs it again. What makes it special is the fact that it mimics the speaking style of the person doing the talking, and there is real dubbing. Thus, the new audio track is supposed to be adapted to the lip movements of the person speaking. Already the previous Google translator is able to transcribe and translate spoken words. The only new feature would be synchronization.

Google is aware that such a tool does not only bring advantages. For example, it could be used to put statements into the mouths of people that they have never made. To prevent such misuse of the tool, Google wants to watermark all Universal Translator products. This will make it clear at all times that the video was edited with the tool.

In addition to questions of abuse, there are also questions of use for the Universal Translator. It is questionable whether it will be of real use in everyday life. On the other hand, it could make sense to use it in media production. Here, however, the translator would have to deliver outstanding performance in order to keep up with previous translation and, above all, dubbing work. It is not yet clear if and when the product will be launched on the market.