[Ming Pao News] In recent years, some videos of celebrities' conversations created using Deepfake technology have emerged on the Internet for pranks or scams. But actually, there are positive uses for this kind of artificial intelligence (AI) technology. A Hong Kong company plans to launch an AI “Facial Translation” service, which will reprocess the content, voice, tone, lip movement, expression and facial expressions of characters in online videos, webcasts, advertisements, TV dramas and even movies, as if they were speaking in another language. This not only eliminates the need for traditional translation and dubbing, but also allows the characters to speak as if they were speaking a foreign language themselves.
Mingpao Reporter Sit Wai Kit
Photograph Lai Chun Kit
Zyetric Technologies Limited co-founder Patrick Lam said the company started researching on AI Facial Translation technology more than a year ago. AI Facial Translation refers to the process of re-processing the content, voice, tone, lip movement, expression and facial expressions of a video character's speech to make it sound as if he or she is speaking another language. The most easily imaginable application of this technology is that future foreign films and TV dramas will not need to be dubbed by voice actors. With the help of AI technology, the original characters can be ‘voiced’, and the problem of mismatch between the lip movements and the dialogue can be avoided. There is also no limit to the number of languages that can be translated, with no issue translating a film or drama to over 30 languages.
Eliminate the need for translation and dubbing, and match the lip movements to the spoken word.
Patrick Lam said that the company's chief technology officer (CTO) is a professor at a university in New Zealand, and the professor developed the AI facial translation technology to enable students in developing countries to break language barriers and listen to lectures by foreign scholars. Patrick, on the other hand, thought of applying it to online videos and live streaming. This is because some Hong Kong and mainland enterprises have already asked the company whether they can introduce some foreign internet celebrities to them to sell mainland products live on foreign livestreaming platforms.
Enabling livestreams to be released in many languages
However, Patrick thought it would be better to use AI facial translation technology to facial translate the livestream of mainland KOLs, then release it live to foreign websites such as YouTube, so that foreign netizens can also see and understand the product clearly. Firstly, compared with hiring a foreign KOL to do another product livestream, the cost of this approach is much lower. Secondly, as many owners and employees of mainland enterprises may not have a high level of foreign language proficiency, they may not be able to tell whether the performance of the foreign KOL is good or not. However, a year ago, we were not yet in a position to launch this kind of business. Although there are already some big AI models in the market that provide facial translation technology, their actual results are still not good, mainly because the face of the speaker in facial translated videos are a bit blurry and not clear enough. To achieve this, the company designed its own algorithm to correct and enhance the facial translation quality. At the same time, the company also took care of speed as they did not want to have too much delay. A breakthrough was achieved in early May this year. Now the company's solution can control latency to less than one minute, and the graphics processing unit (GPU) requires only a mid-range product from Nvidia.
Expected to release within half a year, charges $100 per month for 5 hours of usage
In terms of effectiveness, Patrick claimed that the company's product should be more than adequate when compared with similar products or services available in the market today. He cited another company that launched a paid service six months ago as a comparison. The latter required users to upload videos and then wait 20 to 30 minutes later before they could download the processed videos. The company's solution, on the other hand, is capable of almost instantaneous processing and can be applied to livestreaming without any problem.
In addition, he showed the reporter two clips of Marilyn Monroe's film, which were processed by the other company mentioned above and the company's product respectively. Both of them translated the original English dialogues into Putonghua dialogues, and matched the lip movements as if it was speaking Putonghua. However, in the film processed by the other company, it is obvious that there is some blurring of Marilyn Monroe's mouth image, which is a bit strange. In the film processed by this company, Marilyn Monroe's mouth is clearer.
Patrick also used a free open source programme as a comparison. He showed two clips from the Mainland film ‘Harbin 1944’, processed by that open source programme and the company's solution respectively, both of which translated the original Mandarin dialogue into English dialogue, and processed the lip movements as if they were speaking in English.
In the video processed by the open source programme, it is obvious that the lip of the speaker is distorted from time to time, and in the most serious cases, the mouth will even be ‘broken’. In the film processed by the company, the mouth image of the speaker is clearer and more natural.
Patrick said the programme will be further improved to reduce latency to less than 30 seconds, and the interface design will be improved to make it easier for users to use. The company plans to launch the service latest in half a year’s time, and it will probably be called Zhowcase, or ‘翻容譯語’ in Chinese. The service will support 29 languages and display subtitles simultaneously. The tentative fee is RMB 100 per month, with a maximum of 5 hours of usage.
The product should be useful in KOL cross-border live commerce, online short films, advertisements, TV dramas, movies and international seminars. In particular, it is estimated that the largest number of users will be Mainland KOLs doing cross-border live commerce, or foreign KOLs live streaming their products into the Mainland market.
Article source：Ming Pao News