from Hacker News

Open-source tool translates and dubs videos into other languages using AI

by oldcai on 8/13/24, 12:15 PM with 116 comments

• by netsharc on 8/13/24, 3:51 PM

  Huh, if it's good enough, movies/TV shows dubbed with AI-clone of the original voice would be great (if we can ignore the ethics of using the actor's voice and the loss of work for the dubbing companies and actors).

  For example here's how weird Friends is in German: https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian: https://www.youtube.com/watch?v=wO5qTzvyQ1s

  Can AI detect the emotional tone of sentences yet, and recreate it in the target language?

• by cubbic on 8/13/24, 2:48 PM

  Oh I made something similar but for Netflix!

  https://github.com/cubbK/dubbing_ai_netflix_client

  I want to learn swedish and because there are so few dubbed movies in Swedish I take the subtitles(Netflix is good at having subtitles in different languages) and text-to-speech it :)

• by vouaobrasil on 8/13/24, 2:06 PM

  Unlike other methods of automation, AI is replacing human beings too fast. And before you say, "new jobs will be created" -- look at history. After the computer, new jobs have been created, but what kind of jobs? Every year, we are becoming more entwined in wage slavery as the wealth accumulates at the top and jobs become more meaningless.

  So, no, new jobs will not be created, except the kind of jobs that crush the human spirit into oblivion so that the rich tech oligarchs can play God.

• by skummetmaelk on 8/14/24, 7:32 AM

  Localization and dubbing is a sad endeavour. By trying to accomodate everyone's individual preference for information transmission we accomplish nothing more than reducing our ability to understand each other in the long run.

  Having a Babelfish is all well and good. Until it stops working, and you realise no one can understand each other any more.

  Ironically localization is often pushed by well meaning Americans who only speak one language. "Oh, you're in a French speaking region. You MUST want French language. Let me force it down your throat while I prance around virtue signalling about how inclusive we are"

• by LeoPanthera on 8/13/24, 7:21 PM

  For years I've wanted this for live TV. Even just subtitles would be amazing. I've always wanted to be able to watch news TV from other countries.

• by codedokode on 8/13/24, 10:34 PM

  As I understand, it first extracts text from original video into subtitles, translates them using external LLM, and then converts text to speech. All of this is done using thrid-party solutions, and the project seems to be just a GUI app that allows to integrate them.

  You obviously cannot use this to translate songs or movies because this method loses important information like voice, intonation, etc.

  So it is still better to use subtitles.

• by mrtksn on 8/13/24, 10:02 PM

  Back in high school, when I got my first PC a plumber came over to fix some stuff and when he saw the computer he got excited and asked some questions and one of the questions was “how do you translate the VCD with this, I have a movie to watch but hate subtitles”.

  I was like “silly dude doesn’t know how computers work” but maybe I was the silly one who can’t dare to imagine how something like that can work.

• by gagabity on 8/14/24, 2:19 AM

  Cool what languages can it do?

  Yandex browser does the most impressive version of this and for free but only to Russian I believe, its quite amazing it does appropriate different voices and follows the correct intonation for everyone, just takes a few seconds for a YT video.

• by nsonha on 8/14/24, 4:18 AM

  This could be useful in combating fake news. In many videos especially in political news, foreign languages are dub over with sometimes nuanced translation that can skew audiences to (mis)understand the content in certain ways.

• by cyberax on 8/13/24, 9:52 PM

  I would pay a lot for a tool that removes the freaking laugh track from videos.

  I just physically can't watch them. I wanted to watch the Blackadder series, but I couldn't even get through one episode.

• by lossolo on 8/13/24, 6:18 PM

  Based on english doc it seems it's not dubbing but voice over.

• by alphabetatheta on 8/13/24, 8:48 PM

  Some of my friends have been using: https://app.vivalabs.ai/ as a more managed/paid service recently

• by underdeserver on 8/14/24, 10:48 AM

  https://deepdub.ai/ have been doing this commercially for a while.

• by hulitu on 8/14/24, 6:49 PM

  > Open-source tool translates and dubs videos into other languages using AI

  Is there any assessment about how good the translation is ?

• by CyberDildonics on 8/13/24, 4:42 PM

  Is this using an open source text to speech model or is it going out to some other internet service?

• by paulkon on 8/14/24, 1:13 AM

  Is there an open source speech-to-speech model which retains intonation, cadence and delivery?

• by randomgiy3142 on 8/13/24, 4:37 PM

  Because translations are copyrighted so it is complex to get legal rights for them.

• by ewuhic on 8/13/24, 11:38 PM

  This one does dubbing, but is there an equivalent tool for subs?

• by cyanydeez on 8/13/24, 11:51 PM

  So uh, what's with AI products throwing out the gold standard in testing these claims.

• by ranger_danger on 8/13/24, 3:05 PM

  An English README would have been nice