This is what I'd like to build (the tutor part at least, not the VR game part yet). I'm planning to extend my current English only rough prototype[1] to support Mandarin. (I happen to be learning Mandarin myself at the moment, and there are a bunch of open source bilingual Mandarin LLMs and speech synthesizers from China to choose from.)
I think a lot of people are working on similar things right now. I know of one called http://yourteacher.ai
Is there a high quality speech synthesizer (ideally local) for Mandarin you have found? There are some subtleties with tone sandhi rules and how they interact with prosody that I feel are lacking with current TTS voices I’ve tried.
I love the idea of LLMs being super-efficient language tutors. And you have a good point; coming soon: "We've been getting a lot of these tourists here lately, they're eerily fluent, but all seem to have the same minor speech impediment" (read: messed-up weights in a commonly used speech model).
I've been using ChatGPT 4 to translate and explain various texts in Mandarin and it's been very on point (checking with native speakers from time to time, or internet searches). As expected, it has trouble with slang and cross-language loanwords from time to time. However for languages with much lower information online, it hallucinates like crazy.
> coming soon: "We've been getting a lot of these tourists here lately, they're eerily fluent, but all seem to have the same minor speech impediment"
Haha, if that were to pass, that would still be a far better outcome than our current situation of completely blind machine translation (this is especially for various Asian languages that are very sensitive to phrasing) and mispronunciation by non-native speakers.
Kind of, Accents are typically derived from the intersection of natural languages, specifically which ones you learned the phonetics of first. (With the exception of the Mid-Atlantic accent...)
This would be something quite novel as the speech irregularities would not have their origin in people
I don't know what you would call it but it needs at least some adjective before accent to differentiate it IMO
I don't have the expertise to judge the quality of Mandarin pronunciation myself, being a beginner. But it sounds OK in English and it's made by native Mandarin speakers in China so I expect that it sounds better in Mandarin than English.
the azure neural tts voices in chinese are the best i’ve heard, specifically the “xiaochen” voice. i use it in anki daily to generate sentences for my mandarin decks with an api key/plugin. it’s not something you run locally of course, but they have a decent enough free tier.
i’m hoping a voice as realistic as this becomes a local app soon, but i’ve not found anything that’s nearly as natural sounding yet. (also, honorable mention to chatgpt’s “sky.” she pronounces mandarin with a funnily american accent, but it sounds natural and not as robotic as the open-source alternatives i’ve tried)
To me the key functionally for any language learning app is giving you feedback on your pronunciation and general understanding. I’ve been using Duolingo to learn Mandarin and when I try to speak to anyone it’s difficult for them to understand me, because my pronunciation is all wrong. The app is just feeding info to me one way, and I can try my best to recreate what I’m hearing, but there’s no way to know if I’m messing it up. They do have a speaking feature but it doesn’t work very well, certainly not to the same level as speaking with a real person who is fluent in the language and having them correct you.
As a quick solution, you should try recording yourself speaking and then listen to it to check your pronunciation against some reference. So for example, find a YouTube video in the language you're learning that also has good subtitles (use https://filmot.com/ ) and listen to how they say the phrase and then record yourself saying the same phrase and play it back and compare.
I practiced for a long time using the below pronunciation trainer and I get a ton of compliments from native speakers on how accurate my pronunciation is.
It's the same struggle language learners have faced for a long time regardless of app or not. I did careful studies of the French grammar, read French books, listened to audio tapes in French, and I still got prononciation wrong often.
I resisted using Duolingo because I knew their speaking feature sucks. But the only reason I need an app rather than books or audio tapes is that I need something to correct my prononciation.
I would love a game that helped you learn a language (not necessarily VR though as I don't have that equipment). The game drops you into a world (a country of the language the game is meant to teach you) where no one speaks your language and you have to figure out what people are saying in order to fulfill quests. You get some hints, like maybe you have a simple translation guide in your inventory or sometimes you meet people who can speak a few words of your language. That would motivate me to learn faster than self-taught tutorials.
I'd love to learn French and the game would take place in locations all around modern France.
It would have to a good story. Maybe something in the style of Professor Layton series could be interesting, or something more open world.
If Professor Layton itself has a French translation then you're more than half of the way there! Existing games are already quite good for language learning. But indeed they're missing the "realistic" element that you're after.
Absolutely, what I've noticed is that the current apps are great for beginners but after a certain point the only way to improve your ability to speak a new language is to well... speak it. I built Proseable to help people move beyond the generic how to order a coffee or ask to go to the bathroom, and have more meaningful conversations in the real world. Check it out!
(Duolingo problem(, AIUI): Duolingo is designed around such premise that, by exposing your subconsciousness to such small set of words and phrases in target languages, your brain should be able to trivially construct output shims from Universal Grammar, which must exist, to desired languages; but that doesn't work in practice and you end up with small set of words and phrases your subconsciousness had recorded)
Duo has a different problem for me. The lack of focus means some languages don't get features. Chinese still doesn't have Stories (there's an unofficial version of it, but we've been waiting years).
the Duolingo's problem it is not because they have a bunch of languages, it is because achieving fluency in a target language it is about been able to produce/generate phrases, and they just move you to consume and sort words and phrases. in the case of any AI Language tutor, the student must produce phrases in order to practice, and that makes them advance in the path to achieving fluency
https://chat.quazel.com teaches languages through speaking and having specific goals on where to steer the conversation. It's actually pretty nice. Has many language combinations to learn, ex. French speaker learning German. They seem to use LLMs and text to speech for now but I'm sure they'll get there with this tech progressing so fast. Duolinguo is from yesteryear in my opinion, learning by speaking is way better.
Yes! Better yet, you're a spy, or a hostage negotiator, or the leader of any kind of enterprise (army, business, aid organization) ...
Programming games like that will resemble directing improv theater. You can't program every response; you'll have to instead fit each character with beliefs and motivations.
For Language Acquisition, Input Is All You Need. (Mostly)
What would be really cool is something that can autodub videos or audio into your target language. The hardest problem learning languages that aren't English is often finding content to consume in them.
Disclaimer : I am Krashenist so this take is biased
Sure, if you're not into personal growth. Not everyone wants to become the useless bit of lard sitting in a chair while a computer does everything for them. Yet. Some of us still like to do the actual things, but just need some assistance along the way. We still have a bit of time before we're all the humanoids from Wall-E
> We still have a bit of time before we're all the humanoids from Wall-E
Obligatory reminder that the movie itself explains that people are what they are not because of their lifestyle, but because of the time spent in low-gravity environment.
That isn't an extreme example at all, people used to mill grain and make clothing by hand, now we don't. We somehow are not sitting around getting fat even though technology takes care of those tasks.
The parents suggestion is that if we don't have to learn languages that will lead to us all laying down drinking big gulps while robot slaves take care of us. Their take is the extreme example. People have literally made this same suggestion about every technological advance and it never comes true.
Not necessarily. It depends on the use case. For taking a vacation, having an AI that can instantly translate to your native language would be amazing. That’d solve a lot of real world problems, no doubt.
However, translation has a great deal of subjectivity embedded in it, particularly when there aren’t 1:1 translations. Case-in-point: there are many English translations of the Christian bible, all similar enough, but there are enormous variations in some cases. And there are at least as many branches of Christianity as there are English translations of the Bible. Some of them strictly recommend the same translation, and they still disagree on the meaning of various passages.
Besides the problems inherent to translation, learning another language gives you another paradigm of thinking. The words we use, the way we construct sentences, etc., all impact our view of the world. Here’s a paper that discusses the impact of the over-reliance on English in cognitive sciences, and how this has downstream effects:
https://www.sciencedirect.com/science/article/pii/S136466132...
Learning languages as an adult also has protective benefits. It reduces the probability of Alzheimer’s (maybe dementia, overall?).
It depends on what your goal is; for some tasks it's possible that getting the AI to do it is best, but, e.g. the existence of auto-pilot doesn't mean that hobbyist pilots wouldn't benefit from/enjoy exercising the same skills manually.
Maybe prior to fluency, for something like an odd business or tourist trip.
But there's a point in language learning where you can come to express yourself directly in a new language without intermediary "thinking" in your first tongue. The communicative and expressive potential of that mode is much higher than trying to squeeze one's intent through any kind of translation, machine or internal.
I'm learning Hindi and there are somethings that are easy (phonetic alphabet, nothing like 7 different sounds for 'ough') but the sentence structure is very different and can be hard to get right. Pronunciation isn't too bad for the most part but there a few tricky things, for example four different 't' sounds and four different 'd' sounds. The hardest part is that there really aren't that many resources. Even though Hindi is the third most spoken language in the world, you will find far more resources for many of the less spoken European languages.
Everyone gets a personal tutor for hours a day.
I would absolutely love a VR game where I just need to work in China or Mexico all day and pick up the language that way.