On the availability of the Rust survey 2023 in languages other than english

First of all, let me be clear on one thing: I really appreciate that any material is made available in languages other than English. It really helps on the goal of further spreading the knowledge worldwide.

That being said, I would like to know what's the process for making such material available within the respective working groups (i.e. the Rust Survey Working Group in this case).

I love my mother language (spanish), but totally hate when people don't put the effort needed when creating written material, specially when the task doesn't involve content creation (i.e. the content was translated). And this takes me to my complain regarding the Rust survey 2023:

Given the option, I opt-in to consume a given content in spanish (just for the sake of supporting the idea of making the content available in more languages), but just in the very first line of text I found out a typing error:

2023 - Encuesta anual de la Comunidad del Lenguaje Rustaje Rust

Which, translated, becomes something along the lines of:

2023 - Annual survey of the Rust Langust Language Community

With mistakes as evident as this one (and on the very first line of text, on top of that), I think that the idea being communicated goes from:

We try to support communities other than english-speaking ones

To:

We don't want to support non engish-speaking communities. If we have to do so, we will make the minimal effort, as in using unverified machine translations.

The rest of the text on that first page feels very machine-translated. This discouraged me from continue answering the survey in spanish, and went back to answer it in english instead.

I didn't check other languages, but I was wondering if other members of the community have similar experiences with their mother languages and if so, what can be done to improve the translation of the surveys.

If the problem is the lack of funds, one idea would be to ask within the community several weeks in advance for volunteers to help out with the translations.

3 Likes

Looking at German, it’s definitely a lot better than pure machine translation, but not optimal… translations are almost never really good, especially if they aren’t created professionally (translating is also hard), which is why I share your approach of usually using the English original as it’s clearly higher quality. I could also spot at least one typo edit: my bad, it wasn’t even a typo, just a word I wasn't used to reading already in the German translation - typos are not necessarily an indication for machine translation though, perhaps rather the opposite.

(One particularly funny encounter[1], where the Google Translate translation would have been awful but official German survey is perfectly reasonable was "Custom fork" as an answer to " Which version(s) of Rust do you use for local development?". Google Translate would give me "Kundenspezifische Gabel" which would mean "customer specific (dining) fork", whereas the survey just says "eigener Fork" (== roughly "one's/your own fork") which is a perfectly reasonable/good & correct translation; especially as "fork" isn't translated in this meaning for software development, and any more specific translation of "custom" would become overly specific for this answer option.)

Possibly, the translations were created by starting with a machine translation, which has the effects that human reviewers might just leave everything that’s not obviously wrong, so that it still sounds more machine-translated in style. Also perhaps German got better human review than Spanish. (I can’t speak a word of Spanish, so I couldn’t compare the quality.)


  1. and one of many clear indicators that some proper humans seem to have participated in the translation ↩︎

2 Likes

That approach could definitely explain what I saw, although I didn't go any further than the first page in order to verify if the rest of the survey had a better quality in its translations.

Glad to hear that it wasn't as awful in german :smile:.

I found the Dutch version lacking, to be honest :slight_smile:

Literal translation of "crates" to Russian as "ящики" was also hilarious. You need a fair bit of mental gymnastics to understand meaning of the question with a translation like this, since in practice everyone uses transliterated version of the word ("крейт"). This is why I usually ignore translations in cases like this and use original English versions.

I think it should be mandatory for translations to be checked by at least one-two native speakers before publishing them widely.

7 Likes

I know that there was at least an effort made to reach out to fluent speakers to verify things, though I can't speak as to how successful the outreach was. I was contacted last week about verifying the machine translation of Spanish, but declined as I felt my knowledge was not sufficient (having ironically returned from Mexico the day prior).

2 Likes

Usually, survey stuff is talked about on zulipchat. I found two thread:

Hi, I'm posting here for better visibility. We are nearing the launch of the 2023 Rust Annual survey, and we'd like to ask for help with checking translations of the survey, which were generated automatically with machine translation. If you'd like to help, please check this message. Currently, we are missing someone who could check Spanish, Portuguese, Ukraine, Traditional Chinese or Korean translations.
src: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/Asking.20for.20help.20with.20translating.20the.20annual.20survey

I created a new topic to have a single message with a guide on how to help with translating the 2023 Rust Annual Survey. Sadly, the process is not very automated, but it's the best we can do this year. Note that the translations have been generated automatically, and we mostly want help with checking that they are OK-ish, and finding out potential fixes/improvements.

Here is how you can help with checking/translating the survey:

  1. Download a PDF with the English version of the survey from here
  2. Open the survey and choose the language that you want to help translate
  3. Go through the survey and cross-check that the translation makes sense, w.r.t. the English version from the PDF. You can write your notes e.g. into a hackMD document.
  4. After you're finished, please send your notes/changes to Gracie Gregory at graciegregoryrustfoundation.org.

It would be great if we could finish reviewing the translations by 15th December.

People that volunteered to translate the survey (I will be updating this list dynamically, if new people sign up):

French: Albert Larsan
Simplified Chinese: Charles Lew
Japanese: tanakakz
German: llogiq
Russian: Yoh Deadfall
Spanish: Iñaki Garay, Manuel Drehwald
Portuguese:
Ukraine:
Traditional Chinese:
Korean:
If you want to help, please write what language would you like to check/translate in this stream and I will add you to the list.

src: https://rust-lang.zulipchat.com/#narrow/stream/402479-t-community.2Frust-survey/topic/Translation.20guide

3 Likes

Thanks for the info., Vague. I'll try to register on Zulip too to not miss posts like this one. I understand that proper translations are hard and take a considerable amount of time, therefore I would like to help whenever possible.

3 Likes

I also found these info

If you have concrete proposals on how to improve the translation, you can add them here: Rust Survey 2023 Translation Corrections - Google Docs.

src: reddit

Some people started filling translation edits (using above google doc link)

src: zulipchat - survey feedback

For each specific question, there is a repo where an issue for it is created: Issues · rust-lang/surveys · GitHub

2 Likes

Hi (member of the Rust survey team here), thanks for the feedback! We are sorry for the bad quality of the Spanish (possibly also other) translation(s).

This year, we were scrambling to put the survey out at all, since there were not a lot of people left in the team from previous years, so we had to restart the effort. Regarding translations, I think that a few years back, we had a team of volunteer translators that translated everything from scratch, but we were not able to find so many volunteers this year, and since the survey was prepared quite late, I don't think that there was even time for that.

Last year, I think that a professional translation company was used, but I think that the quality also wasn't stellar, so we decided to abandon it.

This year (well, in the 2023 survey), we have used the buit-in machine translation of SurveyHero (the survey system that we use), which uses Google Translate. We knew that the translations are far from perfect, so we tried to enlist the help of at least a few volunteers to check and correct the AI generated translations. We hoped that this would catch the most glaring errors, as we knew that unreviewed translations are not a good idea, and that's also why we have decided to ditch languages that were auto-translated, but for which we weren't able to find reviewers.

The Spanish translation specifically was reviewed by two separate reviewers, and I'm not sure what happened, if they didn't catch these mistakes or if the corrections didn't appear in the final version by accident. I have to say that the translation process is quite complicated in SurveyHero, since it can only export the questions in PDF, but it doesn't support accented/Unicode characters, so we were not able to even easily export the translated versions, and reviewers thus had to manually go through the survey to find errors, which was really error-prone. And the fixes also had to be manually applied by a single person to the survey, which was quite challenging for N language versions.

It's clear that the translations were again lacking this year, and I'm sorry for that. Hopefully, we will be able to finalize the survey questions much sooner for the next survey and thus have more time to ask for help with translating from community members (maybe we could try to engage a more broader community, e.g. on Reddit).

(Sorry for answering so late, I didn't notice this forum thread, I mostly lurk on Zulip.)

5 Likes

Hey @moy2010 I usually don't read urlo, so just saw it now. Asking more people earlier in a more public space for feedback would indeed have been helpful. I have seen it by coincidence a few days before publication in a rust-lang zulip channel and offered to check the spanish one, since I grew up bilingual and they did not find anyone else. By then they had already used the auto-translate feature from the website, which is somewhere between entertaining and terrible. I added hundred lines of corrections, so it would have been definetly easier to just either do it from scratch, or at least start with a version translated from something smarter like chatGPT and not the survey-hero AI.

A second user also helped last minute, but his town got hit by a storm, leaving him without electricity and wifi, so he also couldn't fix too much. After around 200 corrections I simply ran out of energy and ended up focusing on the worst issues, like translating (Ventanas10) back to Windows10. As mentioned by Kobzol, the website unfortunately made the correction process for everyone involved quite exhausting.

I assume that by next year chatGPT should be able to solve that easily, I just really hope the survey-hero AI isn't used again. Also I think they will ask in a more visible place than this rust-lang zulip channel, so people will have more time to fix the remaining issues.

1 Like