Perception of quality depends on translation provider
Written by ForeignExchange Translations on Tuesday, May 25, 2010
A couple of months ago, Ethan Shen of Gabble On called for participants in a machine translation research project. More than 1,000 reviewers answered his call. Ethan recently published his findings in summary and in full detail [PDF link].
The most interesting finding is related to brand bias. In other words, does the name of the machine translation tool affect the perceived quality? The answer is an unequivocal yes.
After survey participants noted this issue, Ethan adjusted the methodology:"Midway through our data collection period, we implemented changes to our survey-taking platform to address brand and primacy biases in our experiment design pointed out by users."
The results of this change were significant:"Across the general populace it can be seen that users selected Google over Microsoft Translator 21% more often when they knew the brands compared to when the brands were hidden."
And:"Google relative brand bias effect over Yahoo Babelfish is even more stark..."
While these findings are fascinating, unfortunately, they kind of put the rest of the results in question. Ethan's report notes that "When you take this bias into account ... many more languages pairing would be hotly contested or favoring Bing Translator or Babelfish". That's too bad because it negates any real findings from the survey but hopefully phase 2 will address this.
This "hiccup" begs a different question though: Do translation clients and users perceive quality differently, depending on who provided the translation?
There seems to be ample anecdotal evidence for this:
- When you buy a new DVD player that was made in Japan or China, don't you automatically expect the translation to be of substandard quality?
- Which translation provider hasn't heard the phrase "this reads like it was translated by a machine", implying that MT always produces substandard quality.
- And who remembers IBM ingeniously playing on IT staff's fears with the axiom "nobody ever got fired for buying IBM"?
Will clients prefer company A's translation over company B's translation simply because it was provided by company A? And does any translation company have the type of "brand muscle" to influence clients' perceptions?
Did you like this post? Subscribe to Medical Translation Insight via email or RSS!
Please forgive my poor english, I am much better at translating.
@Anonymous: The blog is an open forum. It is our hope to spur conversations. If translators want their voices heard, or disagree with our posts, please leave us a comment! We'd love to hear it. Additionally, if there is a specific topic you'd like tackled, or would like to write a guest blog for us, please contact us! We're always open to other points of view.
This article, however, really is more about perceived quality when someone knows the 'brand.' While this article was written based on MT testing, I wonder if the same would happen for human translations? If an agency/client, etc knows WHO did the translations, would there be a preference? Human nature dictates: probably. What are your thoughts? If you're working with two separate editors, say, and you know editor A, have worked with them for years, and editor B is someone you don't know, do you think you'd be biased towards editor A's comments?
I wrote a criticism of the Shen experiment that might be useful to look at as it was quite predicatable.
http://kv-emptypages.blogspot.com/2010/03/ongoing-quest-for-best-mt-translation.html
But unfortunately it is a matter of perception and it would be useful to everybody if there was a clear way to rate/rank every translation in a mostly objective way
Reading your post made me remember a little tool that was built for some MT fun.
Check out www.translationparty.com It takes a sentence you determine, translates to Japanese, then back translates to English. It continues in this fashion until it reaches what the site calls 'equilibrium.' This often produces hilarious results!
This is also a fun way to show someone how backtranslations are not an exact science.