A couple of months ago, Ethan Shen of Gabble On called for participants in a machine translation research project. More than 1,000 reviewers answered his call. Ethan recently published his findings in summary and in full detail [PDF link].
The most interesting finding is related to brand bias. In other words, does the name of the machine translation tool affect the perceived quality? The answer is an unequivocal yes.
After survey participants noted this issue, Ethan adjusted the methodology:
"Midway through our data collection period, we implemented changes to our survey-taking platform to address brand and primacy biases in our experiment design pointed out by users."The results of this change were significant:
"Across the general populace it can be seen that users selected Google over Microsoft Translator 21% more often when they knew the brands compared to when the brands were hidden."And:
"Google relative brand bias effect over Yahoo Babelfish is even more stark..."While these findings are fascinating, unfortunately, they kind of put the rest of the results in question. Ethan's report notes that "When you take this bias into account ... many more languages pairing would be hotly contested or favoring Bing Translator or Babelfish". That's too bad because it negates any real findings from the survey but hopefully phase 2 will address this.
This "hiccup" begs a different question though: Do translation clients and users perceive quality differently, depending on who provided the translation?
There seems to be ample anecdotal evidence for this:
- When you buy a new DVD player that was made in Japan or China, don't you automatically expect the translation to be of substandard quality?
- Which translation provider hasn't heard the phrase "this reads like it was translated by a machine", implying that MT always produces substandard quality.
- And who remembers IBM ingeniously playing on IT staff's fears with the axiom "nobody ever got fired for buying IBM"?
Will clients prefer company A's translation over company B's translation simply because it was provided by company A? And does any translation company have the type of "brand muscle" to influence clients' perceptions?
Did you like this post? Subscribe to Medical Translation Insight via email or RSS!