According to the results of the DiBiMT benchmark, it is ahead of Google Translate and even large language models
According to a study by the international independent benchmark DiBiMT, Yandex Translator translated text from English into Russian better than other language models and online translators. Benchmark DiBiMT — it is a benchmark for measuring error rates in machine translation. Along with Yandex Translator, Google Translate and DeepL were tested, as well as open large language models: mBART-50, M2M100 and the OPUS project.
Researchers have compiled a database of several thousand English sentences with homonyms — words that are spelled the same but have different meanings. Models and services translated these sentences, and benchmark algorithms determined the percentage of correctly translated words. In addition, the correct options were selectively checked by professional translators, and then a public rating was formed based on these indicators.
Yandex Translator also took third place in terms of quality of translation from English into Chinese, German, Italian and Spanish. The test showed that the service is better than others at determining the meaning of specific English words in sentences and makes fewer semantic errors.