MULTIDIMENSIONAL EVALUATION OF A DOMAIN-SPECIFIC MACHINE TRANSLATION ENGINE
DOI:
https://doi.org/10.33919/JCAL.24.2.3Keywords:
WIPO, patent translation, machine translation, automatic evaluation, quality metricsAbstract
The study presented in this article aims to evaluate the performance of WIPO Translate, a domain-specific machine translation (MT) tool, against three commercially available MT systems: Google Translate, DeepL and OpenAI’s ChatGPT 3.5. The evaluation focuses on assessing the translation outputs of patent excerpts across five dimensions, based on the Multidimensional Quality Metrics framework (MQM), namely Accuracy, Audience Appropriateness, Linguistic Conventions, Style and Terminology, from English to Brazilian Portuguese, deriving an overall quality score from the error annotations. The findings suggest that WIPO Translate, despite being specifically trained on patent corpora and performing reasonably well, does not outperform the commercially available MT systems in terms of translation quality. Generic systems demonstrate higher OQS averages in most cases, with DeepL generally achieving the highest scores among them. Automatic quality metrics were also implemented with BLEU, BERTScore, COMET and COMET-QE. WIPO Translate also ranked lowest in the automatic evaluation metrics. These results indicate that the broad training datasets and advanced algorithms used by general-purpose MT systems provide a competitive edge, even in specialized domains such as patent translation. Future improvements to WIPO Translate could focus on refining its algorithms
and expanding its training data to bridge this performance gap. Google and ChatGPT also presented superior performances, the latter being an interesting resource for users to customise the translation through detailed prompts. WIPO Translate still performs well according to the automatic metrics, and can also be very useful as a corpora tool, assisting users in gauging patent style and drafting better patents on their own.
References
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., Farhan, L. (2021). ‘Review of Deep Learning: Concepts, CNN Architectures, Challenges, Applications, Future Directions’. Journal of Big Data, 8(1), 1–74. Available at: https://doi.org/10.1186/s40537-021-00444-8.
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., Macherey, W., Chen, Z., Wu, Y. (2019). ‘Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1907.05019.
Artstein, R., Poesio, M. (2008). ‘Inter-Coder Agreement for Computational Linguistics’. Computational Linguistics, 34(4), 555–596. Available at: https://doi.org/10.1162/coli.07-034-R2.
Bonyadi, A. (2020). ‘Google Translate Performance in Translating English Passive Voice into Indonesian’. PIONEER: Journal of Language and Literature. Available at: https://unars.ac.id/ojs/index.php/pioneer/article/view/1292.
Briva-Iglesias, V., Cavalheiro Camargo, J. L., Dogru, G. (2024). ‘Large Language Models “Ad Referendum”: How Good Are They at Machine Translation in the Legal Domain?’. Arxiv.org. Cornell University. Available at: https://doi.org/10.48550/arXiv.2402.07681.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C. Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. (2020). ‘Language Models are Few-Shot Learners’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/2005.14165.
Callison-Burch, C., Osborne, M., Koehn, P. (2006). ‘Re-evaluating the Role of BLEU in Machine Translation Research’. In: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 249–256. Available at: https://aclanthology.org/E06-1032.pdf.
Castilho, S., Doherty, S., Gaspari, F., Moorkens, J. (2018). ‘Approaches to Human and Machine Translation Quality Assessment’. In: Moorkens, J., Castilho, S., Gaspari, F., Doherty, S. (eds.) Translation Quality Assessment. Machine Translation: Technologies and Applications, vol. 1, 9–38. Springer, Cham. Available at: https://doi.org/10.1007/978-3-319-91241-7_2.
Chimoto, E., Bassett, B. (2022). ‘COMET-QE and Active Learning for Low-Resource Machine Translation’. In: Findings of the Association for Computational Linguistics: EMNLP 2022, 4735–4740, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Available at: https://aclanthology.org/2022.findings-
emnlp.348/.
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2019). ‘BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1810.04805.
German Research Center for Artificial Intelligence [DFKI] and QTLaunchPad (2015). ‘Multidimensional Quality Metrics (MQM) Definition’. Web.archive.org. Available at: https://web.archive.org/web/20160425060914/http://www.qt21.eu/mqmdefinition/definition-2015-12-30.html.
Jana, S., Biswas, R., Pal, K., Biswas, S., Roy, K. (2024). ‘The Evolution and Impact of Large Language Model Systems: A Comprehensive Analysis’. Alochana, 10-AJ2169. Available at: https://alochana.org/wp-content/uploads/10-AJ2169.pdf.
Jooste, W., Haque, R., Way, A. (2021). ‘Philipp Koehn: Neural Machine Translation – Cambridge University Press’. Cambridge University Press. Available at: https://www.researchgate.net/publication/353142102_Philipp_Koehn_Neural_Machine_Translation.
Kocmi, T., Zouhar, V., Federmann, C., Post, M. (2024). ‘Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/2401.06760.
Koehn, P. (2010). Statistical Machine Translation. Cambridge: Cambridge University Press.
Koehn, P. (2017). ‘Neural Machine Translation’. Statistical Machine Translation. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1709.07809.
Koehn, P., Knowles, R. (2017). ‘Six Challenges for Neural Machine Translation’. In: Proceedings of the First Workshop on Neural Machine Translation, 28–39. Vancouver.
Association for Computational Linguistics. Available at: https://aclanthology.org/W17-3204/.
Krippendorff, K. (2013). ‘Computing Krippendorff’s Alpha-Reliability’. University of Pennsylvania Annenberg School for Communication. Available at: https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf.
Lommel, A., Uszkoreit, H., Burchardt, A. (2014). ‘Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics’. Revista Tradumàtica: Tecnologies de la Traducció, (12), 455–463. Available at: https://ddd.uab.cat/pub/tradumatica/tradumatica_a2014n12/tradumatica_
a2014n 12p455.pdf.
Maučec, M., Donaj, G. (2019). ‘Machine Translation and the Evaluation of its Quality’. Recent Trends in Computational Intelligence. IntechOpen. Available at: https://doi.org/10.5772/intechopen.89063.
Morita, T. (2024). ‘Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary’. Arxiv.org. Cornell University Available at: https://arxiv.org/abs/2402.00236.
Moslem, Y., Way, A., Haque, R., Kelleher, J. D. (2022). ‘Domain-Specific Text Generation for Machine Translation’. In: Proceedings from the 15th Conference of the Association for Machine Translation in the Americas. Available at: https://aclanthology.org/2022.amta-research.
Papineni, K., Roukos, S., Ward, T., Zhu, W. (2002). ‘Bleu: A Method for Automatic Evaluation of Machine Translation’. In: Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, 311–318. Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. Available at: https://aclanthology.
org/P02-1040.pdf.
Pouliquen, B., Mazenc, C., Iorio, A. (2011). ‘Tapta: A User-Driven Translation System for Patent Documents Based on Domain-Aware Statistical Machine Translation’. In: Proceedings of the 15th Annual Conference of the European Association for Machine Translation, Leuven, Belgium. European Association for Machine Translation.
Available at: https://aclanthology.org/2011.eamt-1.2/.
Rei, R., Stewart, C., Farinha, A., Lavie, A. (2020). ‘COMET: A Neural Framework for MT Evaluation’. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2685–2702. Association for Computational Linguistics. Available at: https://aclanthology.org/2020.emnlp-main.213/.
Reiter, E. (2018). A structured review of the validity of BLEU. Computational Linguistics, 44(3), 393-401. Available at: https://doi.org/10.1162/coli_a_00322.
Rivera-Trigueros, I. (2022). ‘Machine Translation Systems and Quality Assessment: A Systematic Review’. Computers and the Humanities, 56, 593–619. Available at: https://doi.org/10.1007/s10579-021-09537-5.
Rossi, L., Wiggins, D. (2013). ‘Applicability and Application of Machine Translation Quality Metrics in the Patent Field’. World Patent Information, 35(1), 20–26. Available at: http://dx.doi.org/10.1016/j.wpi.2012.12.001.
Savoy, J., Dolamic, L. (2009). ‘How Effective is Google’s Translation Service in Search?’ Communications of the ACM, 52(10), 139–143. Available at: https://doi.org/10.1145/1562764.1562799.
Sutrisno, A. (2020). ‘The Accuracy and Shortcomings of Google Translate. Translating English Sentences to Indonesian’. Education Quarterly Reviews, Vol. 3(4). Available at: https://ssrn.com/abstract=3747888.
Tinsley, J. (2017). Machine translation and the challenge of patents. In: Multilingual Information Management. Springer, 295–316. Available at: https://doi.org/10.1007/978-3-662-53817-3_16.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H. Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Macduff, H., Dean, J. (2016). ‘Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1609.08144.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., Artzi, Y. (2020). ‘BERTScore: Evaluating Text Generation with BERT’. In: Proceedings of the 8th International Conference on Learning Representations (ICLR 2020). Available at: https://openreview.net/pdf?id=SkeHuCVFDr.
Zhao, Y., Zhang, J., Zong, C. (2023). ‘Transformer: A General Framework from Machine Translation to Others’. International Journal of Automation and Computing, 20(4), 514–538. Available at: https://doi.org/10.1007/s11633-022-1393-5.
