How far do we agree on the quality of translation?
DOI:
https://doi.org/10.33919/esnbu.15.1.2Keywords:
TQA, translation mistakes, inter-rater reliability, error-based evaluation, error-annotated corpus, RusLTCAbstract
The article aims to describe the inter-rater reliability of translation quality assessment (TQA) in translator training, calculated as a measure of raters’ agreement either on the number of points awarded to each translation under a holistic rating scale or the types and number of translation mistakes marked by raters in the same translations. We analyze three different samples of student translations assessed by several different panels of raters who used different methods of assessment and draw conclusions about statistical reliability of real-life TQA results in general and objective trends in this essentially subjective activity in particular. We also try to define the more objective data as regards error-analysis based TQA and suggest an approach to rank error-marked translations which can be used for subsequent relative grading in translator training.
References
Artstein, R. & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4), 555–596. https://doi.org/10.1162/coli.07-034-R2
Freelon, D. G. (2010). ReCal: Intercoder Reliability Calculation as a Web Service. International Journal of Internet Science, 5(1), 20–33.
Kelly, D. (2005). A Handbook for Translator Trainers. A Guide to Reflective Practice. Manchester: St. Jerome Publishing.
Knyazheva, E & Pirko, E. (2013). Otsenka kachestva perevoda v rusle metodologii sistemnogo analiza [TQA and Systems Analysis Methodology]. Journal of Voronezh State University. Linguistics and Intercultural Communication Series, 1, 145-151.
Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology. Sage Publications.
Krippendorff, K. (2011). Computing Krippendorff's Alpha-Reliability. Retrieved from http://repository.upenn.edu/asc_papers/43/
Neubert, A. (2000). Competence in Language, in Languages, and in Translation. In Schäffner, C. & Adab, B. (Eds.). Developing Translation Competence. Amsterdam/Philadelphia: John Benjamins Publishing Company (pp. 3–17). https://doi.org/10.1075/btl.38
Strijbos, J.-W. & Stahl, G. (2007). Methodological Issues in Developing a Multidimensional Coding Procedure for Small-group Chat Communication. Learning and Instruction, 17(4), 394-404. https://doi.org/10.1016/j.learninstruc.2007.03.005
Waddington, Ch. (2001) Should Translations be Assessed Holistically or through error
analysis?. Hermes, 26, 15-37. Retrieved from http://download2.hermes.asb.dk/archive/download/H26_03.pdf
Williams, M. (2009). Translation Quality Assessment. Mutatis Mutandis, 2(1), 3–23.
Zwilling, M. (2009). O kriteriiakh otsenki perevoda [On Translation Quality Assessment Criteria]. In Zwilling, M. (Ed.), O perevode i perevodtchikakh [On Translation and Translators] (pp. 56–63). Мoskva: Vostotchnaia kniga.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2015 Maria Kunilovskaya
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Access Policy and Content Licensing
All published articles on the ESNBU site are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, even for commercial purposes. The terms on which the article is published allow the posting of the published article (Version of Record) in any repository by the author(s) or with their consent.
Note that prior to, and including, Volume 10, Issue 2, 2024, articles were licensed under the Non-commercial (CC BY-NC 4.0) license. The transition to CC BY 4.0 is effective as of Volume 11, Issue 1, 2025.
In other words, under the CC BY 4.0 license users are free to
Share — copy and redistribute the material in any medium or format for any purpose, even commercially.
Adapt — remix, transform, and build upon the material for any purpose, even commercially.
Under the following terms:
Attribution (by) - You must give appropriate credit (Title, Author, Source, License), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
Notice: No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
If the law requires that the article be published in the public domain, authors will notify ESNBU at the time of submission, and in such cases the article shall be released under the Creative Commons Public Domain Dedication waiver CC0 1.0 Universal.
Copyright
Copyright for articles published in ESNBU are retained by the authors, with first publication rights granted to the journal. Authors retain full publishing rights and are encouraged to upload their work to institutional repositories, social academic networking sites, etc. ESNBU is not responsible for subsequent uses of the work. It is the author's responsibility to bring an infringement action if so desired by the author.
Exceptions to copyright policy
Occasionally ESNBU may co-publish articles jointly with other publishers, and different licensing conditions may then apply.
