Quality assessment of machine translation output: cognitive evaluation approach in an eye tracking experiment / Avaliação da qualidade da produção de tradução automática: abordagem de avaliação cognitiva em um experimento com rastreamento ocular

Ramunė Kasperavičienė, Jurgita Motiejūnienė, Irena Patašienė

Resumo


ABSTRACT:Despite fast development of machine translation, the output quality is less than acceptable in certain language pairs. The aim of this paper is to determine the types of errors in machine translation output that cause comprehension problems to potential readers. The study is based on a reading task experiment using eye tracking and a retrospective survey as a complementary method to add more value to the research as eye tracking as a method is considered to be problematic and challenging (O’BRIEN, 2009; ALVES et al., 2009). The cognitive evaluation approach is used in an eye tracking experiment to determine the complexity of the errors in the English–Lithuanian language pair from easiest to hardest as seen by the readers of a machine-translated text. The tested parameters – gaze time and fixation count – demonstrate that a different amount of cognitive effort is required to process different types of errors in machine-translated texts. The current work aims at contributing to other research in the Translation Studies field by providing the analysis of error assessment of machine translation output.

KEYWORDS: machine translation; cognitive evaluation approach; translation error(s); eye tracking; acceptability.

 

RESUMO:Apesar do rápido desenvolvimento da tradução automática, a qualidade do texto produzido é bastante pobre em algumas combinações linguísticas. O objetivo deste artigo é determinar os tipos de erros na produção de tradução automática que acarretam dificuldades de compreensão para os potenciais leitores. O estudo é baseado em um experimento que utiliza rastreamento ocular e um questionário retrospetivo como método complementar de forma a acrescentar mais valor à pesquisa, visto que o rastreamento ocular enquanto método é muitas vezes considerado problemático e desafiador (O’BRIEN, 2009; ALVES et al., 2009). A abordagem de avaliação cognitiva é utilizada em um experimento com rastreamento ocularpara determinar a complexidade dos erros na combinação linguística inglês-lituano dos mais fáceis aos mais difíceis, conforme visto pelos leitores do texto traduzido automaticamente. Os parâmetros testados (duração do olhar e número de fixações) demonstram que é necessário um esforço cognitivo diferente para processar diferentes tipos de erros em textos traduzidos de forma automática. Este trabalho almeja contribuir para outras pesquisas neste campo, pois fornece análise de avaliação de erros da produção de tradução automática.

PALAVRAS-CHAVE: tradução automática; abordagem de avaliação cognitiva; erro(s) de tradução; rastreamento ocular; aceitabilidade.


Palavras-chave


machine translation; cognitive evaluation approach; translation error(s); eye tracking; acceptability.

Texto completo:

PDF (English)

Referências


ALVES, F.; SARTO SZPAK, K.; GONÇALVES, J. L.; SEKINO, K.; AQUINO, M.; ARAUJO E CASTRO, R.; KOGLIN, A.; DE LIMA FONSECA, N. B.; MESA-LAO, B. Investigating cognitive effort in post-editing: A relevance-theoretical approach. In: HANSEN-SCHIRRA, S. and GRUCZA, S. (eds.). Eyetracking and applied linguistics, Berlin: language science press, 2016. p. 109-142.

ALVES, F.; PAGANO, A.; DA SILVA, I. A new window on translators’ cognitive activity: methodological issues in the combined use of eye tracking, key logging and retrospective protocols. In: MEES, I. M.; ALVES, F.; GÖPFERICH, S. (eds.). Methodology, Technology and Innovation in Translation Process Research. Copenhagen: Samfundslitteratur, 2009. p. 267-292.

CARL, M.; DRAGSTED, B.; ELMING, J.; HARDT, D.; LYKKE JAKOBSEN, A. The process of post-editing: a pilot study. In: Proceedings of the 8th International Natural Language Processing and Cognitive Science Workshop, p. 131-142, 2011.

CARL, M.; GUTERMUTH, S.; HANSEN-SCHIRRA, S. Post-editing machine translation: Efficiency, strategies, and revision processes in professional translation settings. Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. In: FERREIRA, A.; SCHWIETER, J. W. (eds.). Psycholinguistic and cognitive inquiries into translation and interpreting. Amsterdam: John Benjamins, 2015. p. 145-174.

CASTILHO, S. Measuring Acceptability of Machine Translated Enterprise Content (PhD thesis). Dublin: Dublin City University, 2016.

CASTILHO, S.; DOHERTY, S.; GASPARI, F.; MOORKENS, J. Approaches to Human and Machine Translation Quality Assessment. In: MOORKENS J.; CASTILHO S.; GASPARI F.; DOHERTY S. Translation Quality Assessment. Machine Translation: Technologies and Applications, 2018. p. 9-38.

CHOMSKY, N. Aspects of the theory of syntax. Cambridge: MIT Press, 1969.

DAEMS, J.; VANDEPITTE, S.; HARTSUIKER, R. J.; MACKEN, L. Identifying the machine translation Error types with the greatest impact on post-editing effort. In: Frontiers in Psychology, v. 8, 2017. p. 1-15.

DE BEAUGRANDE, R.; DRESSLER, W. Introduction to text linguistics. London, New York, 1981.

FLANAGAN, A. M. Error classification for MT evaluation. In: Proceedings of the first Conference of the Association for Machine Translation in the Americas, 1994. p. 65-72.

GONÇALVES, J. L. Investigating Saccades as an index of cognitive effort in post-editing and translation. In: Proceedings of EST Congress. 2016, Aarhus, Denmark, 2016.

GRAHAM, Y. Improving Evaluation of Machine Translation Quality Estimation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015. p. 1804-1813.

HVELPLUND, K. T. Four fundamental types of reading during translation. In: JAKOBSEN, A. L.; MESA-LAO, B. (eds.), Translation in Transition: Between Cognition, Computing and Technology. Amsterdam / Philadelphia: John Benjamins Publishing Company, 2017. p. 55-77.

HUTCHINS, J.; SOMERS, H. An introduction to machine translation. London: Academic Press, 1992.

JAKOBSEN, A. L.; JENSEN K. T. H. Eye movement behaviour across four different types of reading task. In: GÖPFERICH, S.; JAKOBSEN, A. L.; MEES, I. M. (eds.). Looking at eyes: eye-tracking studies of reading and translation processing, 2008. p. 78-98.

KORNACKI, M. The application of eye-tracking in translator training. inTRAlinea: New Insights into Translator Training, 2019.

MOORKENS, J. Eye tracking as a measure of cognitive effort for post-editing of machine translation. In: WALKER, C.; FEDERICI, F. M. (eds.). Eye tracking and multidisciplinary studies on translation, 2018. p. 55-69.

NITZKE, J. Monolingual post-editing: An exploratory study on research behavior and target text quality. In: HANSEN-SCHIRRA, S.; GRUCZA, S. (eds.). Eyetracking and applied linguistics, Berlin: Language Science Press, 2016. p. 83-109.

O’BRIEN, SH. Towards predicting post-editing productivity. Machine Translation, v. 25, n. 3, p. 197-215, 2011.

O’BRIEN, SH. Eye tracking in translation-process research: methodological challenges and solutions. In: MEES, I. M.; ALVES, F.; GÖPFERICH, S. (eds.). Methodology, Technology and Innovation in Translation Process Research. Copenhagen: Samfundslitteratur, 2009. p. 251-266.

O’BRIEN, SH. Machine-translatability and post-editing effort: An empirical study using translog and choice network analysis. Dublin: Dublin City University, 2006.

PETKEVIČIŪTĖ, I.; TAMULYNAS, B. Kompiuterinis vertimas į lietuvių kalbą: alternatyvos ir jų lingvistinis vertinimas. Studies about Languages, v. 18, Kaunas: Technologija, p. 38-45, 2011.

POPOVIC, M. Error classification and analysis for machine translation quality assessment. In: MOORKENS, J.; CASTILHO, S.; GASPARI, F.; DOHERTY, S. (eds.). Translation quality assessment: from principles to practice, Springer international publishing AG, 2018. p. 129-158.

ROTURIER, J. An investigation into the impact of controlled English rules on the comprehensibility, usefulness and acceptability of machine-translated technical documentation for French and German users (PhD thesis). Dublin: Dublin City University, 2006.

SPECIA, L. Exploiting objective annotations for measuring translation post-editing effort. In: FORCADA, M. L.; DEPRAETERE H.; VANDEGHINSTE V. (eds.). Proceedings of the 15th Conference of the European Association for Machine Translation, 2011. p. 28-37.

STYMNE, S.; DANIELSSON, H.; BREMIN, S.; HU, H.; KARLSSON, J.; PRYTZ LILLKULL, A.; WESTER, M. Eye Tracking as a Tool for Machine Translation Error Analysis. In: CALZOLARI, N.; CHOUKRI, K.; DECLERCK, T.; UĞUR DOĞAN, M.; MAEGAARD, B.; MARIANI, J.; MORENO, A.; ODIJK, J.; PIPERIDIS, S. (eds.). Proceedings of the 8th international conference on language resources and evaluation, 2012. p. 1121-1126.

TEMNIKOVA, I. Cognitive evaluation approach for a controlled language post-editing experiment. In: Proceedings of the 7th international conference on language resources and evaluation, 2010. p. 3485-3490.

TEMNIKOVA, I.; ZAGHOUANI W.; VOGEL, S.; HABASH, N. Applying the Cognitive Machine Translation Evaluation Approach to Arabic. In: Proceedings of the International Conference on Language Resources and Evaluation, 2016. p. 3644-3651.

TURENKO, V.; BALTULIONIS, S.; VASILJEVAS, M.; DAMAŠEVIČIUS, R. Analysing program source code reading skills with eye tracking technology, Information Technology. In: Proceedings of IVUS’2019, Vytautas Magnus University, 2019. p. 50-54.

VAN SLYPE, G. Critical study of methods for evaluating the quality of machine translation. Brussels: Buerau Marcel van Dijk, 1979.

VILAR, D.; XU, J.; D’HARO, L. F.; NEY, H. Error analysis of statistical machine translation output. In: Proceedings of the 5th International Conference on Language Resources and Evaluation, 2006. p. 697-702.




DOI: http://dx.doi.org/10.17851/1983-3652.13.2.%25p

Apontamentos

  • Não há apontamentos.




Texto Livre: Linguagem e Tecnologia
ISSN 1983-3652 (eletrônica)

Faculdade de Letras da Universidade Federal de Minas Gerais

Belo Horizonte - Minas Gerais (Brasil)

Licença Creative Commons

Esta obra está licenciada com uma Licença Creative Commons Atribuição 4.0 Internacional.
SCImago Journal & Country Rank