EMOTIONALTURK, A NEW STANDARD IN EVALUATING AUTOMATIC TURKISH EMOTION DETECTION: A CASE STUDY IN A CROSSLINGUAL TRANSFER SETTING
DOI:
https://doi.org/10.33919/JCAL.24.2.2Keywords:
emotion detection from text in Turkish, cross-lingual transfer, lower-resource languages, multilingual transformers, classifiersAbstract
Current Turkish emotion detection from text (EDFT) research, and EDFT research in general, does not justify the rationale behind the emotion framework used in their EDFT datasets. Adapting emotion frameworks directly from psychology research, the most popular ones based on physiology, creates a gap between what is latently available in textual data and what the model output is, regardless of its performance. Therefore, the article introduces a dataset, EmotionalTurk, built with an emotion framework empirically grounded in textual data. While EmotionalTurk is currently limited to a test set in size, the article seizes the opportunity to explore cross-lingual transfer capabilities of multilingual transformers – an area that has not been previously investigated – using mBERT-based and XLM-RoBERTaBASE-based models, and establishes the SOTA score, which also serves as the baseline in this study in cross-lingual Turkish EDFT.
References
Cohen, J. (1960). ‘A Coefficient of Agreement for Nominal Scales’. Educational and Psychological Measurement, 20 (1), 37–46. Available at: https://doi.org/10.1177/001316446002000104.
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V. (2020). Unsupervised Cross-lingual Representation Learning at Scale [Preprint]. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1911.02116.
De Bruyne, L., Karimi, A., De Clercq, O., Prati, A., Hoste, V. (2022). ‘Aspect-Based Emotion Analysis and Multimodal Coreference: A Case Study of Customer Comments on Adidas Instagram Posts’. ACLWeb. Marseille, France: European Language Resources Association. Available at: https://aclanthology.org/2022.lrec-1.61/.
De Bruyne, L., De Clercq, O., Hoste, V. (2019). ‘Towards an Empirically Grounded Framework for Emotion Analysis’. HUSO. Available at: https://core.ac.uk/download/pdf/225020418.pdf.
De Bruyne, L., Van der Meer, T., De Clercq, O., Hoste, V. (2024). ‘Using State-of-the-art Emotion Detection Models in a Crisis Communication Context’. Computational Communication Research, 6 (1). Available at: https://doi.org/10.5117/ccr2024.1.1.debr.
Desmet, B., Hoste, V. (2013). ‘Emotion Detection in Suicide Notes’. Expert Systems with Applications, 40 (16), 6351–6358. Available at: https://doi.org/10.1016/j.eswa.2013.05.050.
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2018). ‘BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1810.04805.
Ekman, P. (1992). ‘An Argument for Basic Emotions’. Cognition and Emotion, 6 (3–4), 169–200. Available at: https://doi.org/10.1080/02699939208411068.
Fleiss, J. L. (1971). ‘Measuring Nominal Scale Agreement Among Many Raters’. Psychological Bulletin, 76 (5), 378–382. Available at: https://doi.org/10.1037/h0031619.
Hayes, A. F., Krippendorff, K. (2007). ‘Answering the Call for a Standard Reliability Measure for Coding Data’. Communication Methods and Measures, 1 (1), 77–89. Available at: https://doi.org/10.1080/19312450709336664.
Landis, J. R., Koch, G. G. (1977). ‘The Measurement of Observer Agreement for Categorical Data’. Biometrics, 33 (1), 159–176. https://doi.org/10.2307/2529310.
Maladry, A., Singh, P., Lefever, E. (2024). ‘Findings of the WASSA 2024 EXALT shared task on Explainability for Cross-Lingual Emotion in Tweets’. In: Proceedings of the 14th Workshop of on Computational Approaches to Subjectivity, Sentiment Social Media Analysis@ACL 2024’, Bangkok, Thailand: Association for Computational
Linguistics. Available at: https://aclanthology.org/2024.wassa-1.43/.
Nandwani, P., Verma, R. (2021). ‘A Review on Sentiment Analysis and Emotion Detection from Text’. Social Network Analysis and Mining, 11(1), 81. Available at: https://doi.org/10.1007/s13278-021-00776-6.
Plutchik, R. (1980). ‘A General Psychoevolutionary Theory of Emotion’. In: Theories of Emotion, 3–33. Available at: https://doi.org/10.1016/B978-0-12-558701-3.50007-7.
Sailunaz, K., Dhaliwal, M., Rokne, J., Alhajj, R. (2018). ‘Emotion Detection from Text and Speech: a Survey’. Social Network Analysis and Mining, 8(1), 28. Available at: https://doi.org/10.1007/s13278-018-0505-2.
Shaver, P., Schwartz, J., Kirson, D., O’Connor, C. (1987). ‘Emotion Knowledge: Further Exploration of a Prototype Approach’. Journal of Personality and Social Psychology, 52(6), 1061–1086. Available at: https://doi.org/10.1037//0022-3514.52.6.1061.
Singh, P., Maladry, A., Lefever, E. (2023). ‘Annotation Guidelines for Labeling Emotion in Multilingual Tweets’. LT3 Technical Report Series. Ghent University. Available at: https://biblio.ugent.be/publication/01HHSH3CDPHWA217Y73PHAVK5R.
Sosea, T., Pham, C., Tekle, A., Caragea, C., Li, J. J. (2022). ‘Emotion Analysis and Detection During COVID-19’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/2107.11020.
Tocoglu, M.A., Alpkocak, A. (2018). ‘TREMO: A Dataset for Emotion Analysis in Turkish’. Journal of Information Science, 44(6), 848–860. Available at: https://doi.org/10.1177/0165551518761014.
Tocoglu, M. A., Ozturkmenoglu, O., Alpkocak, A. (2019). ‘Emotion Analysis From Turkish Tweets Using Deep Neural Networks’. IEEE Access, 7, 183061–183069. Available at: https://doi.org/10.1109/ACCESS.2019.2960113.
Tocoglu, M. A., Alpkocak, A. (2019). ‘Lexicon-based Emotion Analysis in Turkish’. TÜBİTAK Academic Journals. Available at: https://journals.tubitak.gov.tr/elektrik/vol27/iss2/40/.
Ucan, A., Dorterler, M., Akcapinar Sezer, E. (2021). ‘A Study of Turkish Emotion Classification with Pretrained Language Models’. Journal of Information Science, 48(6), 857–865. Available at: https://doi.org/10.1177/0165551520985507.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., Polosukhin, I. (2017). ‘Attention Is All You Need’. Arxiv.org. Cornell University. Available at: https://arxiv.org/abs/1706.03762.
