Quality of argumentation in political tweets: what is and how to measure it / Qualidade da argumentação em tweets de política: o que e como avaliar

Cássio Faria da Silva; Amanda Pontes Rassi; Jackson Wilke da Cruz Souza; Renata Ramisch; Roger Alfredo de Marci Rodrigues Antunes; Helena de Medeiros Caseli

doi:10.17851/2237-2083.29.4.2537-2586

Quality of argumentation in political tweets: what is and how to measure it / Qualidade da argumentação em tweets de política: o que e como avaliar

Cássio Faria da Silva, Amanda Pontes Rassi, Jackson Wilke da Cruz Souza, Renata Ramisch, Roger Alfredo de Marci Rodrigues Antunes, Helena de Medeiros Caseli

Abstract

Abstract: Argumentation is something inherent to human beings and essential to written and spoken communication. Because of the popularization of Internet access, social media are one of the main means of creation and profusion of argumentative texts in various fields, such as politics. As a way to contribute to research related to the assessment of the quality of argumentation in Portuguese, we aim in this paper to propose and validate criteria and guidelines for the assessment of the quality of argumentation in Twitter posts in the domain of politics. For this purpose, a corpus was produced and annotated with tweets whose content is related to the Brazilian political scenario. The texts were collected in the first months of 2021, resulting in 1,649,674 posts. From the analysis of a sample, we defined linguistic criteria that would potentially characterize relevant aspects of the rhetorical dimension of argumentation, namely: (i) Clarity, (ii) Arrangement, (iii) Credibility, and (iv) Emotional appeal. After this phase of analysis, we proposed the annotation of a new set of 400 tweets, by four annotators. As a result, an agreement of around 70% for three out of four annotators was obtained. It is worth noting that this is the first work that proposes linguistic criteria for the evaluation of the quality of argumentation in social medias for Brazilian Portuguese. It is intended to construct a computer model that can automatically evaluate the quality of argumentation in social media messages, such as Twitter, based on the establishment of linguistic criteria, annotation rules, and annotated corpus.

Keywords: argumentation; corpus; quality; rhetorical dimension; tweets; politics.

Resumo: A argumentação é algo inerente ao ser humano e essencial para a comunicação escrita e falada. Por conta da popularização do acesso à Internet, as redes sociais são um dos principais meios de criação e profusão de textos argumentativos de vários domínios, como a política. Como forma de contribuir com as pesquisas relacionadas à avaliação da qualidade da argumentação em português, este trabalho tem como objetivo propor e validar critérios e diretrizes para a avaliação da qualidade da argumentação em postagens no Twitter no domínio da política. Para tanto, produziu-se um corpus anotado com tweets cujo conteúdo relaciona-se ao cenário político brasileiro. Os textos foram coletados nos primeiros meses de 2021, resultando em 1.649.674 postagens. A partir da análise de uma amostra, foram definidos critérios linguísticos que potencialmente caracterizariam aspectos relevantes da dimensão retórica da argumentação, a saber: (i) Clareza, (ii) Organização, (iii) Credibilidade e (iv) Apelo emocional. Após essa fase de análise, propôs-se a anotação de um novo conjunto de 400 tweets, por quatro anotadores. Como resultado, obteve-se uma concordância de cerca de 70% entre 3 dos 4 anotadores. Vale ressaltar que esse é o primeiro trabalho que propõe critérios linguísticos para a avaliação da qualidade da argumentação em redes sociais para o português brasileiro. A partir da definição dos critérios linguísticos, diretrizes de anotação e corpus anotado, espera-se construir um modelo computacional que possa avaliar automaticamente a qualidade da argumentação em textos de redes sociais, como o Twitter.

Palavras-chave: argumentação; corpus; qualidade; dimensão retórica; tweets; política.

Keywords

argumentation; corpus; quality; rhetorical dimension; tweets; politics; argumentação; corpus; qualidade; dimensão retórica; tweets; política.

Full Text:

PDF

References

BENCH-CAPON, T. J.; DUNNE, P. E. Argumentation in Artificial Intelligence. Artificial Intelligence, [S.l.], v. 171, n. 10-15, p. 619-641, 2007. DOI: https://doi.org/10.1016/j.artint.2007.05.001

BLAIR, J. A. Rhetoric, Dialectic, and Logic as Related to Argument. Philosophy & Rhetoric, Universtity Park, PA, v. 45, n. 2, p. 148-164, 2012. DOI: https://doi.org/10.5325/philrhet.45.2.0148. Available in: http://www.jstor.org/stable/10.5325/philrhet.45.2.0148. Access on: May 20, 2021.

BOUDRY, M.; PAGLIERI, F.; PIGLIUCCI, M. The Fake, the Flimsy, and the Fallacious: Demarcating Arguments in Real Life. Argumentation, [S.l.], v. 29, n. 4, p. 431-456, 2015. DOI: http://doi.org/10.1007/s10503-015-9359-1

BRADY, W. J.; WILLS, J. A.; JOST, J. T.; TUCKER, J. A.; VAN BAVEL, J. J. Emotion Shapes the Diffusion of Moralized Content in Social Networks. Proceedings of the National Academy of Sciences, [S.l.], v. 114, n. 28, p. 7313-7318, 2017. DOI: https://doi.org/10.1073/pnas.1618923114

CARLILE, W.; GURRAPADI, N.; KE, Z.; NG, V. Give Me More Feedback: Annotating Argument Persuasiveness and Related Attributes in Student Essays. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 56., 2018, Melbourne. Proceedings […]. Melbourne: Association for Computational Linguistics, 2018. p. 621-631. DOI: https://doi.org/10.18653/v1/P18-1058. Available in: https://www.aclweb.org/anthology/P18-1058. Access on: May. 20, 2021.

EEMEREN, F. H. V.; GROOTENDORST, R. Fallacies in Pragma-Dialectical Perspective. Argumentation, [S.l.], v. 1, n. 3, p. 283-301, 1987. DOI: http://doi.org/10.1007/BF00136779.

EEMEREN, F. H. V.; GROOTENDORST, R. A Systematic Theory of Argumentation: The Pragma-Dialectical Approach. Cambridge: Cambridge University Press, 2003. DOI: https://doi.org/10.1017/CBO9780511616389

FREEDOM HOUSE. Freedom on the Net 2019. 2019. Available in: https://freedomhouse.org/sites/default/files/2019-11/11042019_Report_FH_FOTN_2019_final_Public_Download.pdf. Access on: May 20, 2021.

FREITAS, E. C.; BARTH, P. A. Gênero ou suporte? O entrelaçamento de gêneros no Twitter. Revista (Con)Textos Linguísticos, Vitória, v. 9, n. 12, p. 8-26, 2015.

GARCÍA-GORROSTIETA, J. M.; LÓPEZ-LÓPEZ, A. Identifying Argumentative Paragraphs: Towards Automatic Assessment of Argumentation in Theses. In: SILBERZTEIN, M.; ATIGUI, F.; KORNYSHOVA, E.; MÉTAIS, E.; MEZIANE, F. (ed.). Natural Language Processing and Information Systems. Cham: Springer International Publishing, 2018. p. 83-90. DOI: https://doi.org/10.1007/978-3-319-91947-8_9

GARCÍA-GORROSTIETA, J. M.; LÓPEZ-LÓPEZ, A.; GONZÁLEZ-LÓPEZ, S. Automatic Argument Assessment of Final Project Reports of Computer Engineering Students. Computer Applications in Engineering Education, [S.l.], v. 26, n. 5, p. 1217-1226, 2018. DOI: https://doi.org/10.1002/cae.21996

GLEIZE, M.; SHNARCH, E.; CHOSHEN, L.; DANKIN, L.; MOSHKOWICH, G.; AHARO-NOV, R.; SLONIM, N. Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 57., 2019, Florence. Proceedings […]. Florence: Association for Computational Linguistics, 2019. p. 967-976. DOI: https://doi.org/10.18653/v1/P19-1093. Available in: https://www.aclweb.org/anthology/P19-1093. Access on: May. 20, 2021.

HABERNAL, I.; GUREVYCH, I. Which Argument Is More Convincing? Analyzing and Predicting Convincingness of Web Arguments Using Bidirectional LSTM. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 54., 2016, Berlin. Proceedings […]. Berlin: Association for Computational Linguistics, 2016. p. 1589-1599. DOI: https://doi.org/10.18653/v1/P16-1150. Available in: https://www.aclweb.org/anthology/P16-1150. Access on: May. 20, 2021.

HABERNAL, I.; GUREVYCH, I. Argumentation Mining in User-Generated Web Discourse. Computational Linguistics, Singapore, v. 43, n. 1, p. 125-179, 2017. DOI: https://doi.org/10.1162/COLI_a_00276. Available in: https://www.aclweb.org/anthology/J17-1004. Access on: May. 20, 2021.

HOVY, E.; LAVID, J. Towards a ‘Science’ of Corpus Annotation: A New Methodological Challenge for Corpus Linguistics. International Journal of Translation, [S.l.], v. 22, n. 1, p. 13-36, 2010.

KRIPPENDORFF, K. Computing Krippendorff’s Alpha-Reliability. 2011. Available in: http://repository.upenn.edu/asc_papers/43. Access on: May. 20, 2021.

LACY, S.; WATSON, B. R.; RIFFE, D.; LOVEJOY, J. Issues and Best Practices in Content Analysis. Journalism & Mass Communication Quarterly, [S.l.], v. 92, n. 4, p. 791-811, 2015. DOI: https://doi.org/10.1177/1077699015607338.

LAUSCHER, A.; NG, L.; NAPOLES, C.; TETREAULT, J. Rhetoric, Logic, and Dialectic: Advancing Theory-Based Argument Quality Assessment in Natural Language Processing. In: INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS, 28., 2020, Barcelona, Proceedings […]. Barcelona: International Committee on Computational Linguistics, 2020. p. 4563-4574. DOI: https://doi.org/10.18653/v1/2020.coling-main.402

LYTOS, A.; LAGKAS, T.; SARIGIANNIDIS, P.; BONTCHEVA, K. The Evolution of Argumentation Mining: From Models to Social Media and Emerging Tools: Information. Processing & Management, [S.l.]., v 56, n. 6, p. 1-22, 2019. DOI: https://doi.org/10.1016/j.ipm.2019.102055.

MARCUSCHI, L. A. Gêneros textuais: definição e funcionalidade. In: DIONISIO, A. P. et al. Gêneros textuais e ensino. 2. ed. Rio de Janeiro: Lucerna, 2002. p. 19-36.

PERSING, I.; NG, V. Modeling Argument Strength in Student Essays. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 53.; INTERNATIONAL JOINT CONFERENCE ON NATURAL LANGUAGE PROCESSING, 7., 2015, Beijing. Proceedings […]. Beijing: Association for Computational Linguistics, 2015. p. 543-552. DOI: https://doi.org/10.3115/v1/P15-1053. Available in: https://www.aclweb.org/anthology/P15-1053. Access on: May. 20, 2021.

POTTHAST, M.; GIENAPP, L.; EUCHNER, F.; HEILENKöTTER, N.; WEIDMANN, N.;WACHSMUTH, H.; STEIN, B.; HAGEN, M. Argument Search: Assessing Argument Relevance. In: INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL, 42., 2019, New York. Proceedings […]. New York: Association for Computing Machinery, 2019. p. 1117-1120. DOI: https://doi.org/10.1145/3331184.3331327

ROSENFELD, A.; KRAUS, S. Providing Arguments in Discussions Based on the Prediction of Human Argumentative Behavior. 2015. Available in: https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9522. Access on: May. 20, 2021.

ROSSINI, P. Disentangling Uncivil and Intolerant Discourse. In: BOATRIGHT, R.; SOBIERAJ, S.; SHAFFER, T.; YOUNG, D. (ed.). A Crisis of Civility? Contemporary Research on Civility, Incivility, and Political Discourse. New York: Routlegde, 2019. p. 142-157. DOI: https://doi.org/10.4324/9781351051989-9

ROSSINI, P. Beyond Incivility: Understanding Patterns of Uncivil and Intolerant Discourse in Online Political Talk. Communication Research, [S.l.], ahead of print, p. 1-27, 2020. DOI: https://doi.org/10.1177/0093650220921314

STAB, C.; GUREVYCH, I. Parsing Argumentation Structures in Persuasive Essays. Computational Linguistics, Singapore, v. 43, n. 3, p. 619–659, 2017a. DOI: https://doi.org/10.1162/COLI_a_00295. Available in: https://www.aclweb.org/anthology/J17-3005. Access on: May. 20, 2021.

STAB, C.; GUREVYCH, I. Recognizing insufficiently supported arguments in argumentative es-says. In: CONFERENCE OF THE EUROPEAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 15., 2017b, Valencia. Proceedings […]. Valencia: Association for Computational Linguistics, 2017b. p. 980-990. DOI: https://doi.org/10.18653/v1/E17-1092. Available in: https://www.aclweb.org/anthology/E17-1092. Access on: May. 20, 2021.

SWANSON, R.; ECKER, B.; WALKER, M. Argument mining: Extracting arguments from online dialogue. In: ANNUAL MEETING OF THE SPECIAL INTEREST GROUP ON DISCOURSE AND DIALOGUE, 16., 2015, Prague. Proceedings […]. Prague: Association for Computational Linguistics, 2015. p. 217-226. DOI: https://doi.org/10.18653/v1/W15-4631. Available in: https://www.aclweb.org/anthology/W15-4631. Access on: May. 20, 2021.

TOULMIN, S. E. The Uses of Argument. Cambridge: Cambridge University Press, 2003. DOI: https://doi.org/10.1017/CBO9780511840005

WACHSMUTH, H.; AL-KHATIB, K.; STEIN, B. Using argument mining to assess the argumentation quality of essays. In: INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS: TECHNICAL PAPERS, 26., 2016, Osaka. Proceedings […]. Osaka: The COLING 2016 Organizing Committee, 2016. p. 1680-1691.

WACHSMUTH, H.; NADERI, N.; HABERNAL, I.; HOU, Y.; HIRST, G.; GUREVYCH, I.; STEIN, B. Argumentation quality assessment: Theory vs. practice. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 55., 2017, Vancouver. Proceedings […]. Vancouver: Association for Computational Linguistics, 2017a. p. 250-255. DOI: https://doi.org/10.18653/v1/P17-2039. Available in: https://www.aclweb.org/anthology/P17-2039. Access on: May. 20, 2021.

WACHSMUTH, H.; NADERI, N.; HOU, Y.; BILU, Y.; PRABHAKARAN, V.; THIJM, T. A.; HIRST, G.; STEIN, B. Computational argumentation quality assessment in natural language. In: CONFERENCE OF THE EUROPEAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 15, 2017, Valencia. Proceedings […]. Valencia: Association for Computational Linguistics, 2017b. p. 176-187. DOI: https://doi.org/10.18653/v1/E17-1017. Available in: https://www.aclweb.org/anthology/E17-1017. Access on: May. 20, 2021.

WACHSMUTH, H.; POTTHAST, M.; AL-KHATIB, K.; AJJOUR, Y.; PUSCHMANN, J.; QU, J.; DORSCH, J.; MORARI, V.; BEVENDORFF, J.; STEIN, B. Building an Argument Search Engine for the Web. In: WORKSHOP ON ARGUMENT MINING (ARGMINING 2017) AT EMNLP, 4., 2017, Copenhagen. Proceedings […]. Copenhagen: Association for Computational Linguistics, 2017c. p. 49-59. DOI: https://doi.org/10.18653/v1/W17-5106. Available in: https://www.aclweb.org/anthology/W17-5106. Access on: May. 20, 2021.

WACHSMUTH, H.; STEIN, B.; AJJOUR, Y. “PageRank” for argument relevance. In: CONFERENCE OF THE EUROPEAN CHAPTER OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 15., 2017, Valencia. Proceedings […]. Valencia: Association for Computational Linguistics, 2017d. p. 1117-1127. DOI: https://doi.org/10.18653/v1/E17-1105. Available in: https://www.aclweb.org/anthology/E17-1105. Access on: May. 20, 2021.

WACHSMUTH, H.; WERNER, T. Intrinsic Quality Assessment of Arguments. In: INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS, 28., 2020, Barcelona. Proceedings […]. Barcelona: International Committee on Computational Linguistics, 2020. p. 6739-6745. DOI: https://doi.org/10.18653/v1/2020.coling-main.592. Available in: https://www.aclweb.org/anthology/2020.coling-main.592. Access on: May. 20, 2021.

WALTON, D. N.; WALTON, D. N. Informal Logic: A Handbook for Critical Argument. Cambridge: Cambridge University Press, 1989.

WARDLE, C. Misinformation Has Created a New World Disorder. Scientific American, [S.l.], v. 321, p. 88-93, 2019.

WEI, Z.; LIU, Y.; LI, Y. Is this Post Persuasive? Ranking Argumentative Comments in Online Forum. In: ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 54., 2016, Berlin. Proceedings […]. Berlin: Association for Computational Linguistics, 2016. p. 195-200. DOI: https://doi.org/10.18653/v1/P16-2032. Available in: https://www.aclweb.org/anthology/P16-2032. Access on: May. 20, 2021.

WELTZER-WARD, L.; BALTES, B.; LYNN, L. K. Assessing Quality of Critical Thought in Online Discussion. Campus-Wide Information Systems, [S.l.], v. 26, n. 3, p. 168-177, 2009. DOI: http://doi.org/10.1108/10650740910967357

DOI: http://dx.doi.org/10.17851/2237-2083.29.4.2537-2586