Overview of DETESTS at IberLEF 2022DETEction and classification of racial STereotypes in Spanish

  1. Ariza, Alejandro
  2. Schmeisser-Nieto, Wolfgang S.
  3. Nofre, Montserrat
  4. Taulé Delor, Mariona
  5. Amigó Cabrera, Enrique
  6. Chulvi, Berta
  7. Rosso, Paolo
Revista:
Procesamiento del lenguaje natural

ISSN: 1135-5948

Año de publicación: 2022

Número: 69

Páginas: 217-228

Tipo: Artículo

DIALNET GOOGLE SCHOLAR lock_openRUA editor

Otras publicaciones en: Procesamiento del lenguaje natural

Resumen

This paper presents an overview of the DETESTS shared task as part of the IberLEF 2022 Workshop on Iberian Languages Evaluation Forum, within the framework of the SEPLN 2022 conference. We proposed two hierarchical subtasks: For subtask 1, participants had to determine the presence of stereotypes in sentences. For subtask 2, participants had to classify the sentences labeled with stereotypes into ten categoriesEste artículo presenta un resumen de la tarea DETESTS como parte del workshop IberLEF 2022, dentro de la conferencia SEPLN 2022. The DETESTS dataset contains 5,629 sentences in comments in response to newspaper articles related to immigration in Spanish. 51 teams signed up to participate, of which 39 sent runs, and 5 of them sent their working notes. In this paper, we provide information about the training and test datasets, the systems used by the participants, the evaluation metrics of the systems and their results

Referencias bibliográficas

  • Allport, G. W., K. Clark, and T. Pettigrew. 1954. The nature of prejudice. Addison wesley Reading, M
  • Amigo, E. and A. D. Delgado. 2022. Evaluating extreme hierarchical multi label classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 2227, 2022, pages 5809–5819.
  • Amigo, E., F. Giner, J. Gonzalo, and F. Verdejo. 2020. On the foundations of similarity in information access. Inf. Retr. J., 23(3):216–254.
  • Basile, V., M. Fell, T. Fornaciari, D. Hovy, S. Paun, B. Plank, M. Poesio, and A. Uma. 2021. We need to consider dis agreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online, August. Association for Computational Linguistics.
  • Cabestany, D., C. Adsuar, and M. Lopez. 2022. DaMinCi at IberLEF2022 DE TESTS task: Detection and Classification of Racial Stereotypes in Spanish. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Work shop Proceedings, CEURWS.org.
  • Chiril, P., F. Benamara, and V. Moriceau. 2021. “Be Nice to your wife! The Restaurants are Closed”: Can Gender Stereotype Detection Improve Sexism Classification? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2833–2844, Punta Cana, Dominican Re public, November. Association for Computational Linguistics. Costa, E. P., A. C. Lorena, A. C. Carvalho, and A. A. Freitas. 2007. A review of performance evaluation measures for hierarchical classifiers. AAAI Workshop Technical Report, 01.
  • Cryan, J., S. Tang, X. Zhang, M. Metzger, H. Zheng, and B. Y. Zhao, 2020. Detecting Gender Stereotypes: Lexicon vs. Super vised Learning Methods, page 1–11. As sociation for Computing Machinery, New York, NY, USA.
  • Fersini, E., P. Rosso, and M. E. Anzovino. 2018. Overview of the task on automatic misogyny identification at ibereval 2018. In IberEval@SEPLN.
  • Fokkens, A., N. Ruigrok, C. Beukeboom, S. Gagestein, and W. Van Atteveldt. 2019. Studying muslim stereotyping through microportrait extraction. In H. Isahara, B. Maegaard, S. Piperidis, C. Cieri, T. Declerck, K. Hasida, H. Mazo, K. Choukri, S. Goggi, J. Mariani, A. Moreno, N. Calzolari, J. Odijk, and T. Tokunaga, editors, Proceedings of the LREC 2018, Eleventh International Conference on Language Resources and Evaluation, pages 3734–3741. European Language Resources Association (ELRA). Conference date: 07052018 Through 12 052018.
  • Garcıa-Dıaz, J. A., S. M. Jimenez-Zafra, and R. Valencia-Garcıa. 2022. UMUTeam at IberLEF2022 DETESTS task: Feature Engineering for the Identification and Categorization of Racial Stereotypes in Spanish. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Workshop Proceedings, CEURWS.org.
  • Jain, H., Y. Prabhu, and M. Varma. 2016. Extreme multilabel loss functions for recommendation, tagging, ranking & other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 935–944, New York, NY, USA. Association for Computing Machinery.
  • Kiritchenko, S., S. Matwin, and F. Famili. 2004. Hierarchical text categorization as a tool of associating genes with gene ontology codes. Proceedings of the 2nd European Workshop on Data Mining and Text Mining in Bioinformatics, 01.
  • Laknani, F. and M. Garcıa-Martinez. 2022. Lak NLP at IberLEF2022 DETESTS task: Automatic Classification of Stereo types in Text. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Workshop Proceedings, CEURWS.org.
  • Ramirez Ortal, J., M. V. Sabando, M. Maisonnave1, and E. Milios. 2022. MALNIS at IberLEF2022 DETESTS Task: A MultiTask Learning Approach for LowResource Detection of Racial Stereotypes in Spanish. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Workshop Proceedings, CEURWS.org
  • Rodrıguez Sanchez, F., J. C. de Albornoz, L. Plaza, J. Gonzalo, P. Rosso, M. Comet, and T. Donoso. 2021. Overview of exist 2021: sexism identification in social networks. Procesamiento del Lenguaje Natural, 67(0): 195–207.
  • Sanguinetti, M., G. Comandini, E. di Nuovo, S. Frenda, M. Stranisci, C. Bosco, T. Caselli, V. Patti, and I. Russo. 2020. Haspeede 2 @ evalita2020: Overview of the evalita 2020 hate speech detection task. In V. Basile, D. Croce, M. Di Maro, and L. Passaro, editors, Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020), volume 2765. CEUR Workshop Proceedings (CEURWS.org). Conference date: 17122020.
  • Sap, M., S. Gabriel, L. Qin, D. Jurafsky, N. A. Smith, and Y. Choi. 2020. Social bias frames: Reasoning about Social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online, July. Association for Computational Linguistics.
  • Schmeisser Nieto, W., M. Nofre, and M. Taule. 2022. Criteria for the annotation of implicit stereotypes. In Proceedings of the Language Resources and Evaluation Conference, pages 753–762, Marseille, France, June. European Language Resources Association.
  • Sanchez-Junquera, J., B. Chulvi, P. Rosso, and S. P. Ponzetto. 2021. How do you speak about immigrants? taxonomy and stereoimmigrants dataset for identifying stereotypes about immigrants. Applied Sciences, 11(8).
  • Tajfel, H. 1984. Grupos humanos y categorias sociales. Herder.
  • Tajfel, H., A. A. Sheikh, and R. C. Gardner. 1964. Content of stereotypes and the inference of similarity between members of stereotyped groups. Acta Psychologica, 22(3):191–201.
  • Taule, M., A. Ariza, M. Nofre, E. Amigo, and P. Rosso. 2021. Overview of DETOXIS at IberLEF 2021: DEtection of TOXicity in comments In Spanish. Procesamiento del Lenguaje Natural, 67(0):209–221.
  • Uma, A., T. Fornaciari, A. Dumitrache, T. Miller, J. Chamberlain, B. Plank, E. Simpson, and M. Poesio. 2021. SemEval2021 Task 12: Learning with Disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval2021), pages 338– 347, Online, August. Association for Computational Linguistics.
  • Vazquez, J. M., V. P. Alvarez, C. T. Taybi, and P. P. Sanchez. 2022. I2C at IberLEF 2022 DETESTS task: Detection of Racist Stereotypes in Spanish Comments using UnderBagging and Transformers. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Work shop Proceedings, CEURWS.org
Fuente de los datos: Dialnet