Overview of EXIST 2021:sEXism Identification in Social neTworks

  1. Rodríguez-Sánchez, Francisco
  2. Carrillo-de-Albornoz, Jorge
  3. Plaza Morales, Laura
  4. Gonzalo Arroyo, Julio
  5. Rosso, Paolo
  6. Comet, Miriam
  7. Donoso, Trinidad
Journal:
Procesamiento del lenguaje natural

ISSN: 1135-5948

Year of publication: 2021

Issue: 67

Pages: 195-207

Type: Article

DIALNET GOOGLE SCHOLAR lock_openRUA editor

More publications in: Procesamiento del lenguaje natural

Abstract

The paper describes the organization, goals, and results of the sEXism Identification in Social neTworks (EXIST) challenge, a shared task proposed for the first time at IberLEF 2021. EXIST 2021 proposes two challenges: sexism identification and sexism categorization of tweets and gabs, both in Spanish and English. We have received a total of 70 runs for the sexism identification task and 61 for the sexism categorization challenge, submitted by 31 different teams from 11 countries. We present the dataset, the evaluation methodology, an overview of the proposed systems, and the results obtained. The final dataset consists of more than 11,000 annotated texts from two social networks (Twitter and Gab) and its development has been supervised and monitored by experts in gender issues.

Bibliographic References

  • Amigó, E., J. Carrillo-de Albornoz, M. Almagro-Cádiz, J. Gonzalo, J. Rodríguez-Vidal, and F. Verdejo. 2017. Evall: Open access evaluation for information access systems. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1301–1304.
  • Amigó, E., J. Gonzalo, S. Mizzaro, and J. Carrillo-de Albornoz. 2020. An effectiveness metric for ordinal classification: Formal properties and experimental results. arXiv preprint arXiv:2006.01245.
  • Amigó, E., D. Spina, and J. Carrillo-de Albornoz. 2018. An axiomatic analysis of diversity evaluation metrics: Introducing the rank-biased utility metric. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 625–634.
  • Barbieri, F., L. E. Anke, and J. CamachoCollados. 2021. Xlm-t: A multilingual language model toolkit for twitter. arXiv preprint arXiv:2104.12250.
  • Barbieri, F., J. Camacho-Collados, L. Espinosa Anke, and L. Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online, November. Association for Computational Linguistics.
  • Basile, V., C. Bosco, E. Fersini, N. Debora, V. Patti, F. M. R. Pardo, P. Rosso, M. Sanguinetti, et al. 2019. Semeval2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In 13th International Workshop on Semantic Evaluation, pages 54–63. Association for Computational Linguistics.
  • Bassignana, E., V. Basile, and V. Patti. 2018. Hurtlex: A multilingual lexicon of words to hurt. In 5th Italian Conference on Computational Linguistics, CLiCit 2018, volume 2253, pages 1–6. CEURWS.
  • Baumgartner, J., S. Zannettou, B. Keegan, M. Squire, and J. Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830–839.
  • Berg, S. H. 2006. Everyday sexism and posttraumatic stress disorder in women: A correlational study. Violence Against Women, 12(10):970–988. Canete, J., G. Chaperon, R. Fuentes, and J. Pérez. 2020. Spanish pre-trained bert model and evaluation data. PML4DC at ICLR, 2020.
  • Chiril, P., V. Moriceau, F. Benamara, A. Mari, G. Origgi, and M. CoulombGully. 2020. He said “who’s gonna take care of your children when you are at ACL?”: Reported sexist acts are not sexist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4055–4066, Online, July. Association for Computational Linguistics.
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2019. Unsupervised crosslingual representation learning at scale. arXiv preprint arXiv:1911.02116.
  • Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  • Donoso-Vázquez, T. and Rebollo-Catalán. 2018. Violencias de género en entornos virtuales. Ediciones Octaedro.
  • Fast, E., B. Chen, and M. S. Bernstein. 2016. Empath: Understanding topic signals in large-scale text. In Proceedings of the 2016 CHI conference on human factors in computing systems, pages 4647–4657.
  • Fersini, E., P. Rosso, and M. Anzovino. 2018. Overview of the task on automatic misogyny identification at ibereval 2018. IberEval@ SEPLN, 2150:214–228.
  • Fox, J., C. Cruz, and J. Y. Lee. 2015. Perpetuating online sexism offline: Anonymity, interactivity, and the effects of sexist hashtags on social media. Computers in Human Behavior, 52:436–442.
  • Frenda, S., B. Ghanem, M. Montes-y Gómez, and P. Rosso. 2019. Online hate speech against women: Automatic identification of misogyny and sexism on twitter. Journal of Intelligent & Fuzzy Systems, 36(5):4743–4752.
  • He, P., X. Liu, J. Gao, and W. Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
  • Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431. Association for Computational Linguistics, April.
  • Liu, Y., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Manne, K. 2017. Down girl: The logic of misogyny. Oxford University Press.
  • Martínez-Cámara, E., M. Díaz-Galiano, M. García-Cumbreras, M. García-Vega, and J. Villena-Román. 2017. Overview of tass 2017. IberEval@ SEPLN, 1896:13–21.
  • Mills, S. 2008. Language and sexism. Cambridge University Press.
  • Montes, M., P. Rosso, J. Gonzalo, E. Aragón, R. Agerri, M. Angel Alvarez Carmona, E. Alvarez Mellado, J. C. de Albornoz, L. Chiruzzo, L. Freitas, H. G. Adorno, Y. Gutiérrez, S. Lima, A. Montejo-Ráez, F. M. P. de Arco, and M. Taulé. 2021. Proceedings of the iberian languages evaluation forum (iberlef 2021). In CEUR Workshop Proceedings.
  • Parikh, P., H. Abburi, P. Badjatiya, R. Krishnan, N. Chhaya, M. Gupta, and V. Varma. 2019. Multi-label categorization of accounts of sexism using a neural framework. arXiv preprint arXiv:1910.04602.
  • Rodríguez-Sánchez, F., J. Carrillo-de Albornoz, and L. Plaza. 2020. Automatic classification of sexism in social networks: An empirical study on twitter data. IEEE Access, 8:219563–219576.
  • Swim, J., L. Hyers, L. Cohen, and M. Ferguson. 2001. Everyday sexism: Evidence for its incidence, nature, and psychological impact from three daily diary studies. Journal of Social Issues, 57:31 – 53.
  • Waseem, Z. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas, November. Association for Computational Linguistics.
  • Waseem, Z. and D. Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California, June. Association for Computational Linguistics.
Data source: Dialnet