Contributors

🎉 🎉 We are making a more advance reading list for hate speech papers here. Please checkout and provide feedback

Reading List for Hate Speech Research 📖

A reading list of relevant research papers on Hate speech and related issues. This list is maintained by Binny Mathew and Punyajoy Saha, at CNeRG Lab, IIT Kharagpur

In the past few years we have witnessed an increase in the number of hate speech incidents world wide. While there is a rich literature in the social sciences, the research on the computational aspects have just started. This list is an effort to create a one stop comprehensive guide for all research related to Hate Speech. The list is still incomplete and the categorization might be inappropriate.

We will keep adding papers and improving the list. Any suggestions are super welcome 😄

Table of Contents

1. Introduction
2. Dataset Papers
3. Survey Papers
4. Interesting Papers
5. Code Available
6. Shared Tasks
7. Papers
   • Amharic
   • Arabic
   • Dutch
   • English
   • German
   • Hindi
   • Indonesian
   • Italian
   • Kenya
   • Vietnamese
8. Videos
9. Blogs
10. Interesting Reads

Introduction

The Internet is one of the greatest innovations of mankind which has brought together people from every race, religion, and nationality. Social media sites such as Twitter and Facebook have connected billions of people and allowed them to share their ideas and opinions instantly. That being said, there are several ill consequences as well such as online harassment, trolling, cyber-bullying, and hate speech.

Datasets

• Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. CONAN - COunter NArratives through Nichesourcing:a Multilingual Dataset of Responses to Fight Online Hate Speech. 2019. ACL
• Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. A Benchmark Dataset for Learning to Intervene in Online Hate Speech". 2019. arXiv
• Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. Learning to Decipher Hate Symbols. 2019. NAACL
• Binny Mathew, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherje. Thou shalt not hate: Countering Online Hate Speech. 2019. ICWSM
• Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. 2018. ICWSM
• Ziqi Zhang, David Robinson, and Jonathan Tepper. Detecting hate speech on Twitter using a convolution-GRU based deep neural network. 2018. European Semantic Web Conference
• Ona de Gibert, Naiara Perez, Aitor García Pablos, and Montse Cuadros. Hate Speech Dataset from a White Supremacy Forum. 2018. ALW2
• Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Stranisci Marco. An italian Twitter corpus of hate speech against immigrants. 2018. LREC ELRA
• Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. Hate lingo: A target-based linguistic analysis of hate speech in social media. 2018. ICWSM
• GAB Dataset Link --> What is gab: A bastion of free speech or an alt-right echo chamber. 2018 . WWW
• Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. 2017. ICWSM
• Lei Gao, and Ruihong Huang. Detecting Online Hate Speech Using Context Aware Models. 2017. RANLP
• Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. 2016. NAACL SRW
• Dataset Link for the paper A Measurement Study of Hate Speech in Social Media
• Dataset Link for the paper Measuring the Reliability of Hate Speech Annotations:The Case of the European Refugee Crisis.

Survey Papers

• Paula Fortuna and Sérgio Nunes. A survey on automatic detection of hate speech in text. 2018. ACM Computing Surveys (CSUR)
• Anna Schmidt and Michael Wiegand. A survey on hate speech detection using natural language processing. 2017. SocialNLP

Interesting Papers/Books

• Aymé Arango, Jorge Pérez and Barbara Poblete,Hate Speech Detection is Not as Easy as You May Think: A Closer Look at Model Validation. 2019 . SIGIR
• Iginio Gagliardone, Danit Gal, Thiago Alves, and Gabriela Martinez. Countering online hate speech. 2015. Unesco Publishing
• Nadine Strossen HATE: Why We Should Resist it with Free Speech, Not Censorship. A short book review

Code Available

• Automated Hate Speech Detection and the Problem of Offensive Language
• Deep Learning for Hate Speech Detection in Tweets
• Hateminers: Detecting Hate speech against Women
• Detecting Online Hate Speech Using Context Aware Models
• CLiPS HAte speech DEtection System (HADES)
• DeEpLearning models for MultIlingual haTespeech

Shared Tasks

• Cristina Bosco, Dell'Orletta Felice, Fabio Poletto, Manuela Sanguinetti, and Tesconi Maurizio. Overview of the EVALITA 2018 Hate Speech Detection Task. 2018. EVALITA
• Valerio Basile, Cristina Bosco,Elisabetta Fersini,Debora Nozza, Viviana Patti, Francisco Rangel ,Paolo Rosso and Manuela Sanguinetti, Page 54-63, Proceedings of 13th SemEval Workshop. Archived competition link-HatEval@SemEval 2019
• Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. 2019. International Workshop on Semantic Evaluation

Language Wise

Amharic

2018

• Zewdie Mossie, and Jenq-Haur Wang. SOCIAL NETWORK HATE SPEECH DETECTION FOR AMHARIC LANGUAGE. 2018. Computer Science & Information Technology

Arabic

2019

• Arijit Ghosh Chowdhury, Aniket Didolkar, Ramit Sawhney and Rajiv Ratn Shah. ARHNet - Leveraging Community Interaction For Detection Of ReligiousHate Speech In Arabic. 2019. ACL:Student Research Workshop

2018

• Nuha Albadi, Maram Kurdi, and Shivakant Mishra. Are they Our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere. 2018. ASONAM
• Sarah Eissa. Use of hate speech in Arabic language newspapers (2018).

Dutch

2016

• Stéphan Tulkens, Lisa Hilte, Elise Lodewyckx, Ben Verhoeven, and Walter Daelemans. A Dictionary-based Approach to Racism Detection in Dutch Social Media. 2016. TA-COS

English

2020

• Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. Exploring Hate Speech Detection in Multimodal Publications. 2020. IEEE Winter Conference on Applications of Computer Vision
• Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou. From Pick-Up Artists to Incels: A Data-Driven Sketch of the Manosphere. 2020. arXiv

2019

• Bertie Vidgen, Taha Yasseri, and Helen Margetts. Trajectories of Islamophobic hate amongst far right actors on Twitter. 2019 arXiv
• Kosisochukwu Judith Madukwe, and Xiaoying Gao. The Thin Line Between Hate and Profanity. 2019. Australasian Joint Conference on Artificial Intelligence
• Sohail Akhtar, Valerio Basile, and Viviana Patti. A New Measure of Polarization in the Annotation of Hate Speech. 2019. International Conference of the Italian Association for Artificial Intelligence (AI*IA)
• Akash Gautam, Puneet Mathur, Rakesh Gosangi, Debanjan Mahata, Ramit Sawhney, and Rajiv Ratn Shah. # MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement. 2019. arXiv
• Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. A BERT-based transfer learning approach for hate speech detection in online social media. 2019. International Conference on Complex Networks and Their Applications
• Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. Stereotypical bias removal for hate speech detection task using knowledge-based generalizations. 2019. The World Wide Web Conference (WWW)
• Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A. Smith. The Risk of Racial Bias in Hate Speech Detection. 2019. ACL
• Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. Racial Bias in Hate Speech and Abusive Language Detection Datasets. 2019. arXiv
• Wafa Alorainy, Pete Burnap, Han Liu, and Matthew L. Williams. "The Enemy Among Us": Detecting Cyber Hate Speech with Threats-based Othering Language Embeddings. 2019. ACM Transactions on the Web (TWEB)
• Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. A Benchmark Dataset for Learning to Intervene in Online Hate Speech". 2019. arXiv
• Kristian Miok, Dong Nguyen-Doan, Blaž Škrlj, Daniela Zaharie, and Marko Robnik-Šikonja. Prediction Uncertainty Estimation for Hate Speech Classification. 2019. arXiv
• Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. Learning to Decipher Hate Symbols. 2019. arXiv
• Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherje. Thou shalt not hate: Countering Online Hate Speech. 2019. ICWSM
• Binny Mathew, Ritam Dutt, Pawan Goyal, and Animesh Mukherjee. Spread of hate speech in online social media. 2019. WebSci
• Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova. Author Profiling for Hate Speech Detection. 2019. arXiv
• T. Y. S. S. Santosh, and K. V. S. Aravind. Hate Speech Detection in Hindi-English Code-Mixed Social Media Text. 2019. CoDS-COMAD
• Junanda Patihullah, and Edi Winarko. Hate Speech Detection for Indonesia Tweets Using Word Embedding And Gated Recurrent Unit. 2019. IJCCS

2018

• Wiktor Soral, Michał Bilewicz, and Mikołaj Winiewski. Exposure to hate speech increases prejudice through desensitization. 2018. Aggressive behavior
• Mai ElSherief, Shirin Nilizadeh, Dana Nguyen, Giovanni Vigna, and Elizabeth Belding. Peer to peer hate: Hate speech instigators and their targets. 2018. ICWSM
• Savvas Zannettou, Barry Bradlyn, Emiliano De Cristofaro, Haewoon Kwak, Michael Sirivianos, Gianluca Stringini, and Jeremy Blackburn. What is gab: A bastion of free speech or an alt-right echo chamber. 2018. WWW
• Punyajoy Saha, Binny Mathew, Pawan Goyal, and Animesh Mukherjee. Hateminers: Detecting Hate speech against Women. 2018. arXiv
• Resham Ahluwalia, Himani Soni, Edward Callow, Anderson Nascimento, and Martine De Cock. Detecting Hate Speech Against Women in English Tweets. 2018. EVALITA
• Ziqi Zhang, and Lei Luo. Hate speech detection: A solved problem? The challenging case of long tail on Twitter. 2018. Semantic Web Preprint
• Karsten Müller, and Carlo Schwarz. Fanning the flames of hate: Social media and hate crime. 2018. SSRN
• Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. Leveraging Intra-User and Inter-User Representation Learning for Automated Hate Speech Detection. 2018. NAACL
• David Robinson, Ziqi Zhang, and Jonathan Tepper. Hate speech detection on twitter: Feature engineering vs feature selection. 2018. European Semantic Web Conference
• Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. Hierarchical CVAE for Fine-Grained Hate Speech Classification. 2018. EMNLP
• Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. All You Need is "Love": Evading Hate-speech Detection. 2018. arXiv
• Shervin Malmasi, and Marcos Zampieri. Challenges in discriminating profanity from hate speech. 2018. Journal of Experimental & Theoretical Artificial Intelligence
• Joel Finkelstein, Savvas Zannettou, Barry Bradlyn, and Jeremy Blackburn. A quantitative approach to understanding online antisemitism. 2018. arXiv
• Alexandra Olteanu, Carlos Castillo, Jeremy Boy and Kush R. Varshney. The Effect of Extremist Violence on Hateful Speech Online. 2018. ICWSM
• Bertie Vidgen, and Taha Yasseri. Detecting weak and strong Islamophobic hate speech on social media. 2018. arXiv
• Manoel Horta Ribeiro, Pedro H. Calais, Yuri A. Santos, Virgílio AF Almeida, and Wagner Meira Jr. Characterizing and detecting hateful users on twitter. 2018. ICWSM
• Joni Salminen, Fabio Veronesi, Hind Almerekhi, Soon-Gvo Jung, and Bernard J. Jansen. Online Hate Interpretation Varies by Country, But More by Individual: A Statistical Analysis Using Crowdsourced Ratings. 2018. SNAMS
• Phoey Lee Teh, Chi-Bin Cheng, and Weng Mun Chee. Identifying and categorising profane words in hate speech. 2018. ICCDA
• Shruti Phadke, Jonathan Lloyd, James Hawdon, Mattia Samory, and Tanushree Mitra. Framing Hate with Hate Frames: Designing the Codebook. 2018. CSCW
• Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard J. Jansen. Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media. 2018. ICWSM

2017

• Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. Deep learning for hate speech detection in tweets. 2017. WWW
• Haji Mohammad Saleem, Kelly P. Dillon, Susan Benesch, and Derek Ruths. A web of hate: Tackling hateful speech in online social spaces. 2017. TA-COS
• Eshwar Chandrasekharan, Umashanthi Pavalanathan, Anirudh Srinivasan, Adam Glynn, Jacob Eisenstein, and Eric Gilbert. You can't stay here: The efficacy of reddit's 2015 ban examined through hate speech. 2017. CSCW
• Björn Gambäck, and Utpal Kumar Sikdar. Using convolutional neural networks to classify hate-speech. 2017. Workshop on Abusive Language Online
• Rijul Magu, Kshitij Joshi, and Jiebo Luo. Detecting the hate code on social media. 2017. ICWSM
• Lucas Wright, Derek Ruths, Kelly P. Dillon, Haji Mohammad Saleem, and Susan Benesch Vectors for counterspeech on Twitter. 2017. Workshop on Abusive Language Online

2016

• Sarah Hewitt, Thanassis Tiropanis, and Christian Bokhove. The problem of identifying misogynist language on Twitter (and other online social spaces. 2016. WebSci
• Pete Burnap, and Matthew L. Williams. Us and them: identifying cyber hate on Twitter across multiple protected characteristics. 2016. EPJ Data Science
• Imran Awan. Islamophobia on Social Media: A Qualitative Analysis of the Facebook's Walls of Hate. 2016. International Journal of Cyber Criminology
• Silva, Leandro, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. Analyzing the targets of hate in online social media. 2016. ICWSM
• Zeerak Waseem. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. 2016. NLP and Computational Social Science

2015

• Njagi Dennis Gitari, Zhang Zuping, Hanyurwimfura Damien, and Jun Long. A lexicon-based approach for hate speech detection. 2015. International Journal of Multimedia and Ubiquitous Engineering
• Shuhua Liu, and Thomas Forss. New classification models for detecting Hate and Violence web content. 2015. IC3K
• Fabio Fasoli, Anne Maass, and Andrea Carnaghi. Labelling and discrimination: Do homophobic epithets undermine fair distribution of resources?. 2015. British Journal of Social Psychology
• Jamie Bartlett and Alex Krasodomski-Jones. Counter-speech examining content that challenges extremism online. 2015 DEMOS
• Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. Hate speech detection with comment embeddings. 2015. WWW

2014

• Jamie Bartlett, Richard Norrie, Sofia Patel, Rebekka Rumpel, and Simon Wibberley. Misogyny on twitter. 2014. Demos
• Susan Benesch. Countering dangerous speech: new ideas for genocide prevention. 2014.

2013

• Irene Kwok, and Yuzhou Wang. Locate the hate: Detecting tweets against blacks. 2013. AAAI
• Michal Bilewicz, Mikołaj Winiewski, Mirosław Kofta, and Adrian Wójcik. Harmful Ideas, The Structure and Consequences of Anti‐Semitic Beliefs in Poland. 2013. Political Psychology

2012

• Karmen Erjavec and Melita Poler Kovačič. “You Don't Understand, This is a New War!” Analysis of Hate Speech in News Web Sites' Comments. 2012. Mass Communication and Society

2002

• Vasu Reddy. Perverts and sodomites: Homophobia as hate speech in Africa. 2002. Southern African Linguistics and Applied Language Studies

German

2019

• Sylvia Jaki, and Tom De Smedt. Right-wing German hate speech on Twitter: Analysis and automatic detection. 2019. arXiv

2017

• Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. 2017. NLP4CMC

Indonesian

2017

• Ika Alfina, Rio Mulia, Mohamad Ivan Fanany, and Yudo Ekanata. Hate speech detection in the indonesian language: A dataset and preliminary study. 2017. ICACSIS

2018

• M. Ali Fauzi and Anny Yuniarti. Ensemble Method for Indonesian Twitter Hate Speech Detection. 2018. Indonesian Journal of Electrical Engineering and Computer Science

Italian

2018

• Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Stranisci Marco. An italian Twitter corpus of hate speech against immigrants. 2018. LREC ELRA

2017

• Fabio Del Vigna, Andrea Cimino, Felice Dell'Orletta, Marinella Petrocchi, and Maurizio Tesconi Hate me, hate me not: Hate speech detection on Facebook. 2017. Italian Conference on Cybersecurity
• Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. Hate speech annotation: Analysis of an italian Twitter corpus. 2017. CLiC-it

Kenya

2014

• Wilson Jeffrey Maloba. Use of regular expressions for multi-lingual detection of hate speech in Kenya. 2014. PhD diss., iLabAfrica

Vietnamese

2020

• Son T. Luu, Hung P. Nguyen, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. Comparison Between Traditional Machine Learning Models And Neural Network Models For Vietnamese Hate Speech Detection. 2020. IEEE RIVF

2019

• Thai Binh Nguyen, Quang Minh Nguyen, Thu Hien Nguyen, Ngoc Phuong Pham, The Loc Nguyen, and Quoc Truong Do. VAIS Hate Speech Detection System: A Deep Learning based Approach for System Combination. 2019. arXiv
• Hang Thi-Thuy Do, Huy Duc Huynh, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen, and Anh Gia-Tuan Nguyen. Hate Speech Detection on Vietnamese Social Media Text using the Bidirectional-LSTM Model. 2019. arXiv

Videos

• Hate Speech Beyond Borders: Nazila Ghanea at TEDxEastEnd. 2012
• Motivation behind studying hatespeech. 2018
• Hate speech and the speech we hate. 2019

Blogs

• No Hate speech Movement

Interesting Reads

• Alexandra A. Siegel. Online Hate Speech. 2018.

TODO's

• Add Abstract and paper's contribution

• Add labels to categorize paper

• Build a Github Page

• A list of papers for Beginners

• Add more papers