LLM and Society

Recent papers on (1) Psychology of LLMs; (2) Biases in LLMs.

Trait Theory

• [Paper] Evaluating and Inducing Personality in Pre-trained Language Models, Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, Yixin Zhu, June 2022
• [Paper] Who is GPT-3? An Exploration of Personality, Values and Demographics, Marilù Miotto, Nicola Rossberg, Bennett Kleinberg, September 2022
• [Paper] Estimating the Personality of White-Box Language Models, Saketh Reddy Karra, Son The Nguyen, Theja Tulabandhula, April 2022
• [Paper] The Self-Perception and Political Biases of ChatGPT, Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Markus Pauly, April 2023
• [Paper] Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics, Peter Romero, Stephen Fitz, Teruo Nakatsuma, May 2023
• [Paper] Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs, Xiaoyang Song, Akshat Gupta, Kiyan Mohebbizadeh, Shujie Hu, Anant Singh, May 2023
• [Paper] ChatGPT an ENFJ, Bard an ISTJ: Empirical Study on Personalities of Large Language Models, Jen-tse Huang, Wenxuan Wang, Man Ho Lam, Eric John Li, Wenxiang Jiao, Michael R. Lyu, May 2023
• [Paper] Personality testing of GPT-3: Limited temporal reliability, but highlighted social desirability of GPT-3's personality instruments results, Bojana Bodroza, Bojana M. Dinic, Ljubisa Bojic, June 2023
• [Paper] Personality Traits in Large Language Models, Mustafa Safdari, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, Maja Matarić, July 2023
• [Paper] Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models, Keyu Pan, Yawen Zeng, July 2023
• [Paper] AI Text-to-Behavior: A Study In Steerability, David Noever, Sam Hyams, August 2023
• [Paper] CharacterChat: Learning towards Conversational AI with Personalized Social Support, Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao, Ran Wang, Rui Yan, August 2023
• [Paper] Investigating the Applicability of Self-Assessment Tests for Personality Measurement of Large Language Models, Akshat Gupta, Xiaoyang Song, Gopala Anumanchipalli, September 2023
• [Paper] Does Role-Playing Chatbots Capture the Character Personalities? Assessing Personality Traits for Role-Playing Chatbots, Xintao Wang, Quan Tu, Yaying Fei, Ziang Leng, Cheng Li, October 2023

Other Psychological Topics

• [Paper] Does GPT-3 Demonstrate Psychopathy? Evaluating Large Language Models from a Psychological Perspective, Xingxuan Li, Yutong Li, Shafiq Joty, Linlin Liu, Fei Huang, Lin Qiu, Lidong Bing, December 2022
• [Paper] "Correct answers" from the psychology of artificial intelligence, Peter S. Park, Philipp Schoenegger, Chongyang Zhu, February 2023
• [Paper] Inducing anxiety in large language models increases exploration and bias, Julian Coda-Forno, Kristin Witte, Akshay K. Jagadish, Marcel Binz, Zeynep Akata, Eric Schulz, April 2023
• [Paper] ChatGPT Is More Likely to Be Perceived as Male Than Female, Jared Wong, Jin Kim, May 2023
• [Paper] Emotional Intelligence of Large Language Models, Xuena Wang, Xueting Li, Zi Yin, Yue Wu, Liu Jia, July 2023
• [Paper] EmotionPrompt: Leveraging Psychology for Large Language Models Enhancement via Emotional Stimulus, Cheng Li, Jindong Wang, Kaijie Zhu, Yixuan Zhang, Wenxin Hou, Jianxun Lian, Xing Xie, July 2023
• [Paper] Is GPT a Computational Model of Emotion? Detailed Analysis, Ala N. Tak, Jonathan Gratch, July 2023
• [Paper] Evaluating the Moral Beliefs Encoded in LLMs, Nino Scherrer, Claudia Shi, Amir Feder, David M. Blei, July 2023
• [Paper] Exploring the Psychology of GPT-4's Moral and Legal Reasoning, Guilherme F. C. F. Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, Marcelo de Araújo, August 2023
• [Paper] The Capability of Large Language Models to Measure Psychiatric Functioning, Isaac R. Galatzer-Levy, Daniel McDuff, Vivek Natarajan, Alan Karthikesalingam, Matteo Malgaroli, August 2023
• [Paper] Exploring ChatGPT's Empathic Abilities, Kristina Schaaff, Caroline Reinig, Tim Schlippe, August 2023
• [Paper] Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench, Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, Michael R. Lyu, August 2023
• [Paper] The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?, Chuanyang Jin, Songyang Zhang, Tianmin Shu, Zhihan Cui, August 2023

Stereotype and Bias

• [Paper] Can ChatGPT Assess Human Personalities? A General Evaluation Framework, Haocong Rao, Cyril Leung, Chunyan Miao, March 2023
• [Paper] Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models, Emilio Ferrara, April 2023
• [Paper] PersonaLLM: Investigating the Ability of GPT-3.5 to Express Personality Traits and Gender Differences, Hang Jiang, Xiajie Zhang, Xubo Cao, Jad Kabbara, May 2023
• [Paper] From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models, Shangbin Feng, Chan Young Park, Yuhan Liu, Yulia Tsvetkov, May 2023
• [Paper] ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages, Sourojit Ghosh, Aylin Caliskan, May 2023
• [Paper] This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models, Bryan Li, Chris Callison-Burch, May 2023
• [Paper] Large Language Models are not Fair Evaluators, Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, Zhifang Sui, May 2023
• [Paper] Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models, Myra Cheng, Esin Durmus, Dan Jurafsky, May 2023
• [Paper] Cognitive network science reveals bias in GPT-3, ChatGPT, and GPT-4 mirroring math anxiety in high-school students, Katherine Elizabeth Abramski, Salvatore Citraro, Luigi Lombardi, Giulio Rossetti, Massimo Stella, May 2023
• [Paper] Exposing Bias in Online Communities through Large-Scale Language Models, Celine Wald, Lukas Pfahler, June 2023
• [Paper] Taught by the Internet, Exploring Bias in OpenAIs GPT3, Ali Ayaz, Aditya Nawalgaria, Ruilian Yin, June 2023
• [Paper] Soft-prompt Tuning for Large Language Models to Evaluate Bias, Jacob-Junqi Tian, David Emerson, Sevil Zanjani Miyandoab, Deval Pandya, Laleh Seyyed-Kalantari, Faiza Khan Khattak, June 2023

Cross-Cultural Difference

• [Paper] The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation, Jochen Hartmann, Jasper Schwenzow, Maximilian Witte, January 2023
• [Paper] Probing Pre-Trained Language Models for Cross-Cultural Differences in Values, Arnav Arora, Lucie-Aimée Kaffee, Isabelle Augenstein, March 2023
• [Paper] Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study, Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, Daniel Hershcovich, March 2023
• [Paper] Whose Opinions Do Language Models Reflect, Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, Tatsunori Hashimoto, March 2023
• [Paper] Having Beer after Prayer? Measuring Cultural Bias in Large Language Models, Tarek Naous, Michael J. Ryan, Wei Xu, May 2023
• [Paper] Don't Trust GPT When Your Question Is Not In English, Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, Grzegorz Kondrak, May 2023
• [Paper] Being Right for Whose Right Reasons?, Terne Sasha Thorn Jakobsen, Laura Cabello, Anders Søgaard, June 2023
• [Paper] Towards Measuring the Representation of Subjective Global Opinions in Language Models, Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, Deep Ganguli, June 2023
• [Paper] Evaluating the Moral Beliefs Encoded in LLMs, Nino Scherrer, Claudia Shi, Amir Feder, David M. Blei, July 2023