source: https://hdsr.mitpress.mit.edu/pub/ujvharkk/release/1 by Yaqub Chaudhary and Jonnie Penn
- Abstract
- 1. Introduction
- 2. ‘Intention’ in LLM Development
- 3. The Weight of Things to Come: Eliciting, Inferring, and Understanding Signals of Intent
- 4. Persuasion and Machinic Projection of Intent
- 5. Conclusion
- Disclosure Statement
Introduction:
- Rapid proliferation of large language models (LLMs) leads to new marketplace for behavioral and psychological data called the "intention economy"
- Invitation to explore features of this emerging marketplace
Background:
- Tech executives position capture, manipulation, and commodification of human intentionality as lucrative extension of attention economy
- Intention economy characterized by:
- Competition between tech players for first-mover advantage on new frontier of persuasive technologies
- Commodification of explicit and implicit data signaling intent
Features of Intention Economy:
- Hyper-personalized manipulation via LLMs
- Combining hyper-personalized manipulation with detailed categorization of online activity
- Automated persuasion drawing on unique capabilities of LLMs and generative AI
- Intervene not only on what users want but also on "what they want to want"
Implications:
- Emerging scholarship and corporate announcements demonstrate eliciting, inferring, collecting, recording, understanding, forecasting, manipulating, modulating, and commodifying human plans and purposes
- Intention economy tests democratic norms by subjecting users to clandestine modes of subverting, redirecting, and intervening on commodified signals of intent.
Keywords:
- Intention economy
- Attention economy
- Large language models
- Generative AI
- Persuasive technology
Expanding Human Data Capture and the Intention Economy
Background:
- Philosopher's definition of 'intention' versus colloquial use in tech industry (Section 2)
Central Aspect of OpenAI's Mission:
- Significantly expand data capture online
- Dominate the intention economy: digital marketplace for commodified signals of intent
Structure:
- Definition and Context (Section 2)
- Philosophers on 'intention' vs tech industry use
- LLMs as Core Infrastructure (Section 3)
- Companies investing in LLMs
- False intentionality projection by developers
- Social Implications of an Intention Economy (Section 5)
- Personalized persuasion-at-scale risks
- Conclusion
Section 2: 'Intention' in LLM Development:
- Philosopher's definition vs colloquial use in tech industry
Section 3: Companies Investing in LLMs:
- Large language models as core infrastructure
- False intentionality projection by developers
Section 4: Persuasion and Machinic Projection of Intent:
- Personalized persuasion at scale risks
Section 5: Social Implications of an Intention Economy:
- Sustained critique needed on the social implications
Intentionality and Large Language Models (LLMs)
Background:
- Wide range of conceptualizations for intention
- Western analytic philosophers focus on purposeful action, consciousness, and mental representations
- Intention's role in corporate strategies and LLM research
Distance from Scientific Conceptualizations:
- Stark and Hoey (2021): No need to accept technical and scientific conceptualizations for psychological phenomena like emotion or intention
- Jarrett (2014): Intentions in digital services are reductive representations of full richness of human motivations
Historical Context:
- Google's "database of intentions" debate: Affirmation vs. concern over impact on human intentionality
- Scholarship from 2000s and 2010s identified Google as a key player in the digital economy, shaping user choices through digital environments
Assumptions Underlying LLM Research:
- Enclosure of Choices: Intentionality arises within a highly structured system, which is assumed to be digital
- Temporal Characteristic: Intentions have a temporal dimension, and the intention economy seeks to profile the arc of users' attention across time scales
Implications for Future Developments:
- LLMs used for anticipating and steering users based on intentional, behavioral, and psychological data
- Concrete example: Intention economy as a digital marketplace for commodified signals of 'intent', differing from present-day attention economy where advertisers can buy access to users' attention in the present or future.
OpenAI Developer Conference (November 6, 2023)
- High-profile event prefiguring events leading to Altman's firing and reorganization of OpenAI's board
- Benefited developers with updates: wider context window, function calling, JSON mode, reproducible outputs, multimodal capabilities, text-to-speech functionality
- Customized versions of ChatGPT called GPTs for competition and revenue sharing
Microsoft's Involvement
- Emphasized focus on building computing infrastructure
- Azure cloud platform accounted for over one-third of company's total revenue in 2022
- Described computational workloads from OpenAI as unprecedented
- Aimed to become the cloud platform, similar to a utility (Azure = "bright blue in color like a cloudless sky")
- Initial one billion dollar investment in OpenAI was significant, comparable to Soviet investment in 1975
Strategic Ambitions
- Inferring human preferences, intentions, motivations from LLMs
- Monopolizing and capitalizing on new data sources for training agentic AI systems (beyond the scope of this article)
OpenAI's Interest in Human Intention Data
- "Looking for data that expresses human intention" (November 9, 2023)
- Continuum from understanding user intent to predicting user action (Miqdad Jaffer, OpenAI Director of Product)
- Large language models as first point of contact between humans and information systems (Jensen Huang, NVIDIA CEO)
Implications for Modern Computing Infrastructure
- LLMs and transformer technologies positioned as primary interface between humans and information systems.
Behavioral Data Extraction from Visual Images
Intentonomy Research:
- Introduced by Meta as a dataset for human intent understanding
- 28 intent categories: security and belonging, power, health, family, ambition, financial success, etc.
- Builds on research on Instagram (Kruk et al., 2019)
Language Models (LLMs):
- Allow for the automation of intent extraction from visual images
- Enabling low-cost and large-scale categorization of human intent and motivation
Applications of LLMs:
- Eliciting human preferences via free-form questions (Li et al., 2023)
- Generating intent taxonomies for user interactions (Shah et al., 2024)
- Incorporated into libraries of products like Teams API library for mapping user intent to actions
Challenges and Ethical Concerns:
- Unestablished scientific grounds for inferring psychological attributes
- Often relies on unrecognized human labor (Gray & Suri, 2019)
- Research not yet peer-reviewed
- Potential for misuse and abuse in digital ad markets, espionage, crime (Ryan & Christl, 2023)
Implications of LLMs:
- Enables highly quantified, dynamic, and personalized targeting based on user intent
- Intimate, low-cost, and ubiquitous conversational brand agents
- Continuous calibration to streams of incoming user data for accurate predictions (Matz et al., 2017)
- Potential for unprecedented modes of hyper-personalized manipulation (Qi Liu et al., 2023; Zhang et al., 2023)
LLMs and Generative AI
- LLMs used to generate content aligned to behavioral profiles, influencing user attitudes, preferences, and behaviors
- Meta's AI agent CICERO: proof-of-concept for inferring human intentions and communicating in natural language
- Goal is to build agents that can plan, coordinate, and negotiate with humans using persuasive communication
- CICERO models opponents' actions and changes their minds through proposals
Exploration of Strategic Reasoning using LLMs
- Corporations investigating use of strategic reasoning in generative AI for higher fidelity calibration of user intents
- Noam Brown, creator of CICERO, joins OpenAI to integrate more planning into language models
Persuasive Capabilities of LLMs
- LLMs shown to transmit false information and biases to humans (Kidd & Birhane, 2023)
- Persuasive characteristics present even without configuring for persuasive messaging
- Latent persuasion: LLM-based predictive suggestions interrupt user thought processes and change views
- Image manipulation used for bias projection and influencing human perception (Vicente & Matute, 2023; Veerabadran et al., 2023)
Generative Models in Synthetic Image Creation
- Generative models create unique QR codes that could be generalized for crafting synthetic images with subliminal or suggestive messages (nhciao, 2023)
Automated Personalized Persuasion using LLMs
- Joyal-Desmarais et al. (2022): LLM-matched content based on users' psychological profiles alters attitudes, intentions, and behaviors
- Matz et al. (2024): LLMs close the loop in automated personalized persuasion by matching message effects to recipients' psychological profiles
- Potential for real-time, dynamically adjusted personalized textual, visual, and audio content (NVIDIA, 2023)
Competition in the Marketplace for Digital Signals of Intent
- OpenAI and Microsoft competing against stiff competition to command a lucrative yet troubling new marketplace for digital signals of intent.
A large-scale intention economy raises concerns about harm and requires scrutiny from scholars, citizens, and regulators. This shift from an "attention" to an "intention" economy would allow diverse actors to influence human actions, impacting democratic values such as free elections, a free press, and fair market competition.
They have no financial or non-financial disclosures to report.