by Louis Rosenberg, Hans Schumann, Anita Woolley https://arxiv.org/abs/2410.03690
- Abstract
- I. Introduction
- II. Conversational Swarm Intelligence (Csi)
- III. MLB FANTASY – GROUP DELIBERATION STUDY
- IV. Results
- V. Conclusions
Conversational Swarm Intelligence (CSI)
Overview:
- AI-powered communication and collaboration technology for real-time conversational deliberations among large, networked groups
- Inspired by decision-making dynamics of fish schools
- Divides human population into small subgroups connected by AI agents, enabling full group to hold unified conversation
Study Description:
- 10 trials with 25 participants each
- Tasked with selecting roster for Fantasy Baseball contest using CSI
- Half the trials used Infobot (AI agent with MLB statistics)
Results:
- CSI-enabled groups outperformed:
- 72% of individually surveyed participants (p=0.016)
- Most popular picks across surveys for each daily contest (p<0.001)
- Infobots promoted more balanced discussions:
- Fewer members dominated the dialog
- Reduced conversational content variance (p=0.039)
Participant Feedback:
- 85% agreed: "Our decisions were stronger because of information provided by the Infobot"
- Only 4% disagreed
Keywords:
- Collective Intelligence
- Human-AI Collaboration
- Decision-Making
- Conversational Swarm Intelligence
- AI, LLMs
Collective Intelligence (CI)
- Human groups can make collaborative estimations, decisions, and forecasts with accuracy beyond individual participants [1, 16, 17]
- Techniques involve collecting and aggregating data from individual members [1]
- Often described as "harnessing the Wisdom of Crowds" (WoC) [2]
- Limited to narrow tasks like numerical estimations and fixed-choice selections [2]
Conversational Swarm Intelligence (CSI)
- New CI methodology and technology that addresses limitations of traditional methods
- Enables large, networked groups (of potentially unlimited size) to hold realtime conversational deliberations
- Groups converge on solutions that increase collective intelligence [3-6]
Infobot Feature in CSI
- New feature introduced for enhancing group decision-making using CSI technology
- Description and academic study testing its use in a real-world forecasting task (Daily Fantasy Baseball) follow.
Study Description
- Comparison of groups' performance via traditional survey aggregation against groups deliberating using:
- Online CSI platform, Thinkscape™ [3-6]
- With and without the use of Infobots
- Collection of subjective feedback about participants' experience and perceived value of Infobots.
Conversational Swarm Intelligence (CSI)
Overview:
- Collaboration, communication, and collective intelligence technology for large networked groups
- Enables real-time conversations online among hundreds of individuals
- Amplifies collective intelligence through thoughtful deliberations
Features:
- Supports text-based conversations (with optional voice-to-text)
- Enables videoconferencing at a large scale in future platforms
- Employs Swarm Intelligence techniques modeled on fish schools for effective decision making
Problem Addressed:
- Ineffective deliberations in large groups due to conversational quality degradation and dominating personalities
- Lack of an efficient solution for productive conversations at scale
Solution:
- CSI technology learns from Mother Nature using Swarm Intelligence techniques
- Splits large human groups into networked subgroups, each with 4 to 7 participants for optimal real-time conversational deliberation
- AI-powered Surrogate Agents in each subgroup observe local deliberations, distill content, and pass critical points to other subgroups
- Fully connected network architecture among subgroups for efficient content propagation
- Scalable structure that can connect hundreds or thousands of participants
Benefits:
- Enables thoughtful real-time conversational deliberations on opinions, debates, brainstorming, challenges, prioritizing factors, and converging on solutions
- Reduces biasing influence of strong personalities and early comments
- Promotes greater dialog per person and more balanced deliberations
- Participants report preferring CSI platform over traditional centralized chat and feeling they have greater impact on the conversation.
Study Findings:
- Group of 245 participants challenged with estimating gumball quantities using Thinkscape CSI platform resulted in a 50% smaller error than traditional survey method [9]
- In another experiment, groups of 35 people answered IQ test questions:
- Participants outperformed individual and WoC aggregation performances [15]
- CSI groups scored an IQ of 128 (97th percentile) compared to average individual IQ score of 100
- No participant's personal IQ score reached the level of the CSI groups
- Study introduced a new intervention: Infobot agent in groupwise deliberations [15]
Infobots:
- Additional conversational AI agent designed to respond factually to queries and bring limited factual information to subgroups
- Primed with specific factual or statistical information regarding tasks/problems
- Participate in subgroup discussions alongside human participants
- Distributed discussion structure allows for efficient use of Infobots, enabling parallel exploration of different factual information [15]
- Figure 2 shows a diagram of CSI system with Infobots primed with MLB player/team info.
Study Overview:
- Conducted on Thinkscape CSI platform developed by Unanimous AI
- Goal: verify effectiveness of technology in groupwise deliberations on complex problems
- Fantasy Baseball contest used as collaborative challenge
Baseline Data Collection:
- Participants create personal rosters using standard survey
- Task requires selecting 5 players within fixed budget
- Six player options provided with salary information
- Making tradeoffs between positions required
- Sessions conducted twice per week for five consecutive weeks
- Bonus awarded to high performers to maximize performance
Study Design:
- Participants self-identified as baseball fans, familiar with fantasy sports challenges
- Groups of approximately 25 people engaged in each session
- Players chosen individually first for "personal roster"
- Then collaboratively using conversational deliberation
- Bonus awarded for strong performance in collaborative task
- Real-world DraftKings contest and data used
- Four positions pre-selected, remaining five to be selected within budget
- Subgroups of 5 people with a Surrogate Agent and Infobot (if applicable)
- Infobot provided expansive statistical data on current MLB players and teams
- Groups had 5 minutes and 30 seconds to collaboratively select each player
- No overbudget rosters allowed
- WoC roster defined using most popular player choices, replacing lowest plurality if budget exceeded
- Exit survey administered for subjective feedback on CSI experience and Infobot use.
Data Collection for MLB Daily Fantasy Contest:
- Data collected for 10 sessions, each requiring a different set of players for MLB games
- One session per week used standard CSI (Collaborative Strategy Intervention) and the other used Infobot augmentation
- Scoring based on official DraftKings methods based on official MLB results
- Scoring was for the five selected positions only
Performance Comparison:
- Collaborative rosters using CSI platform scored 62.4 points per session
- Outperformed median individual's score (47.3 points) and WoC method (43.7 points) on personal rosters
- Significantly higher performance for both CSI and Infobot methods compared to WoC in paired t-test (p=0.004)
Thinkscape (CSI):
- Averaged 62.4 points per session, exceeding 73% of individual rosters' scores
- Significantly more accurate than WoC method which outperformed 39% of individuals and median individual score (50%)
Infobot:
- Small improvement in performance but not statistically significant compared to no Infobot sessions
- Measurably more efficient conversations with fewer characters per minute (183 vs. 197) and less variance between participants
Example of CSI Deliberation:
- Group discussing final position selection for Second Base
- Considered Marcus Semien as the best player but most expensive
- Unique insight raised: Brendan Rogers on unusual hot streak
- Sentiment shifted towards Rogers, group picked him instead of Semien
- Resulted in higher score than expected for Rogers and underperformance of Semien.
Study Findings on CSI Deliberations:
Positive Feedback (aggregated across 10 sessions):
- Over 90% of respondents agreed that their perspectives were heard and considered in group discussions
- Over 80% disagreed with statements about rushing to conclusions or not using all relevant information
Infobot Usage:
- Participants queried Infobots regularly for factual information during all sessions with Infobots present
- Average of 4.1 queries per subgroup per player being selected
- Consistent usage across all questions (2.8 - 5.5 queries per subgroup)
- Over 70% of respondents agreed that decisions were stronger due to Infobot information
Subjective Feedback on CSI Deliberations:
- Over 93.9% agreed that members listened and considered their perspectives
- Over 97.3% felt they could share views openly without judgement or criticism
- Over 86.5% agreed decisions were stronger due to Infobot's contribution
Infobot Usage Statistics:
- Participants made an average of 4.1 queries per subgroup per player selected
- Queries ranged from 2.8 to 5.5 queries per group, indicating consistent usage.
Collaborative Forecasting Study with CSI Platform:
- Based on Daily Fantasy Baseball contest for open-ended, complex task within fixed budget
- Collectives outperformed individually surveyed participants (72% vs. p=0.016) and popular picks (p<0.001)
- Real-time deliberative conversation superior method for harnessing collective forecasting power of a 25-person group
- Tested Infobot AI assistant, "Infobots", in half of the CSI sessions:
- Each subgroup had access to their own Infobot (4.1 queries per player)
- No significant difference in scoring between sessions with and without Infobots
- Positive feedback from participants (85% agreed that decisions were stronger due to Infobot information, only 4% disagreed)
- Deliberations using Infobots showed less conversational content variance (p=0.039)
- Promoted balanced discussions with fewer members dominating dialog
- Participants reported feeling free to express opinions without fear of negative interpersonal repercussions.