Code and Data For the Paper

    Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game
    Prisha Samadarshi, Mariam Mustafa, Anushka Kulkarni, Raven Rothkopf, Tuhin Chakrabarty, Smaranda Muresan

Look at

    - automated_call/all_games.txt for all 442 games
    - results/ for all LLM responses as well as expert/novice responses
    - game_with_knowledge_taxonomy.json for games with categories labeled based on the knowledge taxonomy
    - automated_call/prompt_llm.txt for the actual prompt
    - scoring/ for all model and expert/novice responses scored 

If you use our data or annotations please cite us at

      @article{samadarshi2024connecting,
        title={Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game},
        author={Samadarshi, Prisha and Mustafa, Mariam and Kulkarni, Anushka and Rothkopf, Raven and Chakrabarty, Tuhin and Muresan, Smaranda},
        journal={arXiv preprint arXiv:2406.11012},
        year={2024}
      }