notebooks/demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "edcd342c",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This tutorial covers submission scoring and conversion scripts. \n",
    "\n",
    "Our starter kit includes the following files:\n",
    "* `parsers.py` contains all the code for parsing the XML-like intermediate representations and annotated examples into our `ProblemFormulation` dataclass. \n",
    "* `scoring.py` contains the code to score your submission. The functions here take in the canonical representations of the problem formulations.\n",
    "\n",
    "\n",
    "# Dataclasses\n",
    "\n",
    "The primary dataclass we use is `ProblemFormulation`. \n",
    "\n",
    "```\n",
    "class ProblemFormulation:\n",
    "    objective: ObjectiveDeclaration\n",
    "    constraints: list[ConstraintDeclaration]\n",
    "    entities: dict[str, int]\n",
    "```\n",
    "It has an objective, a list of constraints, and a dictionary of entities which maps each problem entity to an index. This index is used as its index in the canonical representation.\n",
    "\n",
    "`ProblemFormulation` serves as an intermediate representation for participants that can be reused by participants if desired. However, the canonical representation is what will be used for scoring and serves as the unique format in evaluating submissions. To convert from `ProblemFormulation` to `CanonicalFormulation`, we also include a `convert_to_canonical:ProblemFormulation -> CanonicalFormulation` function if you choose to use our dataclasses.\n",
    "\n",
    "\n",
    "![title](demo.drawio.svg)\n",
    "\n",
    "\n",
    "# Canonical Representation\n",
    "\n",
    "As a brief overview. The canonical representation in our case describes the objectives and constraints as a vector and matrix respectively.\n",
    "\n",
    "For the objective declaration, we may have an algebraic formulation of the form\n",
    "\n",
    "$$ \\max 0.007x_0 + 0.002x_1. $$\n",
    "\n",
    "This corresponds to a canonical objective vector as shown:\n",
    "\n",
    "$$ \\begin{bmatrix} 0.007 & 0.002\\end{bmatrix}^T. $$\n",
    "\n",
    "For the constraints, we may have a set of constraints of the form\n",
    "\n",
    "$$ \\begin{align*}\n",
    "x_0 + x_1 &\\le 60000\\\\\n",
    "x_0 &\\ge 0.015(x_0+x_1)\\\\\n",
    "x_1 & \\le 0.6(x_0+x_1)\n",
    "\\end{align*}.$$\n",
    "\n",
    "We aim to have this in the form $Ax \\le b$, and our canonical form would be the matrix $\\begin{bmatrix} A & b \\end{bmatrix}$. We start by moving all constant terms to the right side and all variables to the left. Then we flip inequalities to match:\n",
    "\n",
    "$$ \\begin{align*}\n",
    "x_0 + x_1 &\\le 60000\\\\\n",
    "-0.085 x_0 + 0.015 x_1 &\\le 0\\\\\n",
    "-0.6 x_0 + 0.4 x_1 &\\le 0\n",
    "\\end{align*}.$$\n",
    "\n",
    "Then our canonical form for the constraints would be:\n",
    "\n",
    "$$ \\begin{bmatrix}\n",
    "1 & 1 & 60000\\\\\n",
    "-0.085 & 0.015 & 0\\\\\n",
    "-0.6 & 0.4 & 0\n",
    "\\end{bmatrix}. $$\n",
    "\n",
    "\n",
    "## Order Mapping\n",
    "\n",
    "Note that the order of the columns matter! We will give an `order_mapping` dictionary in the ground truth to ensure consistency in the column indicies. For example, our order mapping may look something like:\n",
    "```\n",
    "{\n",
    "    'apples': 0,\n",
    "    'oranges': 1,\n",
    "    'grapes': 2\n",
    "}\n",
    "```\n",
    "This would mean apples would be in the first column of our canonical form matrices, oranges in the second, and grapes in the third. If you are using a self-written parser, ensure that your parser takes in this order mapping parameter and properly maps it to the correct column. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75faac15",
   "metadata": {},
   "source": [
    "# Parsing \n",
    "\n",
    "You may skip this if you intend to preprocess your model outputs yourself into the canonical output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "003a9c65",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"..\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "33677e5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import parsers\n",
    "import jsonlines\n",
    "import prettyprinter as pp\n",
    "\n",
    "pp.install_extras(exclude=['python', 'django'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8ebbac6",
   "metadata": {},
   "source": [
    "## XML Parsing\n",
    "\n",
    "Here, we will go through parsing the XML output that the baseline model produces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b9e7d130",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<DECLARATION>\n",
      "<OBJ_DIR> maximize </OBJ_DIR>\n",
      "<OBJ_NAME> revenue </OBJ_NAME> [is]\n",
      "<VAR> turnips </VAR> [TIMES] <PARAM> 300 </PARAM>\n",
      "<VAR> pumpkins </VAR> [TIMES] <PARAM> 450 </PARAM>\n",
      "</DECLARATION>\n",
      "<DECLARATION>\n",
      "<CONST_DIR> has </CONST_DIR><LIMIT> 500 </LIMIT>\n",
      "<CONST_TYPE> [SUM_CONSTRAINT] </CONST_TYPE>\n",
      "</DECLARATION>\n",
      "<DECLARATION>\n",
      "<CONST_DIR> available </CONST_DIR>\n",
      "<OPERATOR> LESS_OR_EQUAL </OPERATOR>\n",
      "<LIMIT> 40000</LIMIT>\n",
      "<CONST_TYPE> [LINEAR_CONSTRAINT] </CONST_TYPE> [is]\n",
      "<VAR> Turnips </VAR> [TIMES] <PARAM> 50 </PARAM>\n",
      "<VAR> Pumpkins </VAR> [TIMES] <PARAM> 90 </PARAM>\n",
      "</DECLARATION>\n",
      "<DECLARATION>\n",
      "<CONST_DIR> available </CONST_DIR>\n",
      "<OPERATOR> LESS_OR_EQUAL </OPERATOR>\n",
      "<LIMIT> 34000 </LIMIT>\n",
      "<CONST_TYPE> [LINEAR_CONSTRAINT] </CONST_TYPE> [is]\n",
      "<VAR> Turnips </VAR> [TIMES] <PARAM> 80 </PARAM>\n",
      "<VAR> Pumpkins </VAR> [TIMES] <PARAM> 40 </PARAM>\n",
      "</DECLARATION>\n"
     ]
    }
   ],
   "source": [
    "# read original file \n",
    "# this is an example of the intermediate output\n",
    "with open('demo_examples/modeloutput1.txt', 'r') as fd:\n",
    "    data = fd.read()\n",
    "    print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d83aeb45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "parsers.ProblemFormulation(\n",
      "    objective=parsers.ObjectiveDeclaration(\n",
      "        direction='maximize',\n",
      "        terms={\n",
      "            'turnips': parsers.Term(name='turnips', index=0, value=300.0),\n",
      "            'pumpkins': parsers.Term(name='pumpkins', index=1, value=450.0)\n",
      "        },\n",
      "        entities={'turnips': 0, 'pumpkins': 1},\n",
      "        name='revenue'\n",
      "    ),\n",
      "    constraints=[\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='has',\n",
      "            terms=collections.OrderedDict([]),\n",
      "            entities={'turnips': 0, 'pumpkins': 1},\n",
      "            type='[SUM_CONSTRAINT]',\n",
      "            limit=500.0,\n",
      "            operator=''\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='available',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('turnips', parsers.Term(name='turnips', index=0, value=50.0)),\n",
      "                (\n",
      "                    'pumpkins',\n",
      "                    parsers.Term(name='pumpkins', index=1, value=90.0)\n",
      "                )\n",
      "            ]),\n",
      "            entities={'turnips': 0, 'pumpkins': 1},\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=40000.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='available',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('turnips', parsers.Term(name='turnips', index=0, value=80.0)),\n",
      "                (\n",
      "                    'pumpkins',\n",
      "                    parsers.Term(name='pumpkins', index=1, value=40.0)\n",
      "                )\n",
      "            ]),\n",
      "            entities={'turnips': 0, 'pumpkins': 1},\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=34000.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        )\n",
      "    ],\n",
      "    entities={'turnips': 0, 'pumpkins': 1}\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# instantiate parser\n",
    "parser = parsers.ModelOutputXMLParser()\n",
    "\n",
    "# parse our file\n",
    "# parser.read_xml will read a string already in memory\n",
    "formulation = parser.parse_file('demo_examples/modeloutput1.txt')\n",
    "\n",
    "pp.pprint(formulation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3e05b26d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Objective:\n",
      " [300. 450.]\n",
      "Constraints:\n",
      " [[1.0e+00 1.0e+00 5.0e+02]\n",
      " [5.0e+01 9.0e+01 4.0e+04]\n",
      " [8.0e+01 4.0e+01 3.4e+04]]\n"
     ]
    }
   ],
   "source": [
    "# convert to canonical form\n",
    "canonical = parsers.convert_to_canonical(formulation)\n",
    "print(f'Objective:\\n {canonical.objective}')\n",
    "print(f'Constraints:\\n {canonical.constraints}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fabf141",
   "metadata": {},
   "source": [
    "We use `xml.etree.ElementTree` as our XML parser. If your model outputs are XML, **please post-process your outputs and ensure they are syntactically correct**. We make some best-effort attempts at parsing syntactically incorrect outputs using the BeautifulSoup package, but this does not guarantee that an syntactically incorrect output gets parsed correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4604a7b4",
   "metadata": {},
   "source": [
    "## JSON Formulation Parsing\n",
    "\n",
    "Here, we go through parsing the JSON formatted formulations in the ground truths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "836c373c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    '-640645082': {\n",
      "        'document':\n",
      "            'A coconut seller has to transport coconuts using either rickshaws or '\n",
      "            'ox carts. The rickshaws can take 50 coconuts each and cost $10 per '\n",
      "            'trip. The ox carts can take 30 coconuts each and cost $8 per trip. '\n",
      "            'The seller has at most $200 to spend on transporting the coconuts. '\n",
      "            'Due to pollution, the number of rickshaws must not exceed the number '\n",
      "            'of ox carts. Formulate a LP to maximize the number of coconuts that '\n",
      "            'can be transported.',\n",
      "        'vars': ['rickshaws', 'ox carts'],\n",
      "        'var_mentions': [\n",
      "            'rickshaws',\n",
      "            'ox carts',\n",
      "            'rickshaws',\n",
      "            'ox carts',\n",
      "            'rickshaws',\n",
      "            'ox carts'\n",
      "        ],\n",
      "        'params': ['50', '10', '30', '8'],\n",
      "        'var_mention_to_first_var': {\n",
      "            'rickshaws': 'rickshaws',\n",
      "            'ox carts': 'ox carts'\n",
      "        },\n",
      "        'first_var_to_mentions': {\n",
      "            'rickshaws': ['rickshaws', 'rickshaws', 'rickshaws'],\n",
      "            'ox carts': ['ox carts', 'ox carts', 'ox carts']\n",
      "        },\n",
      "        'obj_declaration': {\n",
      "            'type': 'objective',\n",
      "            'direction': 'maximize',\n",
      "            'name': 'number of coconuts',\n",
      "            'terms': {'rickshaws': '50', 'ox carts': '30'}\n",
      "        },\n",
      "        'const_declarations': [\n",
      "            {\n",
      "                'type': 'linear',\n",
      "                'direction': 'at most',\n",
      "                'limit': '200',\n",
      "                'terms': {'rickshaws': '10', 'ox carts': '8'},\n",
      "                'operator': 'LESS_OR_EQUAL'\n",
      "            },\n",
      "            {\n",
      "                'type': 'xy',\n",
      "                'x_var': 'rickshaws',\n",
      "                'direction': 'must not exceed',\n",
      "                'y_var': 'ox carts',\n",
      "                'operator': 'LESS_OR_EQUAL'\n",
      "            }\n",
      "        ],\n",
      "        'spans': [\n",
      "            {\n",
      "                'text': 'rickshaws',\n",
      "                'start': 56,\n",
      "                'token_start': 9,\n",
      "                'token_end': 9,\n",
      "                'end': 65,\n",
      "                'type': 'span',\n",
      "                'label': 'VAR'\n",
      "            },\n",
      "            {\n",
      "                'text': 'ox carts',\n",
      "                'start': 69,\n",
      "                'token_start': 11,\n",
      "                'token_end': 12,\n",
      "                'end': 77,\n",
      "                'type': 'span',\n",
      "                'label': 'VAR'\n",
      "            },\n",
      "            {\n",
      "                'text': 'rickshaws',\n",
      "                'start': 83,\n",
      "                'token_start': 15,\n",
      "                'token_end': 15,\n",
      "                'end': 92,\n",
      "                'type': 'span',\n",
      "                'label': 'VAR'\n",
      "            },\n",
      "            {\n",
      "                'text': '50',\n",
      "                'start': 102,\n",
      "                'token_start': 18,\n",
      "                'token_end': 18,\n",
      "                'end': 104,\n",
      "                'type': 'span',\n",
      "                'label': 'PARAM'\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 105,\n",
      "                'token_start': 19,\n",
      "                'token_end': 19,\n",
      "                'end': 113,\n",
      "                'type': 'span',\n",
      "                'label': 'OBJ_NAME'\n",
      "            },\n",
      "            {\n",
      "                'text': '10',\n",
      "                'start': 129,\n",
      "                'token_start': 24,\n",
      "                'token_end': 24,\n",
      "                'end': 131,\n",
      "                'type': 'span',\n",
      "                'label': 'PARAM'\n",
      "            },\n",
      "            {\n",
      "                'text': 'ox carts',\n",
      "                'start': 146,\n",
      "                'token_start': 29,\n",
      "                'token_end': 30,\n",
      "                'end': 154,\n",
      "                'type': 'span',\n",
      "                'label': 'VAR'\n",
      "            },\n",
      "            {\n",
      "                'text': '30',\n",
      "                'start': 164,\n",
      "                'token_start': 33,\n",
      "                'token_end': 33,\n",
      "                'end': 166,\n",
      "                'type': 'span',\n",
      "                'label': 'PARAM'\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 167,\n",
      "                'token_start': 34,\n",
      "                'token_end': 34,\n",
      "                'end': 175,\n",
      "                'type': 'span',\n",
      "                'label': 'OBJ_NAME'\n",
      "            },\n",
      "            {\n",
      "                'text': '8',\n",
      "                'start': 191,\n",
      "                'token_start': 39,\n",
      "                'token_end': 39,\n",
      "                'end': 192,\n",
      "                'type': 'span',\n",
      "                'label': 'PARAM'\n",
      "            },\n",
      "            {\n",
      "                'text': 'at most',\n",
      "                'start': 218,\n",
      "                'token_start': 46,\n",
      "                'token_end': 47,\n",
      "                'end': 225,\n",
      "                'type': 'span',\n",
      "                'label': 'CONST_DIR'\n",
      "            },\n",
      "            {\n",
      "                'text': '200',\n",
      "                'start': 227,\n",
      "                'token_start': 49,\n",
      "                'token_end': 49,\n",
      "                'end': 230,\n",
      "                'type': 'span',\n",
      "                'label': 'LIMIT'\n",
      "            },\n",
      "            {\n",
      "                'text': 'rickshaws',\n",
      "                'start': 302,\n",
      "                'token_start': 64,\n",
      "                'token_end': 64,\n",
      "                'end': 311,\n",
      "                'type': 'span',\n",
      "                'label': 'VAR'\n",
      "            },\n",
      "            {\n",
      "                'text': 'must not exceed',\n",
      "                'start': 312,\n",
      "                'token_start': 65,\n",
      "                'token_end': 67,\n",
      "                'end': 327,\n",
      "                'type': 'span',\n",
      "                'label': 'CONST_DIR'\n",
      "            },\n",
      "            {\n",
      "                'text': 'ox carts',\n",
      "                'start': 342,\n",
      "                'token_start': 71,\n",
      "                'token_end': 72,\n",
      "                'end': 350,\n",
      "                'type': 'span',\n",
      "                'label': 'VAR'\n",
      "            },\n",
      "            {\n",
      "                'text': 'maximize',\n",
      "                'start': 370,\n",
      "                'token_start': 78,\n",
      "                'token_end': 78,\n",
      "                'end': 378,\n",
      "                'type': 'span',\n",
      "                'label': 'OBJ_DIR'\n",
      "            },\n",
      "            {\n",
      "                'text': 'number of coconuts',\n",
      "                'start': 383,\n",
      "                'token_start': 80,\n",
      "                'token_end': 82,\n",
      "                'end': 401,\n",
      "                'type': 'span',\n",
      "                'label': 'OBJ_NAME'\n",
      "            }\n",
      "        ],\n",
      "        'tokens': [\n",
      "            {\n",
      "                'text': 'A',\n",
      "                'start': 0,\n",
      "                'end': 1,\n",
      "                'id': 0,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconut',\n",
      "                'start': 2,\n",
      "                'end': 9,\n",
      "                'id': 1,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'seller',\n",
      "                'start': 10,\n",
      "                'end': 16,\n",
      "                'id': 2,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'has',\n",
      "                'start': 17,\n",
      "                'end': 20,\n",
      "                'id': 3,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'to',\n",
      "                'start': 21,\n",
      "                'end': 23,\n",
      "                'id': 4,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'transport',\n",
      "                'start': 24,\n",
      "                'end': 33,\n",
      "                'id': 5,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 34,\n",
      "                'end': 42,\n",
      "                'id': 6,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'using',\n",
      "                'start': 43,\n",
      "                'end': 48,\n",
      "                'id': 7,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'either',\n",
      "                'start': 49,\n",
      "                'end': 55,\n",
      "                'id': 8,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'rickshaws',\n",
      "                'start': 56,\n",
      "                'end': 65,\n",
      "                'id': 9,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'or',\n",
      "                'start': 66,\n",
      "                'end': 68,\n",
      "                'id': 10,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'ox',\n",
      "                'start': 69,\n",
      "                'end': 71,\n",
      "                'id': 11,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'carts',\n",
      "                'start': 72,\n",
      "                'end': 77,\n",
      "                'id': 12,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '.',\n",
      "                'start': 77,\n",
      "                'end': 78,\n",
      "                'id': 13,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'The',\n",
      "                'start': 79,\n",
      "                'end': 82,\n",
      "                'id': 14,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'rickshaws',\n",
      "                'start': 83,\n",
      "                'end': 92,\n",
      "                'id': 15,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'can',\n",
      "                'start': 93,\n",
      "                'end': 96,\n",
      "                'id': 16,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'take',\n",
      "                'start': 97,\n",
      "                'end': 101,\n",
      "                'id': 17,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '50',\n",
      "                'start': 102,\n",
      "                'end': 104,\n",
      "                'id': 18,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 105,\n",
      "                'end': 113,\n",
      "                'id': 19,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'each',\n",
      "                'start': 114,\n",
      "                'end': 118,\n",
      "                'id': 20,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'and',\n",
      "                'start': 119,\n",
      "                'end': 122,\n",
      "                'id': 21,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'cost',\n",
      "                'start': 123,\n",
      "                'end': 127,\n",
      "                'id': 22,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '$',\n",
      "                'start': 128,\n",
      "                'end': 129,\n",
      "                'id': 23,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '10',\n",
      "                'start': 129,\n",
      "                'end': 131,\n",
      "                'id': 24,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'per',\n",
      "                'start': 132,\n",
      "                'end': 135,\n",
      "                'id': 25,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'trip',\n",
      "                'start': 136,\n",
      "                'end': 140,\n",
      "                'id': 26,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '.',\n",
      "                'start': 140,\n",
      "                'end': 141,\n",
      "                'id': 27,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'The',\n",
      "                'start': 142,\n",
      "                'end': 145,\n",
      "                'id': 28,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'ox',\n",
      "                'start': 146,\n",
      "                'end': 148,\n",
      "                'id': 29,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'carts',\n",
      "                'start': 149,\n",
      "                'end': 154,\n",
      "                'id': 30,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'can',\n",
      "                'start': 155,\n",
      "                'end': 158,\n",
      "                'id': 31,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'take',\n",
      "                'start': 159,\n",
      "                'end': 163,\n",
      "                'id': 32,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '30',\n",
      "                'start': 164,\n",
      "                'end': 166,\n",
      "                'id': 33,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 167,\n",
      "                'end': 175,\n",
      "                'id': 34,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'each',\n",
      "                'start': 176,\n",
      "                'end': 180,\n",
      "                'id': 35,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'and',\n",
      "                'start': 181,\n",
      "                'end': 184,\n",
      "                'id': 36,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'cost',\n",
      "                'start': 185,\n",
      "                'end': 189,\n",
      "                'id': 37,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '$',\n",
      "                'start': 190,\n",
      "                'end': 191,\n",
      "                'id': 38,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '8',\n",
      "                'start': 191,\n",
      "                'end': 192,\n",
      "                'id': 39,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'per',\n",
      "                'start': 193,\n",
      "                'end': 196,\n",
      "                'id': 40,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'trip',\n",
      "                'start': 197,\n",
      "                'end': 201,\n",
      "                'id': 41,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '.',\n",
      "                'start': 201,\n",
      "                'end': 202,\n",
      "                'id': 42,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'The',\n",
      "                'start': 203,\n",
      "                'end': 206,\n",
      "                'id': 43,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'seller',\n",
      "                'start': 207,\n",
      "                'end': 213,\n",
      "                'id': 44,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'has',\n",
      "                'start': 214,\n",
      "                'end': 217,\n",
      "                'id': 45,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'at',\n",
      "                'start': 218,\n",
      "                'end': 220,\n",
      "                'id': 46,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'most',\n",
      "                'start': 221,\n",
      "                'end': 225,\n",
      "                'id': 47,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '$',\n",
      "                'start': 226,\n",
      "                'end': 227,\n",
      "                'id': 48,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '200',\n",
      "                'start': 227,\n",
      "                'end': 230,\n",
      "                'id': 49,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'to',\n",
      "                'start': 231,\n",
      "                'end': 233,\n",
      "                'id': 50,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'spend',\n",
      "                'start': 234,\n",
      "                'end': 239,\n",
      "                'id': 51,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'on',\n",
      "                'start': 240,\n",
      "                'end': 242,\n",
      "                'id': 52,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'transporting',\n",
      "                'start': 243,\n",
      "                'end': 255,\n",
      "                'id': 53,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'the',\n",
      "                'start': 256,\n",
      "                'end': 259,\n",
      "                'id': 54,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 260,\n",
      "                'end': 268,\n",
      "                'id': 55,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '.',\n",
      "                'start': 268,\n",
      "                'end': 269,\n",
      "                'id': 56,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'Due',\n",
      "                'start': 270,\n",
      "                'end': 273,\n",
      "                'id': 57,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'to',\n",
      "                'start': 274,\n",
      "                'end': 276,\n",
      "                'id': 58,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'pollution',\n",
      "                'start': 277,\n",
      "                'end': 286,\n",
      "                'id': 59,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': ',',\n",
      "                'start': 286,\n",
      "                'end': 287,\n",
      "                'id': 60,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'the',\n",
      "                'start': 288,\n",
      "                'end': 291,\n",
      "                'id': 61,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'number',\n",
      "                'start': 292,\n",
      "                'end': 298,\n",
      "                'id': 62,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'of',\n",
      "                'start': 299,\n",
      "                'end': 301,\n",
      "                'id': 63,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'rickshaws',\n",
      "                'start': 302,\n",
      "                'end': 311,\n",
      "                'id': 64,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'must',\n",
      "                'start': 312,\n",
      "                'end': 316,\n",
      "                'id': 65,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'not',\n",
      "                'start': 317,\n",
      "                'end': 320,\n",
      "                'id': 66,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'exceed',\n",
      "                'start': 321,\n",
      "                'end': 327,\n",
      "                'id': 67,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'the',\n",
      "                'start': 328,\n",
      "                'end': 331,\n",
      "                'id': 68,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'number',\n",
      "                'start': 332,\n",
      "                'end': 338,\n",
      "                'id': 69,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'of',\n",
      "                'start': 339,\n",
      "                'end': 341,\n",
      "                'id': 70,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'ox',\n",
      "                'start': 342,\n",
      "                'end': 344,\n",
      "                'id': 71,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'carts',\n",
      "                'start': 345,\n",
      "                'end': 350,\n",
      "                'id': 72,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '.',\n",
      "                'start': 350,\n",
      "                'end': 351,\n",
      "                'id': 73,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'Formulate',\n",
      "                'start': 352,\n",
      "                'end': 361,\n",
      "                'id': 74,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'a',\n",
      "                'start': 362,\n",
      "                'end': 363,\n",
      "                'id': 75,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'LP',\n",
      "                'start': 364,\n",
      "                'end': 366,\n",
      "                'id': 76,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'to',\n",
      "                'start': 367,\n",
      "                'end': 369,\n",
      "                'id': 77,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'maximize',\n",
      "                'start': 370,\n",
      "                'end': 378,\n",
      "                'id': 78,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'the',\n",
      "                'start': 379,\n",
      "                'end': 382,\n",
      "                'id': 79,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'number',\n",
      "                'start': 383,\n",
      "                'end': 389,\n",
      "                'id': 80,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'of',\n",
      "                'start': 390,\n",
      "                'end': 392,\n",
      "                'id': 81,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'coconuts',\n",
      "                'start': 393,\n",
      "                'end': 401,\n",
      "                'id': 82,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'that',\n",
      "                'start': 402,\n",
      "                'end': 406,\n",
      "                'id': 83,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'can',\n",
      "                'start': 407,\n",
      "                'end': 410,\n",
      "                'id': 84,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'be',\n",
      "                'start': 411,\n",
      "                'end': 413,\n",
      "                'id': 85,\n",
      "                'ws': True,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': 'transported',\n",
      "                'start': 414,\n",
      "                'end': 425,\n",
      "                'id': 86,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            },\n",
      "            {\n",
      "                'text': '.',\n",
      "                'start': 425,\n",
      "                'end': 426,\n",
      "                'id': 87,\n",
      "                'ws': False,\n",
      "                'disabled': False\n",
      "            }\n",
      "        ],\n",
      "        '_input_hash': -640645082,\n",
      "        'order_mapping': {'rickshaws': 0, 'ox carts': 1}\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "parser = parsers.JSONFormulationParser()\n",
    "# read JSON file\n",
    "with jsonlines.open('demo_examples/annotated.jsonl') as reader:\n",
    "    unparsed = [line for line in reader.iter()]\n",
    "    pp.pprint(unparsed[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45f66e8a",
   "metadata": {},
   "source": [
    "Parsing this data, we get the `ProblemFormulation` representation again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1da96fef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "parsers.ProblemFormulation(\n",
      "    objective=parsers.ObjectiveDeclaration(\n",
      "        direction='maximize',\n",
      "        terms={\n",
      "            'rickshaws': parsers.Term(name='rickshaws', index=0, value=50.0),\n",
      "            'ox carts': parsers.Term(name='ox carts', index=1, value=30.0)\n",
      "        },\n",
      "        entities={'rickshaws': 0, 'ox carts': 1},\n",
      "        name='number of coconuts'\n",
      "    ),\n",
      "    constraints=[\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='at most',\n",
      "            terms=collections.OrderedDict([\n",
      "                (\n",
      "                    'rickshaws',\n",
      "                    parsers.Term(name='rickshaws', index=0, value=10.0)\n",
      "                ),\n",
      "                ('ox carts', parsers.Term(name='ox carts', index=1, value=8.0))\n",
      "            ]),\n",
      "            entities={'rickshaws': 0, 'ox carts': 1},\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=200.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='must not exceed',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('ox carts', parsers.Term(name='ox carts', index=1)),\n",
      "                ('rickshaws', parsers.Term(name='rickshaws', index=0))\n",
      "            ]),\n",
      "            entities={'rickshaws': 0, 'ox carts': 1},\n",
      "            type='[XY_CONSTRAINT]',\n",
      "            limit=0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        )\n",
      "    ],\n",
      "    entities={'rickshaws': 0, 'ox carts': 1}\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# parse JSON representation\n",
    "parsed = [parser.parse(line) for line in unparsed]\n",
    "pp.pprint(parsed[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9548dac",
   "metadata": {},
   "source": [
    "# Scoring"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "969fdb11",
   "metadata": {},
   "source": [
    "When scoring, your model outputs will be compared to the canonical form of the annotated datasest. We have provided in `demo_examples` some examples:\n",
    "* `modeloutput*.txt` are two XML intermediate outputs for two problem formulations.\n",
    "* `annotated.jsonl` contains two annotated JSON examples of the same problem formulations.\n",
    "\n",
    "Suppose that for this following example, your model outputs XML outputs. You must first parse them then convert them to canonical form. If your model outputs the canonical form, you may directly use the canonical form directly without parsing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d1c283c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "json_parser = parsers.JSONFormulationParser()\n",
    "xml_parser = parsers.ModelOutputXMLParser()\n",
    "# regular test pipeline of a few examples\n",
    "with jsonlines.open('demo_examples/annotated.jsonl') as reader:\n",
    "    true_formulations = []\n",
    "    predicted = []\n",
    "\n",
    "    for i, line in enumerate(reader.iter()):\n",
    "        true_formulation = json_parser.parse(line)\n",
    "        true_formulations.append(true_formulation)\n",
    "        # make sure you use the order mapping from the ground truth (true_formulation.entities)\n",
    "        predicted.append(xml_parser.parse_file(f'demo_examples/modeloutput{i}.txt', true_formulation.entities))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2fc69ff3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "parsers.ProblemFormulation(\n",
      "    objective=parsers.ObjectiveDeclaration(\n",
      "        direction='maximize',\n",
      "        terms={\n",
      "            'turnips': parsers.Term(name='turnips', index=0, value=300.0),\n",
      "            'pumpkins': parsers.Term(name='pumpkins', index=1, value=450.0)\n",
      "        },\n",
      "        entities={\n",
      "            'turnips': 0,\n",
      "            'pumpkins': 1,\n",
      "            'Turnips': 0,\n",
      "            'Pumpkins': 1\n",
      "        },\n",
      "        name='turnips'\n",
      "    ),\n",
      "    constraints=[\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='has',\n",
      "            terms=collections.OrderedDict([]),\n",
      "            entities={\n",
      "                'turnips': 0,\n",
      "                'pumpkins': 1,\n",
      "                'Turnips': 0,\n",
      "                'Pumpkins': 1\n",
      "            },\n",
      "            type='[SUM_CONSTRAINT]',\n",
      "            limit=500.0,\n",
      "            operator=''\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='available',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('Turnips', parsers.Term(name='Turnips', index=0, value=50.0)),\n",
      "                (\n",
      "                    'Pumpkins',\n",
      "                    parsers.Term(name='Pumpkins', index=1, value=90.0)\n",
      "                )\n",
      "            ]),\n",
      "            entities={\n",
      "                'turnips': 0,\n",
      "                'pumpkins': 1,\n",
      "                'Turnips': 0,\n",
      "                'Pumpkins': 1\n",
      "            },\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=40000.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='available',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('Turnips', parsers.Term(name='Turnips', index=0, value=80.0)),\n",
      "                (\n",
      "                    'Pumpkins',\n",
      "                    parsers.Term(name='Pumpkins', index=1, value=40.0)\n",
      "                )\n",
      "            ]),\n",
      "            entities={\n",
      "                'turnips': 0,\n",
      "                'pumpkins': 1,\n",
      "                'Turnips': 0,\n",
      "                'Pumpkins': 1\n",
      "            },\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=34000.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        )\n",
      "    ],\n",
      "    entities={\n",
      "        'turnips': 0,\n",
      "        'pumpkins': 1,\n",
      "        'Turnips': 0,\n",
      "        'Pumpkins': 1\n",
      "    }\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "pp.pprint(predicted[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8f257483",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "parsers.ProblemFormulation(\n",
      "    objective=parsers.ObjectiveDeclaration(\n",
      "        direction='maximize',\n",
      "        terms={\n",
      "            'turnips': parsers.Term(name='turnips', index=0, value=300.0),\n",
      "            'pumpkins': parsers.Term(name='pumpkins', index=1, value=450.0)\n",
      "        },\n",
      "        entities={\n",
      "            'turnips': 0,\n",
      "            'pumpkins': 1,\n",
      "            'Turnips': 0,\n",
      "            'Pumpkins': 1\n",
      "        },\n",
      "        name='revenue'\n",
      "    ),\n",
      "    constraints=[\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='has',\n",
      "            terms=collections.OrderedDict([]),\n",
      "            entities={\n",
      "                'turnips': 0,\n",
      "                'pumpkins': 1,\n",
      "                'Turnips': 0,\n",
      "                'Pumpkins': 1\n",
      "            },\n",
      "            type='[SUM_CONSTRAINT]',\n",
      "            limit=500.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='available',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('Turnips', parsers.Term(name='Turnips', index=0, value=50.0)),\n",
      "                (\n",
      "                    'Pumpkins',\n",
      "                    parsers.Term(name='Pumpkins', index=1, value=90.0)\n",
      "                )\n",
      "            ]),\n",
      "            entities={\n",
      "                'turnips': 0,\n",
      "                'pumpkins': 1,\n",
      "                'Turnips': 0,\n",
      "                'Pumpkins': 1\n",
      "            },\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=40000.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        ),\n",
      "        parsers.ConstraintDeclaration(\n",
      "            direction='available',\n",
      "            terms=collections.OrderedDict([\n",
      "                ('Turnips', parsers.Term(name='Turnips', index=0, value=80.0)),\n",
      "                (\n",
      "                    'Pumpkins',\n",
      "                    parsers.Term(name='Pumpkins', index=1, value=50.0)\n",
      "                )\n",
      "            ]),\n",
      "            entities={\n",
      "                'turnips': 0,\n",
      "                'pumpkins': 1,\n",
      "                'Turnips': 0,\n",
      "                'Pumpkins': 1\n",
      "            },\n",
      "            type='[LINEAR_CONSTRAINT]',\n",
      "            limit=34000.0,\n",
      "            operator='LESS_OR_EQUAL'\n",
      "        )\n",
      "    ],\n",
      "    entities={\n",
      "        'turnips': 0,\n",
      "        'pumpkins': 1,\n",
      "        'Turnips': 0,\n",
      "        'Pumpkins': 1\n",
      "    }\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "pp.pprint(true_formulations[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b96e26bf",
   "metadata": {},
   "source": [
    "Notice that in the third constraint, `pumpkins` is different in the predicted (40) compared to the ground truth (50). This will count as a false positive. To illustrate as the canonical form constraints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "342729f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.0e+00 1.0e+00 1.0e+00 1.0e+00 5.0e+02]\n",
      " [5.0e+01 9.0e+01 1.0e+00 1.0e+00 4.0e+04]\n",
      " [8.0e+01 4.0e+01 1.0e+00 1.0e+00 3.4e+04]]\n",
      "\n",
      "\n",
      "[[1.0e+00 1.0e+00 1.0e+00 1.0e+00 5.0e+02]\n",
      " [5.0e+01 9.0e+01 1.0e+00 1.0e+00 4.0e+04]\n",
      " [8.0e+01 5.0e+01 1.0e+00 1.0e+00 3.4e+04]]\n"
     ]
    }
   ],
   "source": [
    "p1 = parsers.convert_to_canonical(predicted[1])\n",
    "t1 = parsers.convert_to_canonical(true_formulations[1])\n",
    "\n",
    "print(p1.constraints)\n",
    "print('\\n')\n",
    "print(t1.constraints)\n",
    "# notice that the last row is different"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec23abdf",
   "metadata": {},
   "source": [
    "We compute the false positives (FP), false negatives (FN), and number of ground-truth declarations (D) for each example.\n",
    "* FP is the number of non-matched predicted declarations. For example, an extra predicted constraint or an incorrect value in a constraint.\n",
    "* FN denotes the number of ground-truth declarations without a match. For example, too few predicted constraints.\n",
    "* D is the number of ground-truth declarations. This is equal to the number of constraints + the number of objectives (1) in the ground truth.\n",
    "\n",
    "To get FP, FN, D on one example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "56dd169c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FP: 1\n",
      "FN: 0\n",
      "D: 4\n"
     ]
    }
   ],
   "source": [
    "import scoring\n",
    "\n",
    "fp, fn, d = scoring.per_example_scores(p1.objective, p1.constraints, t1.objective, t1.constraints)\n",
    "print(f'FP: {fp}')\n",
    "print(f'FN: {fn}')\n",
    "print(f'D: {d}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4717a71b",
   "metadata": {},
   "source": [
    "Then to get the overall score on all examples, we pass in the lists of objectives and constraints. These all should be given as `list[np.ndarray]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4542d126",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8571428571428572\n"
     ]
    }
   ],
   "source": [
    "pred_canonical = [parsers.convert_to_canonical(f) for f in predicted]\n",
    "true_canonical = [parsers.convert_to_canonical(f) for f in true_formulations]\n",
    "\n",
    "# get objectives and constraints\n",
    "pred_obj = [f.objective for f in pred_canonical]\n",
    "pred_const = [f.constraints for f in pred_canonical]\n",
    "true_obj = [f.objective for f in true_canonical]\n",
    "true_const = [f.constraints for f in true_canonical]\n",
    "\n",
    "score = scoring.overall_score(pred_obj, pred_const, true_obj, true_const)\n",
    "\n",
    "print(score)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.11 ('tgen')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "vscode": {
   "interpreter": {
    "hash": "02038ad108cabd6bee0df1ad6918c19b80593a8090c3244598ba78841af0e85f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}