sippycup-unit-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/sippycup-small.jpg\" align=\"left\" style=\"padding-right: 30px\"/>\n",
    "\n",
    "<h1 style=\"line-height: 125%\">\n",
    "  SippyCup<br />\n",
    "  Unit 2: Travel queries\n",
    "</h1>\n",
    "\n",
    "<p>\n",
    "  <a href=\"http://nlp.stanford.edu/~wcmac/\">Bill MacCartney</a><br/>\n",
    "  Spring 2015\n",
    "  <!-- <a href=\"mailto:wcmac@cs.stanford.edu\">wcmac@cs.stanford.edu</a> -->\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"margin: 0px 0px; padding: 10px; background-color: #ddddff; border-style: solid; border-color: #aaaacc; border-width: 1px\">\n",
    "This is Unit 2 of the <a href=\"./sippycup-unit-0.ipynb\">SippyCup codelab</a>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our next case study, we're going to look at travel queries, in which a user seeks information about getting from here to there.  Examples include:\n",
    "    \n",
    "    \"birmingham al distance from indianapolish in\"\n",
    "    \"directions from washington to canada\"\n",
    "    \"discount travel flights to austin texas\"\n",
    "\n",
    "It's easy to imagine applications for semantic parsing in the travel domain.  Providing answers to queries like these is a core part of the value proposition of intelligent assistants such as [Apple's Siri][], [Google Now][], [Microsoft's Cortana][], and [Amazon Echo][].\n",
    "\n",
    "[Apple's Siri]: https://www.apple.com/ios/siri/\n",
    "[Google Now]: https://www.google.com/landing/now/\n",
    "[Microsoft's Cortana]: http://www.microsoft.com/en-us/mobile/campaign-cortana/\n",
    "[Amazon Echo]: http://www.amazon.com/oc/echo/\n",
    "\n",
    "The travel domain has some important differences to the domain of natural language arithmetic we looked at in our [first case study](./sippycup-unit-1.ipynb).\n",
    "\n",
    "- **Above all, it's far more diverse.**  The lexicon of travel queries can include thousands of words, not merely dozens — indeed, its potential size is unbounded.  And the words and phrases can be assembled in a dizzying variety of syntactic structures.\n",
    "- **It's much more ambiguous.** Consider `\"directions from washington to canada\"`.  Does that refer to the State of Washington, or to the District of Columbia?  And where exactly in Canada are we headed?\n",
    "- **It can also be much messier.**  Because it's a real-world application, and because users ~~are idiots~~ make mistakes, travel queries can include lots of misspellings (such as `\"indianapolish\"`), mangled syntax, and other linguistic infelicities.  It's not all neat and tidy like the arithmetic case study.\n",
    "- **However, the semantics are flatter.**  The semantic representation for a travel query may need to represent the destination, the origin, the transportation mode, and perhaps a few other bits of information.  But, unlike the expression trees we looked at in the arithmetic case study, it does not need to contain arbitrarily deeply-nested recursive structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whenever we start work in a new domain, we must begin with data collection.  To drive an effective development process, we need a large, realistic, representative sample of the inputs our system is likely to encounter in its intended application.  In the arithmetic case study, we copped out on the data collection, in the name of pedagogical convenience.  This time, we'll try harder.\n",
    "\n",
    "For this purpose, we'll leverage the AOL search query dataset ([Pass et al. 2006][]), a resource which has proven immensely useful over time, despite its [controversial origin][].  This dataset contains approximately 10M unique search queries issued by about 650K AOL users in 2006. Because the dataset was quickly retracted by AOL, it can be difficult to find online, but it is currently available for download [here][aol-dataset].\n",
    "\n",
    "[Pass et al. 2006]: http://people.cs.georgetown.edu/~abdur/publications/pos-infoscale.pdf\n",
    "[controversial origin]: http://en.wikipedia.org/wiki/AOL_search_data_leak\n",
    "[aol-dataset]: http://www.cim.mcgill.ca/~dudek/206/Logs/AOL-user-ct-collection/\n",
    "\n",
    "Of course, the majority of those 10M queries are not related to travel.  (The five most frequent queries are \"google\", \"ebay\", \"yahoo\", \"yahoo.com\", and \"mapquest\".  Remember, it was 2006.  The query \"facebook\" doesn't show up until position 119!)  To construct a query sample suitable for our purposes, we'll need to do some filtering.\n",
    "\n",
    "Since the queries we're interested in always refer to a location, we'll start by filtering for location names.  Recognizing location names is itself a hard problem, so for current purposes we'll adopt a good-enough solution: we'll grep the queries for the names of about 600 U.S. locations (cities, states, mountains, lakes, and so on) listed in [Geobase][].  (We'll be using Geobase again in the next unit.)  This winnows the 10M unique queries to about 1M.  It's a good start, but it's still not enough.  (The top five queries are now \"white pages\", \"florida lottery\", \"wells fargo\", \"whitney houston\", and \"las vegas\".)  We need to find queries which not only contain a location, but also a clear indication of travel intent.\n",
    "\n",
    "[Geobase]: ftp://ftp.cs.utexas.edu/pub/mooney/nl-ilp-data/geosystem/geobase\n",
    "\n",
    "Since travel is about going *from* one location *to* another, let's grep for queries which contain both \"from\" and \"to\".  This cuts the number of queries to about 3,100, and we're now seeing the kind of queries we're looking for:\n",
    "\n",
    "    \"airline flights from raleigh durham to nashville tennessee\"\n",
    "    \"distance from seattle to vancouver\"\n",
    "    \"cheap flights to italy from new york\"\n",
    "    \"international flights from usa to morocco\"\n",
    "    \"milage from portlan oregon to canada\"\n",
    "    \"distance from alabama to arkansas in hours\"\n",
    "    \"driving directions to sault ste marie canada from vermont\"\n",
    "    \"great white sharks life cycle from brith to death\"\n",
    "    \"directions from newark airport to bayonne bridge\"\n",
    "    \"poem it takes courage to love from tyler perrys movie\"\n",
    "\n",
    "A couple of observations here.  First, not all of the queries are actually travel queries.  That's OK — we actually *want* to include some queries which do not belong to the target domain.  Such \"negative\" queries are important for evaluating our system's ability to distinguish between queries it should be parsing and queries it shouldn't.  Second, there are a lot of mispelings.  Again, that's OK.  We want a *realistic* sample which reflects the inputs our system will actually encounter in its intended application.  Messy inputs are a fact of life, and they're one of the things that distinguishes this domain from the highly artificial example of natural language arithmetic we looked at in the last unit.\n",
    "\n",
    "There's still a problem, though.  Not all travel queries contain both \"from\" and \"to\".  Many valid travel queries contain only a destination, but no origin — and some contain an origin, but no destination.  If we relax the conjunction to a disjunction (that is, \"from\" *or* \"to\", we get about 23,000 queries, most of which are still not what we're looking for (for example, \"colorado rent to own\", \"moving to florida\").  However, the list of examples above (and its continuation, not shown) suggest many other terms we can usefully search for: \"airline\", \"flights\", \"distance\", \"mileage\", \"driving\", \"directions\", and so on.  We constructed a list of about 60 such terms (recorded in a comment in [`travel_examples.py`](travel_examples.py)).  We then selected queries which contain a location, either \"from\" or \"to\", and either one of the ~60 travel terms or both \"from\" and \"to\".  This left 6,588 queries, and the results look pretty good:\n",
    "\n",
    "    \"flight one way las vegas nv to rapid city sd\"\n",
    "    \"number to alaska airlines\"\n",
    "    \"travel time to denver colorado\"\n",
    "    \"amtrak schedule in california from burbanlk airport\"\n",
    "    \"airfare las vegas to my\"\n",
    "    \"cycling route san francisco to sacramento\"\n",
    "    \"where to find cheapest gasoline in new castle delaware\"\n",
    "    \"discount tickets to six flags and discounted hotels in new jersey\"\n",
    "    \"public transportation washington dc to new york city\"\n",
    "    \"transportation from newark airport to manhattan\"\n",
    "\n",
    "Three of those are not really travel queries, but that's perfectly fine.\n",
    "\n",
    "Let's recap.  To construct our query sample, we:\n",
    "\n",
    "- Began from the set of unique queries in the AOL dataset (10M queries).\n",
    "- Selected queries containing one of the 600 locations named in Geobase (1M queries).\n",
    "- Selected queries containing \"from\" or \"to\" (23K queries).\n",
    "- Selected queries containing one of about 60 travel terms, or containing both \"from\" and \"to\" (6,588 queries).\n",
    "\n",
    "We've got our inputs; now we need to think about the outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic representation <a id=\"travel-semantic-representation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- TODO: In this section, present a bit more justification for the design. For example, the choice of nested maps.  But maybe also, the specific keys and values, motivated by what appears in the 6588 example inputs.  -->\n",
    "\n",
    "For the travel domain, we're going to use a style of semantic representation based on nested maps of key-value pairs (represented by Python dictionaries).  The keys belong to a small, fixed set of strings.  The values can be numbers, strings, or nested maps, depending upon the key.\n",
    "\n",
    "Here are some examples:\n",
    "\n",
    "    {'domain': 'travel', 'type': 'directions', 'mode': 'car',\n",
    "       'destination': {'id': 4793846, 'name': 'Williamsburg, VA, US'}}\n",
    "    {'domain': 'travel', 'type': 'duration', 'mode': 'bus',\n",
    "       'origin': {'id': 4500546, 'name': 'Atlantic City, NJ, US'},\n",
    "       'destination': {'id': 5128581, 'name': 'New York City, NY, US'}}\n",
    "    {'domain': 'travel', 'type': 'cost', 'mode': 'air',\n",
    "       'origin': {'id': 5101798, 'name': 'Newark, NJ, US'},\n",
    "       'destination': {'id': 4574324, 'name': 'Charleston, SC, US'}}\n",
    "\n",
    "You can probably discern what each example means without any difficulty.  The first is asking for driving directions to Williamsburg.  The second is asking for the duration of a bus trip from Atlantic City to New York City.  And the third is asking for the cost of an airfare from Newark to Charleston.\n",
    "\n",
    "To be a bit more formal, the following keys are allowed in the top-level map:\n",
    "\n",
    "  - `domain` has one of two values: `travel` for travel queries, or `other` for non-travel queries.\n",
    "  - `type` specifies the type of information requested, one of: `directions`, `distance`, `duration`, `cost`, or `schedule`.\n",
    "  - `mode` specifies the mode of travel, one of: `air`, `bike`, `boat`, `bus`, `car`, `taxi`, `train`, or `transit`.\n",
    "  - `destination` specifies the destination of travel using a nested map representing a location.\n",
    "  - `origin` specifies the origin of travel using a nested map representing a location.\n",
    "  \n",
    "In nested maps representating locations, just two keys are allowed:\n",
    "\n",
    "  - `id` is an integer which is the unique identifier of the location in the [GeoNames][] geographical database.\n",
    "  - `names` is a human-readable name for the location whose format is defined by code in [`geonames.py`](./geonames.py).\n",
    "\n",
    "  [GeoNames]: http://www.geonames.org/\n",
    "\n",
    "Note that our semantic representation both resolves ambiguity and canonicalizes.  One the one hand, the phrase \"`new york`\" has a different semantic representation if it refers to the city than if it refers to the state; on the other hand, \"`new york city`\" and \"`nyc`\" have the same semantic representation.  These qualities will make it easy for a downstream executor to turn a semantic representation directly into action.\n",
    "\n",
    "In a real-world application, the executor would likely be a module that translates our semantic representations into requests to backend APIs to compute optimal routes, retrieve bus schedules or airfares, or whatever.  However, because such functionality is not easily captured in a few lines of Python code, SippyCup doesn't include an executor for the travel domain.\n",
    "\n",
    "Since we don't have an executor, we won't be dealing with denotations in the travel domain.  (Abstractly, the denotation of a directions request is some kind of complex computational object representing a route.)  Instead, training and evaluation in this domain will consider only whether we're generating the correct semantics.  We'll return to considering denotations in our [third case study][].\n",
    "\n",
    "  [third case study]: ./sippycup-unit-3.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've described a process for extracting 6,588 (likely) travel queries from the AOL search query dataset, and we've defined a simple semantic representation for such queries.  The next step should be to manually annotate each query with its semantics, in order generate example data for training and evaluation.\n",
    "\n",
    "However, there's a problem.  Manual annotation is tedious and time-consuming, and therefore expensive.  For datasets of significant size, it is usually not practical for one person to do all the work.  This suggests distributing the work over a cluster of graduate students — which is in fact the method that was used to generate the [Geo880][] dataset that we'll use in our [third case study][].  Accordingly, one of the [exercises](#travel-exercises) at the end of this section will ask you to annotate a score of travel queries.\n",
    "\n",
    "Alternatively, if the annotation task does not require specialized knowledge, the work can often be distributed over workers on a crowdsourcing platform such as [Amazon Mechanical Turk][].  Our semantic representation for the travel domain is simple enough that the crowdsourcing approach might be feasible.  Of course, the workers wouldn't write down the semantics directly; rather, they'd fill out a carefully-designed form containing questions about the query, and a script would generate semantic representations based on their responses.  We leave working out the details of the crowdsourcing approach as an exercise for the motivated reader.  :-)\n",
    "\n",
    "  [Geo880]: http://www.cs.utexas.edu/users/ml/geo.html\n",
    "  [third case study]: ./sippycup-unit-1.ipynb\n",
    "  [Amazon Mechanical Turk]: https://www.mturk.com/mturk/welcome\n",
    "\n",
    "The difficulty of producing annotated data, and the consequent paucity of such data, has been the eternal bane of research in semantic parsing.  Indeed, overcoming this obstacle was the chief motivation for the shift made by [Liang et al. 2011][] to learn not from semantics but from denotations, which are easier and cheaper to generate via crowdsourcing.  Despite this shift, annotated data remains scarce.  To this day, the largest standard dataset for semantic parsing (the [WebQuestions][] dataset) contains only 5,810 examples.\n",
    "\n",
    "  [WebQuestions]: http://www-nlp.stanford.edu/software/sempre/\n",
    "  [Liang et al. 2011]: http://www.cs.berkeley.edu/~jordan/papers/liang-jordan-klein-acl2011.pdf\n",
    "\n",
    "For the purposes of this codelab, we have manually annotated 100 examples, given in [`travel_examples.py`](./travel_examples.py).  The examples were sampled uniformly at random from the set of 6,588 AOL queries described earlier, and are divided into a training set of 75 examples and a test set of 25 examples.  You might find yourself quibbling with some of the annotation choices.  In some cases, our semantic representation is too impoverished to capture the user's intent completely.  For example, \"`scenic drive to duluth mn`\", \"`honeymoon trip to hawaii`\", and \"`jet blue flights to new york`\" contain modifiers which cannot be expressed in our semantic representation.  In other cases, the user's intent may be uncertain or obscure: what exactly does \"`airbus from boston to europe`\" mean?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enriching the Grammar class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have some example data for the travel domain, we'd like to start building a grammar.  However, the grammar formalism we introduced in the arithmetic case study was quite restrictive.  Every rule had to be in Chomsky normal form: either a unary lexical rule, or a binary compositional rule.  If you did the [exercises](#arithmetic-exercises) at the end of the arithmetic case study, you will have chafed against those restrictions.  So, to make grammar engineering far more convenient, we're going to loosen things up.  We'll modify the `Grammar` class by adding:\n",
    "\n",
    "- **Annotators**, which are external modules for assigning categories and semantics to phrases of specific types.\n",
    "- **N-ary lexical rules**, such as `Rule('$City', 'new york city')`.\n",
    "- **Unary compositional rules**, such as `Rule('$Location', '$City')`.\n",
    "- **N-ary compositional rules**, such as `Rule('$RouteQuery', '$FromLocation $ToLocation $TravelMode')`.\n",
    "- Rules with **optional elements** on the RHS, such as `Rule('$BikeMode', '?by bike')`, which can match either `\"bike\"` or `\"by bike\"`.\n",
    "- A **designated start symbol**, typically `$ROOT`, to allow us to limit which categories can yield complete parses.\n",
    "\n",
    "In order to make all this stuff work, we'll need to redefine our `Grammar` class and some of its affiliated functions.  First, we'll redefine the `Grammar` constructor to add member variables for storing unary rules, annotators, and the start symbol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Grammar:\n",
    "    def __init__(self, rules=[], annotators=[], start_symbol='$ROOT'):\n",
    "        self.categories = set()\n",
    "        self.lexical_rules = defaultdict(list)\n",
    "        self.unary_rules = defaultdict(list)\n",
    "        self.binary_rules = defaultdict(list)\n",
    "        self.annotators = annotators\n",
    "        self.start_symbol = start_symbol\n",
    "        for rule in rules:\n",
    "            add_rule(self, rule)\n",
    "        print('Created grammar with %d rules.' % len(rules))\n",
    "\n",
    "    def parse_input(self, input):\n",
    "        \"\"\"Returns a list of parses for the given input.\"\"\"\n",
    "        return parse_input(self, input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll redefine `add_rule()`, so that it knows what to do when it encounters unary or n-ary compositional rules, or rule containing optionals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def add_rule(grammar, rule):\n",
    "    if contains_optionals(rule):\n",
    "        add_rule_containing_optional(grammar, rule)\n",
    "    elif is_lexical(rule):\n",
    "        grammar.lexical_rules[rule.rhs].append(rule)\n",
    "    elif is_unary(rule):\n",
    "        grammar.unary_rules[rule.rhs].append(rule)\n",
    "    elif is_binary(rule):\n",
    "        grammar.binary_rules[rule.rhs].append(rule)\n",
    "    elif all([is_cat(rhsi) for rhsi in rule.rhs]):\n",
    "        add_n_ary_rule(grammar, rule)\n",
    "    else:\n",
    "        # One of the exercises will ask you to handle this case.\n",
    "        raise Exception('RHS mixes terminals and non-terminals: %s' % rule)\n",
    "        \n",
    "def is_unary(rule):\n",
    "    \"\"\"\n",
    "    Returns true iff the given Rule is a unary compositional rule, i.e.,\n",
    "    contains only a single category (non-terminal) on the RHS.\n",
    "    \"\"\"\n",
    "    return len(rule.rhs) == 1 and is_cat(rule.rhs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll redefine `parse_input()`, so that it properly invokes functions for applying annotators and unary rules (in addition to existing invocations of functions for applying lexical rules and binary rules)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def parse_input(grammar, input):\n",
    "    \"\"\"Returns a list of all parses for input using grammar.\"\"\"\n",
    "    tokens = input.split()\n",
    "    chart = defaultdict(list)\n",
    "    for j in range(1, len(tokens) + 1):\n",
    "        for i in range(j - 1, -1, -1):\n",
    "            apply_annotators(grammar, chart, tokens, i, j)               # new!\n",
    "            apply_lexical_rules(grammar, chart, tokens, i, j)\n",
    "            apply_binary_rules(grammar, chart, i, j)\n",
    "            apply_unary_rules(grammar, chart, i, j)                      # new!\n",
    "    parses = chart[(0, len(tokens))]\n",
    "    if hasattr(grammar, 'start_symbol') and grammar.start_symbol:        # new!\n",
    "        parses = [parse for parse in parses if parse.rule.lhs == grammar.start_symbol]\n",
    "    return parses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if `grammar.start_symbol` is defined (the usual case), then we will return only parses having that category at the root."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our travel grammar will need to recognize names of cities (and other kinds of locations).  We could achieve this by writing lots of lexical rules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from parsing import Rule\n",
    "\n",
    "rules_cities = [\n",
    "    Rule('$City', 'austin', {'id': 4671654, 'name': 'Austin, TX, US'}),\n",
    "    Rule('$City', 'boston', {'id': 4930956, 'name': 'Boston, MA, US'}),\n",
    "    Rule('$City', 'chicago', {'id': 4887398, 'name': 'Chicago, IL, US'}),\n",
    "    Rule('$City', 'dallas', {'id': 4684888, 'name': 'Dallas, TX, US'}),\n",
    "    # ...\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there are thousands of cities in the world.  Adding thousands of rules to our grammar would be cumbersome and difficult to maintain.  Besides, there is probably already some library or service which knows how to recognize names of cities and map them into unique identifiers.  If we could wrap this external module in the right kind of interface, we could make it directly available to our grammar.\n",
    "\n",
    "We call such a module an *annotator*.  What kind of an interface should it have?  Well, it should take a token span (that is, a list of tokens) and return a list of zero or more interpretations, each a pair consisting of a category and a semantic representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Annotator:\n",
    "    \"\"\"A base class for annotators.\"\"\"\n",
    "    def annotate(self, tokens):\n",
    "        \"\"\"Returns a list of pairs, each a category and a semantic representation.\"\"\"\n",
    "        return []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a moment to implement a couple of simple annotators which will prove useful later on.  The first is a `NumberAnnotator`, which will annotate any string representing a number with category `$Number` and semantics equal to its numeric value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class NumberAnnotator(Annotator):\n",
    "    def annotate(self, tokens):\n",
    "        if len(tokens) == 1:\n",
    "            try:\n",
    "                value = float(tokens[0])\n",
    "                if int(value) == value:\n",
    "                    value = int(value)\n",
    "                return [('$Number', value)]\n",
    "            except ValueError:\n",
    "                pass\n",
    "        return []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the `NumberAnnotator` in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('$Number', 16)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NumberAnnotator().annotate(['16'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another annotator that will prove very useful is the `TokenAnnotator`, which will annotate any single token with category `$Token` and semantics equal to the token itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class TokenAnnotator(Annotator):\n",
    "    def annotate(self, tokens):\n",
    "        if len(tokens) == 1:\n",
    "            return [('$Token', tokens[0])]\n",
    "        else:\n",
    "            return []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the `TokenAnnotator` in action:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('$Token', 'foo')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TokenAnnotator().annotate(['foo'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later in this unit, we'll introduce the `GeoNamesAnnotator`, a more complex annotator designed to annotate names of locations.\n",
    "\n",
    "We saw above that the redesigned `Grammar` class holds a list of annotators, and that the redesigned `parse_input()` function invokes a function called `apply_annotators()` on every token span.  Its operation is straightforward: for every annotator in the grammar, and for every interpretation of the given span by the annotator, it creates a new parse and adds it to the corresponding chart cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def apply_annotators(grammar, chart, tokens, i, j):\n",
    "    \"\"\"Add parses to chart cell (i, j) by applying annotators.\"\"\"\n",
    "    if hasattr(grammar, 'annotators'):\n",
    "        for annotator in grammar.annotators:\n",
    "            for category, semantics in annotator.annotate(tokens[i:j]):\n",
    "                if not check_capacity(chart, i, j):\n",
    "                    return\n",
    "                rule = Rule(category, tuple(tokens[i:j]), semantics)\n",
    "                chart[(i, j)].append(Parse(rule, tokens[i:j]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(The `check_capacity()` function will be defined in the next section.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unary rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When designing a grammar, it's often convenient to write *unary* rules, which have a single category on the RHS.  However, unary rules are not allowed in Chomsky normal form.  One reason why unary rules are problematic is that badly-designed grammars can contain *unary cycles*, which can cause parsing to get stuck in an infinite regress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rules_with_unary_cycle = [\n",
    "    Rule('$ROOT', '$Ping', lambda sems: sems[0]),\n",
    "    Rule('$Ping', '$Pong', lambda sems: sems[0] + '1'),\n",
    "    Rule('$Pong', '$Ping', lambda sems: sems[0]),\n",
    "    Rule('$Ping', 'ping', '1'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a grammar with these rules is used to parse the input `\"ping\"`, it will generate an endless stream of candidate parses, with semantics '1', '11', '111', '1111', and so on.\n",
    "\n",
    "There are various strategies for preventing unary cycles from causing a runaway parsing process.  We'll adopt one of the simplest: rather than trying to identify unary cycles in advance, we'll just impose an upper bound on the capacity of any chart cell, and if the limit is reached, we'll terminate the parsing process.\n",
    "\n",
    "With this in mind, the definition of `apply_unary_rules()` is straightforward.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def apply_unary_rules(grammar, chart, i, j):\n",
    "    \"\"\"Add parses to chart cell (i, j) by applying unary rules.\"\"\"\n",
    "    # Note that the last line of this method can add new parses to chart[(i,\n",
    "    # j)], the list over which we are iterating.  Because of this, we\n",
    "    # essentially get unary closure \"for free\".  (However, if the grammar\n",
    "    # contains unary cycles, we'll get stuck in a loop, which is one reason for\n",
    "    # check_capacity().)\n",
    "    if hasattr(grammar, 'unary_rules'):\n",
    "        for parse in chart[(i, j)]:\n",
    "            for rule in grammar.unary_rules[(parse.rule.lhs,)]:\n",
    "                if not check_capacity(chart, i, j):\n",
    "                    return\n",
    "                chart[(i, j)].append(Parse(rule, [parse]))\n",
    "\n",
    "MAX_CELL_CAPACITY = 10000\n",
    "\n",
    "# Important for catching e.g. unary cycles.\n",
    "def check_capacity(chart, i, j):\n",
    "    if len(chart[(i, j)]) >= MAX_CELL_CAPACITY:\n",
    "        print('Cell (%d, %d) has reached capacity %d' % (\n",
    "            i, j, MAX_CELL_CAPACITY))\n",
    "        return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `check_capacity()` is not just for taming unary cycles — in fact, it's a general-purpose mechanism for bounding the amount of computation invested in a single parsing task.  Accordingly, we've included calls to it, not only in `apply_unary_rules()`, but also in its three sibling functions.  As you build richer, more complex grammars, you may find that some inputs have hundreds or even thousands of possible parses, and you may begin to run up against the chart cell capacity limit.  Feel free to adjust it as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-ary rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like unary rules, n-ary compositional rules are make grammar engineering easier, by allowing grammar rules to express constraints in natural, intuitive ways.\n",
    "\n",
    "In the previous unit, we saw how to eliminate the n-ary rule `Rule('$E', '$E $BinOp $E')` from a grammar by *binarizing* it — that is, by replacing it with two binary rules having equivalent effect.  Obviously, the binarizing operation can be automated.  The redesigned `add_rule()` calls a function named `add_n_ary_rule()` which performs this binarization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def add_n_ary_rule(grammar, rule):\n",
    "    \"\"\"\n",
    "    Handles adding a rule with three or more non-terminals on the RHS.\n",
    "    We introduce a new category which covers all elements on the RHS except\n",
    "    the first, and then generate two variants of the rule: one which\n",
    "    consumes those elements to produce the new category, and another which\n",
    "    combines the new category which the first element to produce the\n",
    "    original LHS category.  We add these variants in place of the\n",
    "    original rule.  (If the new rules still contain more than two elements\n",
    "    on the RHS, we'll wind up recursing.)\n",
    "\n",
    "    For example, if the original rule is:\n",
    "\n",
    "        Rule('$Z', '$A $B $C $D')\n",
    "\n",
    "    then we create a new category '$Z_$A' (roughly, \"$Z missing $A to the left\"),\n",
    "    and add these rules instead:\n",
    "\n",
    "        Rule('$Z_$A', '$B $C $D')\n",
    "        Rule('$Z', '$A $Z_$A')\n",
    "    \"\"\"\n",
    "    def add_category(base_name):\n",
    "        assert is_cat(base_name)\n",
    "        name = base_name\n",
    "        while name in grammar.categories:\n",
    "            name = name + '_'\n",
    "        grammar.categories.add(name)\n",
    "        return name\n",
    "    category = add_category('%s_%s' % (rule.lhs, rule.rhs[0]))\n",
    "    add_rule(grammar, Rule(category, rule.rhs[1:], lambda sems: sems))\n",
    "    add_rule(grammar, Rule(rule.lhs, (rule.rhs[0], category),\n",
    "                           lambda sems: apply_semantics(rule, [sems[0]] + sems[1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional elements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another convenience for manual grammar engineering is the ability to mark an element in the RHS as optional — in SippyCup, by putting '?' in front of it.  With such a mechanism, we can replace these two rules:\n",
    "\n",
    "<!-- Using pre here because otherwise '$' messes up indentation of second line. -->\n",
    "\n",
    "<pre>\n",
    "Rule('$BikeMode, 'bike')\n",
    "Rule('$BikeMode', 'by bike')\n",
    "</pre>\n",
    "\n",
    "with a single rule containing an optional element:\n",
    "\n",
    "    Rule('$BikeMode', '?by bike')\n",
    "\n",
    "The optional mechanism also makes it easy to define one category as a sequence of another category:\n",
    "\n",
    "<pre>\n",
    "Rule('$Thing', 'thing')\n",
    "Rule('$Things', '$Thing ?$Things')\n",
    "</pre>\n",
    "\n",
    "The following functions enable the optional mechanism."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from types import FunctionType\n",
    "\n",
    "def is_optional(label):\n",
    "    \"\"\"\n",
    "    Returns true iff the given RHS item is optional, i.e., is marked with an\n",
    "    initial '?'.\n",
    "    \"\"\"\n",
    "    return label.startswith('?') and len(label) > 1\n",
    "\n",
    "def contains_optionals(rule):\n",
    "    \"\"\"Returns true iff the given Rule contains any optional items on the RHS.\"\"\"\n",
    "    return any([is_optional(rhsi) for rhsi in rule.rhs])\n",
    "\n",
    "def add_rule_containing_optional(grammar, rule):\n",
    "    \"\"\"\n",
    "    Handles adding a rule which contains an optional element on the RHS.\n",
    "    We find the leftmost optional element on the RHS, and then generate\n",
    "    two variants of the rule: one in which that element is required, and\n",
    "    one in which it is removed.  We add these variants in place of the\n",
    "    original rule.  (If there are more optional elements further to the\n",
    "    right, we'll wind up recursing.)\n",
    "\n",
    "    For example, if the original rule is:\n",
    "\n",
    "        Rule('$Z', '$A ?$B ?$C $D')\n",
    "\n",
    "    then we add these rules instead:\n",
    "\n",
    "        Rule('$Z', '$A $B ?$C $D')\n",
    "        Rule('$Z', '$A ?$C $D')\n",
    "    \"\"\"\n",
    "    # Find index of the first optional element on the RHS.\n",
    "    first = next((idx for idx, elt in enumerate(rule.rhs) if is_optional(elt)), -1)\n",
    "    assert first >= 0\n",
    "    assert len(rule.rhs) > 1, 'Entire RHS is optional: %s' % rule\n",
    "    prefix = rule.rhs[:first]\n",
    "    suffix = rule.rhs[(first + 1):]\n",
    "    # First variant: the first optional element gets deoptionalized.\n",
    "    deoptionalized = (rule.rhs[first][1:],)\n",
    "    add_rule(grammar, Rule(rule.lhs, prefix + deoptionalized + suffix, rule.sem))\n",
    "    # Second variant: the first optional element gets removed.\n",
    "    # If the semantics is a value, just keep it as is.\n",
    "    sem = rule.sem\n",
    "    # But if it's a function, we need to supply a dummy argument for the removed element.\n",
    "    if isinstance(rule.sem, FunctionType):\n",
    "        sem = lambda sems: rule.sem(sems[:first] + [None] + sems[first:])\n",
    "    add_rule(grammar, Rule(rule.lhs, prefix + suffix, sem))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- (TODO: consider whether, and where, to introduce the Domain class.) -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grammar engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's time to start writing some grammar rules for the travel domain.  We're going to adopt a data-driven approach.  Using the 75 training examples as a development set, we will iteratively:\n",
    "\n",
    "- look at examples which are not yet parsed correctly,\n",
    "- identify common obstacles or sources of error,\n",
    "- introduce new rules to address those problems, and then\n",
    "- re-evaluate on the 75 training examples.\n",
    "\n",
    "During grammar engineering, the performance metric we'll focus on is *oracle* accuracy (the proportion of examples for which *any* parse is correct), not accuracy (the proportion of examples for which the *first* parse is correct).  Remember that oracle accuracy is an upper bound on accuracy.  Oracle accuracy is a measure of the expressive power of the grammar: does it have the rules it needs to generate the correct parse?  The gap between oracle accuracy and accuracy, on the other hand, reflects the ability of the scoring model to bring the correct parse to the top of the candidate list.  <!-- (TODO: rewrite.) -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase-bag grammars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the travel domain, we're going to develop a style of grammar known as a *phrase-bag grammar*.  To get a sense of how it will work, let's look at ten example inputs from the training data.\n",
    "\n",
    "<!-- green: destinations <span style=\"background-color:#bbffbb\"> -->\n",
    "<!-- blue: origins <span style=\"background-color:#c0c0ff\"> -->\n",
    "<!-- yellow: travel mode <span style=\"background-color:#ffffaa\"> -->\n",
    "<!-- orange: generic travel <span style=\"background-color:#ffccaa\"> -->\n",
    "<!-- red: request type <span style=\"background-color:#ffaabb\"> -->\n",
    "<!-- gray: optionals <span style=\"background-color:#cccccc\"> -->\n",
    "<!-- purple: unused <span style=\"background-color:#ffbbff\"> -->\n",
    "<!-- TODO: figure out a way to use stylesheets, so the following is less verbose -->\n",
    "<pre>\n",
    "<span style=\"background-color:#ffccaa\">travel</span> <span style=\"background-color:#c0c0ff\">boston</span> <span style=\"background-color:#bbffbb\">to fr. myers fla</span>\n",
    "<span style=\"background-color:#cccccc\">how do i get</span> <span style=\"background-color:#c0c0ff\">from tulsa oklahoma</span> <span style=\"background-color:#bbffbb\">to atlantic city. new jersey</span> <span style=\"background-color:#ffffaa\">by air</span>\n",
    "<span style=\"background-color:#ffffaa\">airbus</span> <span style=\"background-color:#c0c0ff\">from boston</span> <span style=\"background-color:#bbffbb\">to europe</span>\n",
    "<span style=\"background-color:#cccccc\">cheap</span> <span style=\"background-color:#ffccaa\">tickets</span> <span style=\"background-color:#bbffbb\">to south carolina</span>\n",
    "<span style=\"background-color:#bbffbb\">birmingham al</span> <span style=\"background-color:#ffaabb\">distance</span> <span style=\"background-color:#c0c0ff\">from indianapolish in</span>\n",
    "<span style=\"background-color:#ffccaa\">transportation</span> <span style=\"background-color:#bbffbb\">to the philadelphia airport</span>\n",
    "<span style=\"background-color:#cccccc\">one day</span> <span style=\"background-color:#ffffaa\">cruise</span> <span style=\"background-color:#c0c0ff\">from fort lauderdale florida</span>\n",
    "<span style=\"background-color:#ffaabb\">directions</span> <span style=\"background-color:#c0c0ff\">from washington</span> <span style=\"background-color:#bbffbb\">to canada</span>\n",
    "<span style=\"background-color:#ffffaa\">flights</span> <span style=\"background-color:#c0c0ff\">from portland or</span> <span style=\"background-color:#bbffbb\">to seattle wa</span>\n",
    "<span style=\"background-color:#cccccc\">honeymoon</span> <span style=\"background-color:#ffccaa\">trip</span> <span style=\"background-color:#bbffbb\">to hawaii</span>\n",
    "</pre>\n",
    "\n",
    "Here we've highlighted phrases in different colors, according to the roles they play in building the meaning of the input.\n",
    "\n",
    "  - <span style=\"background-color:#bbffbb\">Green</span> phrases indicate the destination of travel.\n",
    "  - <span style=\"background-color:#c0c0ff\">Blue</span> phrases indicate the origin of travel.\n",
    "  - <span style=\"background-color:#ffffaa\">Yellow</span> phrases indicate a mode of travel: air, boat, etc.\n",
    "  - <span style=\"background-color:#ffccaa\">Orange</span> phrases indicate travel of some kind, but do not specify a travel mode.\n",
    "  - <span style=\"background-color:#ffaabb\">Red</span> phrases indicate a specific type of information sought: distance, directions, etc.\n",
    "  - <span style=\"background-color:#cccccc\">Gray</span> phrases indicate \"optional\" words which contribute little to the semantics.  (The modifiers \"one day\" and \"honeymoon\" may be quite meaningful to the user, but our semantic representation is too impoverished to capture them, so they are not relevant for our grammar.)\n",
    "  \n",
    "Note that the *ordering* of phrases in a query isn't particularly important.  Whether the user says <span style=\"background-color:#ffaabb\">directions</span> <span style=\"background-color:#c0c0ff\">from washington</span> <span style=\"background-color:#bbffbb\">to canada</span>, or <span style=\"background-color:#c0c0ff\">from washington</span> <span style=\"background-color:#bbffbb\">to canada</span> <span style=\"background-color:#ffaabb\">directions</span>, or even <span style=\"background-color:#bbffbb\">to canada</span> <span style=\"background-color:#ffaabb\">directions</span> <span style=\"background-color:#c0c0ff\">from washington</span>, the intent is clearly the same.  Some of these formulations might be more natural than others — and more common in the query logs — but all of them should be interpretable by our grammar.\n",
    "\n",
    "That's the motivation for the phrase-bag style of grammar.  In a simple phrase-bag grammar, a valid query is made up of one or more *query elements*, which can appear in any order, and *optionals*, which can be scattered freely amongst the query elements.  (More complex phrase-bag grammars can impose further constraints.)  For the travel domain, we can identify two types of query elements: *travel locations* and *travel arguments*.\n",
    "A travel location is either a *to location* (destination) or a *from location* (origin).\n",
    "A travel argument is either a *travel mode*, a *travel trigger*, or a *request type*.\n",
    "\n",
    "We're ready to write our first grammar rules for the travel domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sems_0(sems):\n",
    "    return sems[0]\n",
    "\n",
    "def sems_1(sems):\n",
    "    return sems[1]\n",
    "\n",
    "def merge_dicts(d1, d2):\n",
    "    if not d2:\n",
    "        return d1\n",
    "    result = d1.copy()\n",
    "    result.update(d2)\n",
    "    return result\n",
    "\n",
    "rules_travel = [\n",
    "    Rule('$ROOT', '$TravelQuery', sems_0),\n",
    "    Rule('$TravelQuery', '$TravelQueryElements',\n",
    "         lambda sems: merge_dicts({'domain': 'travel'}, sems[0])),\n",
    "    Rule('$TravelQueryElements', '$TravelQueryElement ?$TravelQueryElements',\n",
    "         lambda sems: merge_dicts(sems[0], sems[1])),\n",
    "    Rule('$TravelQueryElement', '$TravelLocation', sems_0),\n",
    "    Rule('$TravelQueryElement', '$TravelArgument', sems_0),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These rules are incomplete: `$TravelLocation` and `$TravelArgument` are not yet defined, and there is no mention of optionals.  But these rules define the high-level structure of our phrase-bag grammar.\n",
    "\n",
    "Pay attention to the semantic functions attached to each rule.\n",
    "Remember that our semantic representations are maps of key-value pairs.\n",
    "In the rules which define `$TravelQueryElement`, the semantic function propagates the semantics of the child unchanged.\n",
    "In the rule which defines `$TravelQueryElements`, the semantic function merges the semantics of the children.\n",
    "And in the rule which defines `$TravelQuery`, the semantic function adds a key-value pair to the semantics of the child."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Travel locations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rules introduced above left `$TravelLocation` undefined.  Let's add some rules to define it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rules_travel_locations = [\n",
    "    Rule('$TravelLocation', '$ToLocation', sems_0),\n",
    "    Rule('$TravelLocation', '$FromLocation', sems_0),\n",
    "    Rule('$ToLocation', '$To $Location', lambda sems: {'destination': sems[1]}),\n",
    "    Rule('$FromLocation', '$From $Location', lambda sems: {'origin': sems[1]}),\n",
    "    Rule('$To', 'to'),\n",
    "    Rule('$From', 'from'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks good, but we still need something that defines `$Location`, which will match a phrase like \"`boston`\" and assign to it a semantic representation like `{id: 4930956, name: 'Boston, MA, US'}`.  If only we had such a thing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `GeoNamesAnnotator`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "[Geocoding][] is the process of mapping a piece of text describing a location (such as a place name or an address) into a canonical, machine-readable representation (such as [geographic coordinates][] or a unique identifier in a geographic database).  Because geocoding is a common need across many applications, many geocoding services are available.\n",
    "\n",
    "One such service is [GeoNames][].  GeoNames defines a large geographic database in which each location is identified by a unique integer.  For example, Boston is identified by [4930956][].  GeoNames also provides a free, [RESTful][] API for geocoding requests.  For example, a request to geocode \"Palo Alto\" looks like this:\n",
    "\n",
    "[http://api.geonames.org/searchJSON?q=Palo+Alto&username=demo](http://api.geonames.org/searchJSON?q=Palo+Alto&username=demo)\n",
    "\n",
    "  [geocoding]: http://en.wikipedia.org/wiki/Geocoding\n",
    "  [geographic coordinates]: http://en.wikipedia.org/wiki/Geographic_coordinate_system\n",
    "  [GeoNames]: http://www.geonames.org/\n",
    "  [RESTful]: http://en.wikipedia.org/wiki/Representational_state_transfer\n",
    "  [4930956]: http://www.geonames.org/4930956/\n",
    "\n",
    "In SippyCup, the `GeoNamesAnnotator` (defined in [`geonames.py`](./geonames.py)) is implemented as a wrapper around the GeoNames API.  The job of the `GeoNamesAnnotator` is to recognize `$Location`s and generate semantics for them.  Take a few minutes to skim that code now.\n",
    "\n",
    "Note that the `GeoNamesAnnotator` uses a persistent cache which is pre-populated for phrases in the 100 annotated travel examples, avoiding the need for live calls to the GeoNames API.  However, if you run on any other examples, you will be making live calls.  By default, your requests will specify your username as `wcmac`.  That's fine, but each (free) GeoNames account is limited to 2000 calls per hour.  If too many people are making calls as `wcmac`, that quota could be exhausted quickly.  If that happens, you'll want to [create your own account](http://www.geonames.org/login) on GeoNames.\n",
    "\n",
    "You will soon observe that the annotations are far from perfect.  For example, \"`florida`\" is annotated as `{'id': 3442584, 'name': 'Florida, UY'}`, which denotes a city in Uruguay.  The current implementation of `GeoNamesAnnotator` could be improved in many ways.  For example, while the GeoNames API can return multiple results for ambiguous queries such as \"`florida`\", the `GeoNamesAnnotator` considers only the first result — and there is no guarantee that the first result is the best. It may also be possible to improve the quality of the results by playing with some of the API request parameters documented [here](http://www.geonames.org/export/geonames-search.html).  Some of the [exercises](#travel-exercises) at the end of this unit ask you to explore these possibilities.\n",
    "\n",
    "We're now ready to see our grammar parse an input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded GeoNamesAnnotator cache with 1462 items\n",
      "Created grammar with 11 rules\n",
      "{'domain': 'travel', 'origin': {'name': 'Boston, MA, US', 'id': 4930956}, 'destination': {'name': 'Austin, TX, US', 'id': 4671654}}\n"
     ]
    }
   ],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "from geonames import GeoNamesAnnotator\n",
    "from parsing import *\n",
    "\n",
    "geonames_annotator = GeoNamesAnnotator()\n",
    "rules = rules_travel + rules_travel_locations\n",
    "travel_annotators = [geonames_annotator]\n",
    "grammar = Grammar(rules=rules, annotators=travel_annotators)\n",
    "parses = grammar.parse_input('from boston to austin')\n",
    "print(parses[0].semantics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Travel modes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's turn our attention from travel locations to travel arguments.  One kind of travel argument is a travel mode.  Our semantic representation defines eight travel modes (air, bike, boat, bus, car, taxi, train, and transit), and our training examples illustrate some common ways of expressing each travel mode.  In fact, individual examples will suggest specific lexical rules.  Consider this example:\n",
    "\n",
    "    Example(input='flights from portland or to seattle wa',\n",
    "            semantics={'domain': 'travel', 'mode': 'air',\n",
    "                       'origin': {'id': 5746545, 'name': 'Portland, OR, US'},\n",
    "                       'destination': {'id': 5809844, 'name': 'Seattle, WA, US'}}),\n",
    "\n",
    "The pairing of the token \"`flights`\" with the semantic fragment `{'mode': 'air'}` suggests the rule:\n",
    "\n",
    "    Rule('$AirMode', 'flights', {'mode': 'air'})\n",
    "    \n",
    "Actually, for simplicity, we're going to add the '`air`' semantics one level \"higher\" in the grammar, like this:\n",
    "\n",
    "    Rule('$TravelMode', '$AirMode', {'mode': 'air'})\n",
    "    Rule('$AirMode', 'flights')\n",
    "\n",
    "The rules below illustrate this approach.  These rules will allow our grammar to handle the most common ways of referring to each travel mode.  All of the lexical rules below use phrases which are either highly obvious (such as '`taxi`' for `$TaxiMode`) or else are motivated by specific examples in our training dataset (not the test set!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rules_travel_modes = [\n",
    "    Rule('$TravelArgument', '$TravelMode', sems_0),\n",
    "\n",
    "    Rule('$TravelMode', '$AirMode', {'mode': 'air'}),\n",
    "    Rule('$TravelMode', '$BikeMode', {'mode': 'bike'}),\n",
    "    Rule('$TravelMode', '$BoatMode', {'mode': 'boat'}),\n",
    "    Rule('$TravelMode', '$BusMode', {'mode': 'bus'}),\n",
    "    Rule('$TravelMode', '$CarMode', {'mode': 'car'}),\n",
    "    Rule('$TravelMode', '$TaxiMode', {'mode': 'taxi'}),\n",
    "    Rule('$TravelMode', '$TrainMode', {'mode': 'train'}),\n",
    "    Rule('$TravelMode', '$TransitMode', {'mode': 'transit'}),\n",
    "\n",
    "    Rule('$AirMode', 'air fare'),\n",
    "    Rule('$AirMode', 'air fares'),\n",
    "    Rule('$AirMode', 'airbus'),\n",
    "    Rule('$AirMode', 'airfare'),\n",
    "    Rule('$AirMode', 'airfares'),\n",
    "    Rule('$AirMode', 'airline'),\n",
    "    Rule('$AirMode', 'airlines'),\n",
    "    Rule('$AirMode', '?by air'),\n",
    "    Rule('$AirMode', 'flight'),\n",
    "    Rule('$AirMode', 'flights'),\n",
    "    Rule('$AirMode', 'fly'),\n",
    "\n",
    "    Rule('$BikeMode', '?by bike'),\n",
    "    Rule('$BikeMode', 'bike riding'),\n",
    "\n",
    "    Rule('$BoatMode', '?by boat'),\n",
    "    Rule('$BoatMode', 'cruise'),\n",
    "    Rule('$BoatMode', 'cruises'),\n",
    "    Rule('$BoatMode', 'norwegian cruise lines'),\n",
    "\n",
    "    Rule('$BusMode', '?by bus'),\n",
    "    Rule('$BusMode', 'bus tours'),\n",
    "    Rule('$BusMode', 'buses'),\n",
    "    Rule('$BusMode', 'shutle'),\n",
    "    Rule('$BusMode', 'shuttle'),\n",
    "\n",
    "    Rule('$CarMode', '?by car'),\n",
    "    Rule('$CarMode', 'drive'),\n",
    "    Rule('$CarMode', 'driving'),\n",
    "    Rule('$CarMode', 'gas'),\n",
    "\n",
    "    Rule('$TaxiMode', 'cab'),\n",
    "    Rule('$TaxiMode', 'car service'),\n",
    "    Rule('$TaxiMode', 'taxi'),\n",
    "\n",
    "    Rule('$TrainMode', '?by train'),\n",
    "    Rule('$TrainMode', 'trains'),\n",
    "    Rule('$TrainMode', 'amtrak'),\n",
    "\n",
    "    Rule('$TransitMode', '?by public transportation'),\n",
    "    Rule('$TransitMode', '?by ?public transit'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the rules in action on a toy example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created grammar with 54 rules\n",
      "{'mode': 'train', 'domain': 'travel', 'origin': {'name': 'Boston, MA, US', 'id': 4930956}, 'destination': {'name': 'Austin, TX, US', 'id': 4671654}}\n"
     ]
    }
   ],
   "source": [
    "rules = rules_travel + rules_travel_locations + rules_travel_modes\n",
    "grammar = Grammar(rules=rules, annotators=travel_annotators)\n",
    "parses = grammar.parse_input('from boston to austin by train')\n",
    "print(parses[0].semantics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, it works.\n",
    "\n",
    "We're far from done with the travel grammar, but we now have enough in place that we should be able to parse several of the examples in our training data.  This means that we can start hill-climbing on oracle accuracy!\n",
    "\n",
    "To drive our grammar engineering process, we're going to use a SippyCup utility function called `sample_wins_and_losses()`, which will report our oracle accuracy on the training data, and show some examples we're parsing correctly and some examples we're not.\n",
    "\n",
    "(Note that `sample_wins_and_losses()` requires a `Domain`, a `Model`, and a `Metric`.  A description of these classes is tangential to our presentation.  If you're interested, read some SippyCup code!  It's not very complicated.)\n",
    "\n",
    "<!-- TODO: This is an unsatisfying cop-out.  All these pieces should be explained.  Perhaps very briefly, and perhaps earlier, not right here.  But somewhere.  Otherwise the design isn't right. -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded GeoNamesAnnotator cache with 1462 items\n",
      "================================================================================\n",
      "Evaluating on 75 examples\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "Over 75 examples:\n",
      "\n",
      "semantics accuracy                 0.133\n",
      "semantics oracle accuracy          0.133\n",
      "number of parses                   0.173\n",
      "spurious ambiguity                 0.000\n",
      "has travel parse                   0.173\n",
      "\n",
      "5 of 10 wins on semantics oracle accuracy:\n",
      "\n",
      "  airfares to honolulu hawaii\n",
      "  cruises from new york\n",
      "  flight from atlanta to vermont\n",
      "  flights from portland or to seattle wa\n",
      "  train from moscow to st. petersburg\n",
      "\n",
      "10 of 65 losses on semantics oracle accuracy:\n",
      "\n",
      "  cheap tickets to south carolina\n",
      "  cheapest place to live in florida\n",
      "  direct flights from california to loreto mexico\n",
      "  discount travel flights to austin texas\n",
      "  flights newark to raleigh nc\n",
      "  one day cruise from fort lauderdale florida\n",
      "  public transportation to new jersey\n",
      "  ride this train to roseburg oregon now ther's a town for ya\n",
      "  transportation to the philadelphia airport\n",
      "  university of washington transportation to seatac\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from experiment import sample_wins_and_losses\n",
    "from metrics import SemanticsOracleAccuracyMetric\n",
    "from scoring import Model\n",
    "from travel import TravelDomain\n",
    "\n",
    "domain = TravelDomain()\n",
    "model = Model(grammar=grammar)\n",
    "metric = SemanticsOracleAccuracyMetric()\n",
    "\n",
    "sample_wins_and_losses(domain=domain, model=model, metric=metric, seed=31)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, `sample_wins_and_losses()` doesn't print many details — just a few evaluation metrics, and then examples of wins and losses with the current grammar.  (A \"win\" is an example on which our primary metric has a positive value.)  Our primary metric, semantics oracle accuracy, stands at 0.133 — not great, but greater than zero, so it's a start.  The wins make sense: we can parse queries which consist solely of travel locations and travel modes, with no extraneous elements.  The losses are more interesting, because they will provide the motivation for the next phase of work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Travel triggers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of the examples we're currently failing to parse contain phrases (such as \"tickets\" or \"transportation\") which indicate a travel intent, but do not specify a travel mode.  These phrases constitute the second type of travel argument, namely *travel triggers*.  Inspection of the examples in our training dataset suggest a small number of lexical rules, shown here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rules_travel_triggers = [\n",
    "    Rule('$TravelArgument', '$TravelTrigger', {}),\n",
    "\n",
    "    Rule('$TravelTrigger', 'tickets'),\n",
    "    Rule('$TravelTrigger', 'transportation'),\n",
    "    Rule('$TravelTrigger', 'travel'),\n",
    "    Rule('$TravelTrigger', 'travel packages'),\n",
    "    Rule('$TravelTrigger', 'trip'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run `sample_wins_and_losses()` again, to see how much we've gained, and what to work on next."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created grammar with 60 rules\n",
      "================================================================================\n",
      "Evaluating on 75 examples\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "Over 75 examples:\n",
      "\n",
      "semantics accuracy                 0.173\n",
      "semantics oracle accuracy          0.173\n",
      "number of parses                   0.213\n",
      "spurious ambiguity                 0.000\n",
      "has travel parse                   0.213\n",
      "\n",
      "5 of 13 wins on semantics oracle accuracy:\n",
      "\n",
      "  cruises from new york\n",
      "  flight from atlanta to vermont\n",
      "  flight from ft.lauderdale florida to lexington kentucky\n",
      "  train from moscow to st. petersburg\n",
      "  travel packages from hartford ct. to berlin germany\n",
      "\n",
      "10 of 62 losses on semantics oracle accuracy:\n",
      "\n",
      "  birmingham al distance from indianapolish in\n",
      "  cost of gas from cedar rapids ia to las vegas\n",
      "  direct flights from california to loreto mexico\n",
      "  distance of oxnard ca. to rohnert park ca.\n",
      "  distance usa to peru\n",
      "  jet blue flights to new york\n",
      "  one way air fare to chicago\n",
      "  rochester ny bus tours to yankees stadium\n",
      "  scenic drive to duluth mn\n",
      "  what is 1500 miles away from texas city\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rules = rules_travel + rules_travel_locations + rules_travel_modes + rules_travel_triggers\n",
    "grammar = Grammar(rules=rules, annotators=travel_annotators)\n",
    "model = Model(grammar=grammar)\n",
    "sample_wins_and_losses(domain=domain, model=model, metric=metric, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we've gained about four points (that is, 0.04) in oracle accuracy on the training dataset.  We're making progress!  Again, the losses suggest where to go next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Request types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third and last kind of travel argument is a request type, which indicates a specific type of information sought, such as directions or distance.  We'll adopt the same methodology, adding lexical rules motivated by specific training examples, tied together by higher-level rules which add semantics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rules_request_types = [\n",
    "    Rule('$TravelArgument', '$RequestType', sems_0),\n",
    "\n",
    "    Rule('$RequestType', '$DirectionsRequest', {'type': 'directions'}),\n",
    "    Rule('$RequestType', '$DistanceRequest', {'type': 'distance'}),\n",
    "    Rule('$RequestType', '$ScheduleRequest', {'type': 'schedule'}),\n",
    "    Rule('$RequestType', '$CostRequest', {'type': 'cost'}),\n",
    "\n",
    "    Rule('$DirectionsRequest', 'directions'),\n",
    "    Rule('$DirectionsRequest', 'how do i get'),\n",
    "    Rule('$DistanceRequest', 'distance'),\n",
    "    Rule('$ScheduleRequest', 'schedule'),\n",
    "    Rule('$CostRequest', 'cost'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we'll check our progress using `sample_wins_and_losses()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created grammar with 70 rules\n",
      "================================================================================\n",
      "Evaluating on 75 examples\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "Over 75 examples:\n",
      "\n",
      "semantics accuracy                 0.200\n",
      "semantics oracle accuracy          0.200\n",
      "number of parses                   0.240\n",
      "spurious ambiguity                 0.000\n",
      "has travel parse                   0.240\n",
      "\n",
      "5 of 15 wins on semantics oracle accuracy:\n",
      "\n",
      "  cruise from new york to canada\n",
      "  cruises from new york\n",
      "  directions from washington to canada\n",
      "  flight from ft.lauderdale florida to lexington kentucky\n",
      "  flights from portland or to seattle wa\n",
      "\n",
      "10 of 60 losses on semantics oracle accuracy:\n",
      "\n",
      "  cheapest place to live in florida\n",
      "  cost of gas from cedar rapids ia to las vegas\n",
      "  distance cunmming georgia to chattanooga\n",
      "  distance usa to peru\n",
      "  driving distance washington dc to niagara falls\n",
      "  ithaca to scranton trains\n",
      "  one day cruise from fort lauderdale florida\n",
      "  public transportation to new jersey\n",
      "  santiago chile to atlanta ga. airline schedule 3 26 06\n",
      "  transportation to the philadelphia airport\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rules = rules_travel + rules_travel_locations + rules_travel_modes + rules_travel_triggers + rules_request_types\n",
    "grammar = Grammar(rules=rules, annotators=travel_annotators)\n",
    "model = Model(grammar=grammar)\n",
    "sample_wins_and_losses(domain=domain, model=model, metric=metric, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, oracle accuracy is up to 0.20.  But there's still one big piece we're missing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optionals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A key ingredient of the phrase-bag approach to grammar building is the ability to accept *optional* elements interspersed freely among the query elements.  Optionals are phrases which can be either present or absent; typically, they contribute nothing to the semantics.\n",
    "\n",
    "The following rules illustrate one approach to allowing optionals.  The first two rules allow any `$TravelQueryElement` to combine with an `$Optionals` either to the right or to the left, while ignoring its semantics.  The third rule defines `$Optionals` as a sequence of one or more `$Optional` elements, while the following rules define several specific categories of optionals.  As usual, most of the lexical rules are motivated by specific examples from the training dataset, with a few extras included just because they are super obvious.\n",
    "\n",
    "This is not necessarily the best design!  One of the [exercises](#travel-exercises) will challenge you to do better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rules_optionals = [\n",
    "    Rule('$TravelQueryElement', '$TravelQueryElement $Optionals', sems_0),\n",
    "    Rule('$TravelQueryElement', '$Optionals $TravelQueryElement', sems_1),\n",
    "\n",
    "    Rule('$Optionals', '$Optional ?$Optionals'),\n",
    "\n",
    "    Rule('$Optional', '$Show'),\n",
    "    Rule('$Optional', '$Modifier'),\n",
    "    Rule('$Optional', '$Carrier'),\n",
    "    Rule('$Optional', '$Stopword'),\n",
    "    Rule('$Optional', '$Determiner'),\n",
    "\n",
    "    Rule('$Show', 'book'),\n",
    "    Rule('$Show', 'give ?me'),\n",
    "    Rule('$Show', 'show ?me'),\n",
    "\n",
    "    Rule('$Modifier', 'cheap'),\n",
    "    Rule('$Modifier', 'cheapest'),\n",
    "    Rule('$Modifier', 'discount'),\n",
    "    Rule('$Modifier', 'honeymoon'),\n",
    "    Rule('$Modifier', 'one way'),\n",
    "    Rule('$Modifier', 'direct'),\n",
    "    Rule('$Modifier', 'scenic'),\n",
    "    Rule('$Modifier', 'transatlantic'),\n",
    "    Rule('$Modifier', 'one day'),\n",
    "    Rule('$Modifier', 'last minute'),\n",
    "\n",
    "    Rule('$Carrier', 'delta'),\n",
    "    Rule('$Carrier', 'jet blue'),\n",
    "    Rule('$Carrier', 'spirit airlines'),\n",
    "    Rule('$Carrier', 'amtrak'),\n",
    "\n",
    "    Rule('$Stopword', 'all'),\n",
    "    Rule('$Stopword', 'of'),\n",
    "    Rule('$Stopword', 'what'),\n",
    "    Rule('$Stopword', 'will'),\n",
    "    Rule('$Stopword', 'it'),\n",
    "    Rule('$Stopword', 'to'),\n",
    "\n",
    "    Rule('$Determiner', 'a'),\n",
    "    Rule('$Determiner', 'an'),\n",
    "    Rule('$Determiner', 'the'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we'll check our progress using `sample_wins_and_losses()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created grammar with 104 rules\n",
      "================================================================================\n",
      "Evaluating on 75 examples\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "Over 75 examples:\n",
      "\n",
      "semantics accuracy                 0.400\n",
      "semantics oracle accuracy          0.400\n",
      "number of parses                   0.773\n",
      "spurious ambiguity                 0.039\n",
      "has travel parse                   0.493\n",
      "\n",
      "5 of 30 wins on semantics oracle accuracy:\n",
      "\n",
      "  delta flights to oakland\n",
      "  driving from los angeles to seattle\n",
      "  flight from ft.lauderdale florida to lexington kentucky\n",
      "  honeymoon trip to hawaii\n",
      "  what will it cost to drive from kemerovo to st. petersburg russia\n",
      "\n",
      "10 of 45 losses on semantics oracle accuracy:\n",
      "\n",
      "  buses to sacramento rivercats games\n",
      "  cheap tickets to south carolina\n",
      "  cheapest place to live in florida\n",
      "  lawton va to orlando fl amtrak\n",
      "  nyc flights to buffalo ny\n",
      "  photos at las vegas nevada from april 1 to april 5 2006\n",
      "  public transportation to new jersey\n",
      "  rand mcnally direction to horse shoe casino hammond in\n",
      "  transportation to the philadelphia airport\n",
      "  what is 1500 miles away from texas city\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rules = rules_travel + rules_travel_locations + rules_travel_modes + rules_travel_triggers + rules_request_types + rules_optionals\n",
    "grammar = Grammar(rules=rules, annotators=travel_annotators)\n",
    "model = Model(grammar=grammar)\n",
    "sample_wins_and_losses(domain=domain, model=model, metric=metric, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding support for optionals has doubled oracle accuracy on the training dataset, from 0.200 to 0.400.  This is a big gain!  However, there are still many losses, and many of them share a property: they are *negative examples*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Negative examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A semantic parsing model for a given domain should be able to predict that a given input does *not* belong to the domain.  We call such inputs *negative examples*.  For the travel domain, negative examples include:\n",
    "\n",
    "    discount tickets to new york city ballet\n",
    "    george washington borrows 500 000 from pennsylvania farmer to finance war\n",
    "    ride this train to roseburg oregon now ther's a town for ya\n",
    "\n",
    "Much of the academic literature on semantic parsing describes systems which are required always to produce an in-domain semantic representation, no matter what the input.  If the input is *not* in-domain, the result is usually garbage.  In real-world applications, it's much better to have models which can learn when to produce no positive output.\n",
    "\n",
    "The easiest way to achieve this is to introduce some rules which allow any input to be parsed with \"negative\" semantics, and then learn weights for those rule features in the scoring model.  In the travel domain, the \"negative\" semantic representation is the special value `{'domain': 'other'}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rules_not_travel = [\n",
    "    Rule('$ROOT', '$NotTravelQuery', sems_0),\n",
    "    Rule('$NotTravelQuery', '$Text', {'domain': 'other'}),\n",
    "    Rule('$Text', '$Token ?$Text'),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the last rule depends on the `$Token` category, which can be applied to any token by the `TokenAnnotator`.  So let's add the `TokenAnnotator` to our list of annotators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "travel_annotators = [geonames_annotator, TokenAnnotator()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we'll check our progress using `sample_wins_and_losses()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created grammar with 107 rules\n",
      "================================================================================\n",
      "Evaluating on 75 examples\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "Over 75 examples:\n",
      "\n",
      "semantics accuracy                 0.173\n",
      "semantics oracle accuracy          0.573\n",
      "number of parses                   1.773\n",
      "spurious ambiguity                 0.025\n",
      "has travel parse                   0.493\n",
      "\n",
      "5 of 43 wins on semantics oracle accuracy:\n",
      "\n",
      "  cheap flights to hawaii\n",
      "  directions from washington to canada\n",
      "  jet blue flights to new york\n",
      "  travel packages from hartford ct. to berlin germany\n",
      "  what will it cost to drive from kemerovo to st. petersburg russia\n",
      "\n",
      "10 of 32 losses on semantics oracle accuracy:\n",
      "\n",
      "  bike riding-seattle to portland\n",
      "  birmingham al distance from indianapolish in\n",
      "  directions to rupparena in lexington\n",
      "  distance of oxnard ca. to rohnert park ca.\n",
      "  flights newark to raleigh nc\n",
      "  fly boston to myrtle beach spirit airlines\n",
      "  ithaca to scranton trains\n",
      "  norwegian cruises lines to alaska combined with land tours to denali\n",
      "  rochester ny bus tours to yankees stadium\n",
      "  transatlantic cruise southampton to tampa\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rules = rules_travel + rules_travel_locations + rules_travel_modes + rules_travel_triggers + rules_request_types + rules_optionals + rules_not_travel\n",
    "grammar = Grammar(rules=rules, annotators=travel_annotators)\n",
    "model = Model(grammar=grammar)\n",
    "sample_wins_and_losses(domain=domain, model=model, metric=metric, seed=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've achieved another big gain in oracle accuracy, from 0.400 to 0.573, just by ensuring that we offer a \"negative\" prediction for every input.  (Note that the mean number of parses has increased by exactly 1, from 0.773 to 1.773.)  However, for the first time, a big gap has opened between accuracy, at 0.173, and oracle accuracy, at 0.573.  The problem is that we don't yet have a scoring model for the travel domain, so the ranking of parses is arbitrary.  In order to close the gap, we need to create a scoring model.  One of the [exercises](#travel-exercises) will ask you to pursue this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises <a id=\"travel-exercises\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several of these exercises ask you to measure the impact of your change on key evaluation metrics.  Part of your job is to decide which evaluation metrics are most relevant for the change you're making.  It's probably best to evaluate only on training data, in order to keep the test data unseen during development.  (But the test data is hardly a state secret, so whatever.)\n",
    "\n",
    "### Straightforward\n",
    "\n",
    "1. Select 20 of the 6,588 queries in [aol-travel-queries.txt](./aol-travel-queries.txt) and manually annotate them with target semantics.  Select your queries using uniform random sampling — this will minimize overlap between different people completing this exercise, and therefore maximize the value of the aggregate labor.  (In Python, you can sample from a list using `random.sample()`.)  You will likely find some cases where it's not clear what the right semantics are.  Do your best.  The point of the exercise is to develop an awareness of the challenges of annotation, and to recognize that there's no such thing as perfectly annotated data.\n",
    "\n",
    "1. Many of the remaining errors on training examples occur because the origin isn't marked by \"from\".  Examples include \"`transatlantic cruise southampton to tampa`\", \"`fly boston to myrtle beach spirit airlines`\", and \"`distance usa to peru`\".  Extend the grammar to handle examples like these.  Measure the impact on key evaluation metrics.\n",
    "\n",
    "1. Does your solution to the previous exercise handle examples where some other query element intervenes between origin and destination?  Examples include \"`university of washington transportation to seatac`\", \"`birmingham al distance from indianapolish in`\", and \"`nyc flights to buffalo ny`\".  If not, extend the grammar to handle examples like these.  Measure the impact on key evaluation metrics.\n",
    "\n",
    "1. The current structure of the grammar permits parses containing any number of `$FromLocation`s and `$ToLocation`s, including zero.  Find a way to require that (a) there is at least one `$FromLocation` or `$ToLocation`, (b) there are not multiple `$FromLocation`s or `$ToLocation`s.\n",
    "\n",
    "1. The travel grammar is lacking a scoring model, and it shows a big gap between accuracy and oracle accuracy.  Examine and diagnose some examples where accuracy is 0 even though oracle accuracy is 1.  Propose and implement a scoring model, and measure its efficacy in closing that gap.\n",
    "\n",
    "\n",
    "### Challenging\n",
    "\n",
    "1. Extend `Grammar` to allow rules which mix terminals and non-terminals on the RHS, such as `Rule('$RouteQuery', '$TravelMode from $Location to $Location')`.\n",
    "\n",
    "1. Try to improve the precision of the `GeoNamesAnnotator` by fiddling with the GeoNames API request parameters documented [here](http://www.geonames.org/export/geonames-search.html).  For example, the `featureClass`, `countryBias`, or `orderby` parameters seem like promising targets.\n",
    "\n",
    "1. Try to improve the coverage of the `GeoNamesAnnotator` by enabling it to return multiple annotations for ambiguous location names.  Investigate the impact of varying the maximum number of annotations on various performance metrics, including accuracy, oracle accuracy, and number of parses.  How would you characterize the tradeoff you're making?\n",
    "\n",
    "1. Building on the previous exercise, implement a feature which captures information about the result rank of annotations generated by the `GeoNamesAnnotator`, and see if you can use this feature to narrow the gap between accuracy and oracle accuracy.\n",
    "\n",
    "1. You have probably noticed that one of our standard evaluation metrics is something called \"spurious ambiguity\".  Dig into the SippyCup codebase to figure out what spurious ambiguity is.  Here's a hint: it's something bad, so we want to push that metric toward zero.  Find a training example where it's not zero, and figure out why the example exhibits spurious ambiguity.  Are there changes we can make to the grammar to reduce spurious ambiguity?  Also, why is spurious ambiguity undesirable?\n",
    "\n",
    "1. In its current form, the travel grammar parses lots of queries it shouldn't.  (By \"parses\", we mean \"generates a *positive* parse\".)  This problem is known as *overtriggering*.  The overtriggering problem is hard to observe on our tiny dataset of 100 examples, where most of the examples are positive examples.  Investigate overtriggering by downloading the AOL query dataset, identifying the 1,000 most frequent queries, and running them through the grammar.  How many cases of overtriggering do you find?  Can you suggest some simple changes to minimize overtriggering?  (Warning: the AOL dataset contains lots of queries which may be offensive.  Skip this exercise if you're not down with that.)\n",
    "\n",
    "1. Consider the queries \"`flights los angeles hawaii`\" vs. \"`flights los angeles california`\".  Despite the superficial resemblance, the second query seems to mean flights *to* Los Angeles, not flights *from* Los Angeles *to* California.  But the grammar can successfully interpret both queries only if it permits both interpretations for each query.  This creates a ranking problem: which interpretation should be scored higher?  Can you add to the scoring model a feature which solves the problem?\n",
    "\n",
    "1. Develop and execute a strategy to annotate all 6,588 queries in [aol-travel-queries.txt](./aol-travel-queries.txt) with target semantics using crowdsourcing.  Warning: this is an ambitious exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (C) 2015 Bill MacCartney"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}