\n",
- "Extract Structured Information from Tables in PDF Documents\n",
- " using IBM Watson Discovery and Text Extensions for Pandas\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Introduction\n",
- "\n",
- "Many organizations have valuable information hidden in tables inside human-readable documents like PDF files and web pages. Table identification and extraction technology can turn this human-readable information into a format that data science tools can import and use. Text Extensions for Pandas and Watson Discovery make this process much easier.\n",
- "\n",
- "In this notebook, we'll follow the journey of Allison, an analyst at an investment bank. Allison's employer has assigned her to cover several different companies, one of which is IBM. As part of her analysis, Allison wants to track IBM's revenue over time, broken down by geographical region. That detailed revenue information is all there in IBM's filings with the U.S. Securities and Exchange Commission (SEC). For example, here's IBM's 2019 annual report:\n",
- "\n",
- "\n",
- "\n",
- "Did you see the table of revenue by geography? It's here, on page 39:\n",
- "\n",
- "\n",
- "\n",
- "Here's what that table looks like close up:\n",
- "\n",
- "\n",
- "\n",
- "But this particular table only gives two years' revenue figures. Allison needs to have enough data to draw a meaningful chart of revenue over time. 10 years of annual revenue figures would be a good starting point. \n",
- "\n",
- "Allison has a collection of IBM annual reports going back to 2009. In total, these documents contain about 1500 pages of financial information. Hidden inside those 1500 pages are the detailed revenue figures that Allison wants. She needs to find those figures, extract them from the documents, and import them into her data science tools.\n",
- "\n",
- "Fortunately, Allison has [Watson Discovery](https://www.ibm.com/cloud/watson-discovery), IBM's suite of tools for managing and extracting value from collections of human-readable documents.\n",
- "\n",
- "The cells that follow will show how Allison uses Text Extensions for Pandas and Watson Discovery to import the detailed revenue information from her PDF documents into a Pandas DataFrame...\n",
- "\n",
- "\n",
- "\n",
- "...that she then uses to generate a chart of revenue over time:\n",
- "\n",
- "\n",
- "\n",
- "But first, let's set your environment up so that you can run Allison's code yourself.\n",
- "\n",
- "(If you're just reading through the precomputed outputs of this notebook, you can skip ahead to the section labeled [\"Extract Tables with Watson Discovery\"](#watson_discovery))."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Environment Setup\n",
- "\n",
- "This notebook requires a Python 3.7 or later environment with the following packages:\n",
- "* The dependencies listed in the [\"requirements.txt\" file for Text Extensions for Pandas](https://github.com/CODAIT/text-extensions-for-pandas/blob/master/requirements.txt)\n",
- "* `matplotlib`\n",
- "* `text_extensions_for_pandas`\n",
- "\n",
- "You can satisfy the dependency on `text_extensions_for_pandas` in either of two ways:\n",
- "\n",
- "* Run `pip install text_extensions_for_pandas` before running this notebook. This command adds the library to your Python environment.\n",
- "* Run this notebook out of your local copy of the Text Extensions for Pandas project's [source tree](https://github.com/CODAIT/text-extensions-for-pandas). In this case, the notebook will use the version of Text Extensions for Pandas in your local source tree **if the package is not installed in your Python environment**.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Core Python libraries\n",
- "import json\n",
- "import os\n",
- "import sys\n",
- "from typing import *\n",
- "import pandas as pd\n",
- "from matplotlib import pyplot as plt\n",
- "\n",
- "# And of course we need the text_extensions_for_pandas library itself.\n",
- "try:\n",
- " import text_extensions_for_pandas as tp\n",
- "except ModuleNotFoundError as e:\n",
- " # If we're running from within the project source tree and the parent Python\n",
- " # environment doesn't have the text_extensions_for_pandas package, use the\n",
- " # version in the local source tree.\n",
- " if not os.getcwd().endswith(\"notebooks\"):\n",
- " raise e\n",
- " if \"..\" not in sys.path:\n",
- " sys.path.insert(0, \"..\")\n",
- " import text_extensions_for_pandas as tp"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "
Yr.-to-Yr. Percent Change\\n Excluding Divested Businesses And Adjusted for Currency
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
Total revenue
\n",
- "
$77,147
\n",
- "
$79,591
\n",
- "
-3.1
\n",
- "
-1.0
\n",
- "
0.2
\n",
- "
\n",
- "
\n",
- "
Americas
\n",
- "
$36,274
\n",
- "
$36,994
\n",
- "
-1.9
\n",
- "
-1.1
\n",
- "
0.8
\n",
- "
\n",
- "
\n",
- "
Europe/Middle East/Africa
\n",
- "
24,443
\n",
- "
25,491
\n",
- "
-4.1
\n",
- "
0.4
\n",
- "
1.3
\n",
- "
\n",
- "
\n",
- "
Asia Pacific
\n",
- "
16,430
\n",
- "
17,106
\n",
- "
-4.0
\n",
- "
-3.0
\n",
- "
-2.5
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " \\\n",
- " 2019 2018 Yr.-to-Yr. Percent Change \n",
- "Total revenue $77,147 $79,591 -3.1 \n",
- "Americas $36,274 $36,994 -1.9 \n",
- "Europe/Middle East/Africa 24,443 25,491 -4.1 \n",
- "Asia Pacific 16,430 17,106 -4.0 \n",
- "\n",
- " \\\n",
- " Yr.-to-Yr. Percent Change Adjusted for Currency \n",
- "Total revenue -1.0 \n",
- "Americas -1.1 \n",
- "Europe/Middle East/Africa 0.4 \n",
- "Asia Pacific -3.0 \n",
- "\n",
- " \n",
- " Yr.-to-Yr. Percent Change\\n Excluding Divested Businesses And Adjusted for Currency \n",
- "Total revenue 0.2 \n",
- "Americas 0.8 \n",
- "Europe/Middle East/Africa 1.3 \n",
- "Asia Pacific -2.5 "
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "revenue_2019_df = tp.io.watson.tables.make_table(table_data)\n",
- "revenue_2019_df"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- " \n",
- "\n",
- "The reconstructed dataframe looks good! Here's what the original table in the PDF document looked like:\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "If Allison just wanted to create a DataFrame of 2018/2019 revenue figures, her task would be done. But Allison wants to reconstruct ten years of revenue by geographic region. To do that, she will need to combine information from multiple documents. For tables like this one that have multiple levels of header information, this kind of integration is easier to perform over the \"exploded\" version of the table, where each cell in the table is represented a single row containing all the corresponding header values.\n",
- "\n",
- "Allison passes the same table data from the 2019 report through the Text Extensions for Pandas function `make_exploded_df()` to produce the exploded represention of the table:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
Yr.-to-Yr. Percent Change\\n Excluding Divested Businesses And Adjusted for Currency
\n",
- "
Yr.-to-Yr. Percent Change Adjusted for Currency
\n",
- "
\n",
- "
\n",
- "
row_header_texts_0
\n",
- "
\n",
- "
\n",
- "
\n",
- "
\n",
- "
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
Americas
\n",
- "
$36,994
\n",
- "
$36,274
\n",
- "
(1.9)%
\n",
- "
0.8%
\n",
- "
(1.1)%
\n",
- "
\n",
- "
\n",
- "
Asia Pacific
\n",
- "
17,106
\n",
- "
16,430
\n",
- "
(4.0)
\n",
- "
(2.5)
\n",
- "
(3.0)
\n",
- "
\n",
- "
\n",
- "
Europe/Middle East/Africa
\n",
- "
25,491
\n",
- "
24,443
\n",
- "
(4.1)
\n",
- "
1.3
\n",
- "
0.4
\n",
- "
\n",
- "
\n",
- "
Total revenue
\n",
- "
$79,591
\n",
- "
$77,147
\n",
- "
(3.1 )%
\n",
- "
0.2%
\n",
- "
(1.0)%
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- "column_header_texts 2018 2019 Yr.-to-Yr. Percent Change \\\n",
- "row_header_texts_0 \n",
- "Americas $36,994 $36,274 (1.9)% \n",
- "Asia Pacific 17,106 16,430 (4.0) \n",
- "Europe/Middle East/Africa 25,491 24,443 (4.1) \n",
- "Total revenue $79,591 $77,147 (3.1 )% \n",
- "\n",
- "column_header_texts Yr.-to-Yr. Percent Change\\n Excluding Divested Businesses And Adjusted for Currency \\\n",
- "row_header_texts_0 \n",
- "Americas 0.8% \n",
- "Asia Pacific (2.5) \n",
- "Europe/Middle East/Africa 1.3 \n",
- "Total revenue 0.2% \n",
- "\n",
- "column_header_texts Yr.-to-Yr. Percent Change Adjusted for Currency \n",
- "row_header_texts_0 \n",
- "Americas (1.1)% \n",
- "Asia Pacific (3.0) \n",
- "Europe/Middle East/Africa 0.4 \n",
- "Total revenue (1.0)% "
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "exploded_df.pivot(index=\"row_header_texts_0\", columns=\"column_header_texts\", values=\"text\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "But because she is about to merge this DataFrame with similar data from other documents, Allison keeps the data in exploded format for now. \n",
- "\n",
- "Allison's next task is to write some Pandas transformations that will clean and reformat the DataFrame for each source table prior to merging them all together. She uses the 2019 report's data as a test case for creating this code. The first step is to convert the cell values in the Watson Discovery output from text to numeric values. Text Extensions for Pandas includes a more robust version of [`pandas.to_numeric()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_numeric.html) that can handle common idioms for representing currencies and percentages. Allison uses this function, called `convert_cols_to_numeric()`, to convert all the cell values to numbers. She adds a new column \"value\" to her DataFrame to hold these numbers."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
"
- ],
- "text/plain": [
- " Year Region Revenue\n",
- "0 2019 Total revenue 77147.0\n",
- "1 2018 Total revenue 79591.0\n",
- "5 2019 Americas 36274.0\n",
- "6 2018 Americas 36994.0\n",
- "10 2019 Europe/Middle East/Africa 24443.0\n",
- "11 2018 Europe/Middle East/Africa 25491.0\n",
- "15 2019 Asia Pacific 16430.0\n",
- "16 2018 Asia Pacific 17106.0"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "rows_to_retain.rename(\n",
- " columns={\n",
- " \"row_header_texts_0\": \"Region\",\n",
- " \"column_header_texts\": \"Year\",\n",
- " \"value\": \"Revenue\"\n",
- " })[[\"Year\", \"Region\", \"Revenue\"]]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The code from the last few cells worked to clean up the 2019 data, so Allison copies and pastes that code into a Python function:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [],
- "source": [
- "def dataframe_for_file(filename: str):\n",
- " with open(f\"{FILES_DIR}/{filename}\", \"r\") as f:\n",
- " json_output = json.load(f)\n",
- " table_data = tp.io.watson.tables.parse_response(json_output,\n",
- " select_table=\"Geographic Revenue\")\n",
- " exploded_df, _, _ = tp.io.watson.tables.make_exploded_df(\n",
- " table_data, col_explode_by=\"concat\")\n",
- " rows_to_retain = exploded_df[exploded_df[\"column_header_texts\"].str.fullmatch(\"\\d{4}\")\n",
- " & (exploded_df[\"text\"].str.len() > 0)].copy()\n",
- " rows_to_retain[\"value\"] = tp.io.watson.tables.convert_cols_to_numeric(\n",
- " rows_to_retain[[\"text\"]])\n",
- " rows_to_retain[\"file\"] = filename\n",
- " return (\n",
- " rows_to_retain.rename(columns={\n",
- " \"row_header_texts_0\": \"Region\", \"column_header_texts\": \"Year\", \"value\": \"Revenue\"})\n",
- " [[\"Year\", \"Region\", \"Revenue\"]]\n",
- " )"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Then she calls that function on the Watson Discovery output from the 2019 annual report to verify that it produces the same answer. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Year
\n",
- "
Region
\n",
- "
Revenue
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
2019
\n",
- "
Total revenue
\n",
- "
77147.0
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
2018
\n",
- "
Total revenue
\n",
- "
79591.0
\n",
- "
\n",
- "
\n",
- "
5
\n",
- "
2019
\n",
- "
Americas
\n",
- "
36274.0
\n",
- "
\n",
- "
\n",
- "
6
\n",
- "
2018
\n",
- "
Americas
\n",
- "
36994.0
\n",
- "
\n",
- "
\n",
- "
10
\n",
- "
2019
\n",
- "
Europe/Middle East/Africa
\n",
- "
24443.0
\n",
- "
\n",
- "
\n",
- "
11
\n",
- "
2018
\n",
- "
Europe/Middle East/Africa
\n",
- "
25491.0
\n",
- "
\n",
- "
\n",
- "
15
\n",
- "
2019
\n",
- "
Asia Pacific
\n",
- "
16430.0
\n",
- "
\n",
- "
\n",
- "
16
\n",
- "
2018
\n",
- "
Asia Pacific
\n",
- "
17106.0
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " Year Region Revenue\n",
- "0 2019 Total revenue 77147.0\n",
- "1 2018 Total revenue 79591.0\n",
- "5 2019 Americas 36274.0\n",
- "6 2018 Americas 36994.0\n",
- "10 2019 Europe/Middle East/Africa 24443.0\n",
- "11 2018 Europe/Middle East/Africa 25491.0\n",
- "15 2019 Asia Pacific 16430.0\n",
- "16 2018 Asia Pacific 17106.0"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "dataframe_for_file(\"2019.json\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Looks good! Time to run the same function over an entire stack of reports. Allison puts the names of all her Watson Discovery output files into a single Python list."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['2009.json',\n",
- " '2010.json',\n",
- " '2012.json',\n",
- " '2013.json',\n",
- " '2015.json',\n",
- " '2016.json',\n",
- " '2017.json',\n",
- " '2018.json',\n",
- " '2019.json']"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "all_files = sorted([f for f in os.listdir(FILES_DIR) if f.endswith(\".json\")])\n",
- "all_files"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that the annual reports for 2011 and 2014 aren't in the collection of files that Allison has. But that's ok; each report contains the previous year's figures, so Allison can reconstruct the missing data from adjacent years.\n",
- "\n",
- "Allison calls her `dataframe_for_file()` function on each of the files, then concatenates all of the resulting Pandas DataFrames into a single large DataFrame."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Year
\n",
- "
Region
\n",
- "
Revenue
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
2009
\n",
- "
Total revenue:
\n",
- "
95758.0
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
2008
\n",
- "
Total revenue:
\n",
- "
103630.0
\n",
- "
\n",
- "
\n",
- "
4
\n",
- "
2009
\n",
- "
Geographies:
\n",
- "
93477.0
\n",
- "
\n",
- "
\n",
- "
5
\n",
- "
2008
\n",
- "
Geographies:
\n",
- "
100939.0
\n",
- "
\n",
- "
\n",
- "
8
\n",
- "
2009
\n",
- "
Americas
\n",
- "
40184.0
\n",
- "
\n",
- "
\n",
- "
...
\n",
- "
...
\n",
- "
...
\n",
- "
...
\n",
- "
\n",
- "
\n",
- "
6
\n",
- "
2018
\n",
- "
Americas
\n",
- "
36994.0
\n",
- "
\n",
- "
\n",
- "
10
\n",
- "
2019
\n",
- "
Europe/Middle East/Africa
\n",
- "
24443.0
\n",
- "
\n",
- "
\n",
- "
11
\n",
- "
2018
\n",
- "
Europe/Middle East/Africa
\n",
- "
25491.0
\n",
- "
\n",
- "
\n",
- "
15
\n",
- "
2019
\n",
- "
Asia Pacific
\n",
- "
16430.0
\n",
- "
\n",
- "
\n",
- "
16
\n",
- "
2018
\n",
- "
Asia Pacific
\n",
- "
17106.0
\n",
- "
\n",
- " \n",
- "
\n",
- "
82 rows × 3 columns
\n",
- "
"
- ],
- "text/plain": [
- " Year Region Revenue\n",
- "0 2009 Total revenue: 95758.0\n",
- "1 2008 Total revenue: 103630.0\n",
- "4 2009 Geographies: 93477.0\n",
- "5 2008 Geographies: 100939.0\n",
- "8 2009 Americas 40184.0\n",
- ".. ... ... ...\n",
- "6 2018 Americas 36994.0\n",
- "10 2019 Europe/Middle East/Africa 24443.0\n",
- "11 2018 Europe/Middle East/Africa 25491.0\n",
- "15 2019 Asia Pacific 16430.0\n",
- "16 2018 Asia Pacific 17106.0\n",
- "\n",
- "[82 rows x 3 columns]"
- ]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "revenue_df = pd.concat([dataframe_for_file(f) for f in all_files])\n",
- "revenue_df"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Allison can see that the first four lines of this DataFrame contain total worldwide revenue; and that this total occurred\n",
- "under different names in different documents. Allison is interested in the fine-grained revenue figures, not\n",
- "the totals, so she needs to filter out all these rows with worldwide revenue.\n",
- "\n",
- "What are all the names of geographic regions that IBM annual reports have used over the last ten years?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
Region
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
Total revenue:
\n",
- "
\n",
- "
\n",
- "
4
\n",
- "
Geographies:
\n",
- "
\n",
- "
\n",
- "
8
\n",
- "
Americas
\n",
- "
\n",
- "
\n",
- "
12
\n",
- "
Europe/Middle East/Africa
\n",
- "
\n",
- "
\n",
- "
16
\n",
- "
Asia Pacific
\n",
- "
\n",
- "
\n",
- "
0
\n",
- "
Total revenue
\n",
- "
\n",
- "
\n",
- "
4
\n",
- "
Geographies
\n",
- "
\n",
- "
\n",
- "
16
\n",
- "
Asia Pacifi c
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " Region\n",
- "0 Total revenue:\n",
- "4 Geographies:\n",
- "8 Americas\n",
- "12 Europe/Middle East/Africa\n",
- "16 Asia Pacific\n",
- "0 Total revenue\n",
- "4 Geographies\n",
- "16 Asia Pacifi c"
- ]
- },
- "execution_count": 15,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "revenue_df[[\"Region\"]].drop_duplicates()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "It looks like all the worldwide revenue figures are under some variation of \"Geographies\" or \"Total revenue\". \n",
- "Allison uses Pandas' string matching facilities to filter out the rows whose \"Region\" column contains the \n",
- "words \"geographies\" or \"total\"."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
"
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
- "source": [
- "plt.rcParams.update({'font.size': 16})\n",
- "_ = revenue_table.transpose().plot(title=\"Revenue by Geographic Region\",\n",
- " ylabel=\"Revenue (Millions of US$)\",\n",
- " figsize=(12, 7), ylim=(0, 50000))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now Allison has a clear picture of the detailed revenue data that was hidden inside those 1500 pages of PDF\n",
- "files. As she works on her analyst report, Allison can use the same process to extract DataFrames for\n",
- "other financial metrics too!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/notebooks/Understand_Tables_API.ipynb b/notebooks/Understand_Tables_API.ipynb
new file mode 100644
index 00000000..7618c986
--- /dev/null
+++ b/notebooks/Understand_Tables_API.ipynb
@@ -0,0 +1,4761 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Understand_Tables_API.ipynb:\n",
+ "
\n",
+ "Extract Structured Information from Tables in PDF Documents\n",
+ " using IBM Watson Discovery's Python SDK and Text Extensions for Pandas\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Introduction\n",
+ "\n",
+ "Many organizations have valuable information hidden in tables inside human-readable documents like PDF files and web pages. Table identification and extraction technology can turn this human-readable information into a format that data science tools can import and use. Text Extensions for Pandas and Watson Discovery make this process much easier.\n",
+ "\n",
+ "In this notebook, we'll follow the journey of Allison, an analyst at an investment bank. Allison's employer has assigned her to cover several different companies, one of which is IBM. As part of her analysis, Allison wants to track IBM's revenue over time, broken down by geographical region. That detailed revenue information is all there in IBM's filings with the U.S. Securities and Exchange Commission (SEC). For example, here's IBM's 2019 annual report:\n",
+ "\n",
+ "\n",
+ "\n",
+ "Did you see the table of revenue by geography? It's here, on page 39:\n",
+ "\n",
+ "\n",
+ "\n",
+ "Here's what that table looks like close up:\n",
+ "\n",
+ "\n",
+ "\n",
+ "But this particular table only gives two years' revenue figures. Allison needs to have enough data to draw a meaningful chart of revenue over time. 10 years of annual revenue figures would be a good starting point. \n",
+ "\n",
+ "Allison has a collection of IBM annual reports going back to 2009. In total, these documents contain about 1500 pages of financial information. Hidden inside those 1500 pages are the detailed revenue figures that Allison wants. She needs to find those figures, extract them from the documents, and import them into her data science tools.\n",
+ "\n",
+ "Fortunately, Allison has [Watson Discovery](https://www.ibm.com/cloud/watson-discovery), IBM's suite of tools for managing and extracting value from collections of human-readable documents.\n",
+ "\n",
+ "The cells that follow will show how Allison uses Text Extensions for Pandas and Watson Discovery to import the detailed revenue information from her PDF documents into a Pandas DataFrame...\n",
+ "\n",
+ "\n",
+ "\n",
+ "...that she then uses to generate a chart of revenue over time:\n",
+ "\n",
+ "\n",
+ "\n",
+ "But first, let's set your environment up so that you can run Allison's code yourself.\n",
+ "\n",
+ "(If you're just reading through the precomputed outputs of this notebook, you can skip ahead to the section labeled [\"Extract Tables with Watson Discovery\"](#watson_discovery))."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Environment Setup\n",
+ "\n",
+ "This notebook requires a Python 3.7 or later environment with the following packages:\n",
+ "* The dependencies listed in the [\"requirements.txt\" file for Text Extensions for Pandas](https://github.com/CODAIT/text-extensions-for-pandas/blob/master/requirements.txt)\n",
+ "* `matplotlib`\n",
+ "* `text_extensions_for_pandas`\n",
+ "\n",
+ "You can satisfy the dependency on `text_extensions_for_pandas` in either of two ways:\n",
+ "\n",
+ "* Run `pip install text_extensions_for_pandas` before running this notebook. This command adds the library to your Python environment.\n",
+ "* Run this notebook out of your local copy of the Text Extensions for Pandas project's [source tree](https://github.com/CODAIT/text-extensions-for-pandas). In this case, the notebook will use the version of Text Extensions for Pandas in your local source tree **if the package is not installed in your Python environment**.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Core Python libraries\n",
+ "import json\n",
+ "import os\n",
+ "import sys\n",
+ "import requests\n",
+ "import glob\n",
+ "from typing import *\n",
+ "import pandas as pd\n",
+ "from matplotlib import pyplot as plt\n",
+ "from IPython.display import HTML\n",
+ "from base64 import b64encode\n",
+ "\n",
+ "# IBM libraries\n",
+ "from ibm_watson import DiscoveryV2\n",
+ "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator\n",
+ "\n",
+ "# And of course we need the text_extensions_for_pandas library itself.\n",
+ "try:\n",
+ " import text_extensions_for_pandas as tp\n",
+ "except ModuleNotFoundError as e:\n",
+ " # If we're running from within the project source tree and the parent Python\n",
+ " # environment doesn't have the text_extensions_for_pandas package, use the\n",
+ " # version in the local source tree.\n",
+ " if not os.getcwd().endswith(\"notebooks\"):\n",
+ " raise e\n",
+ " if \"..\" not in sys.path:\n",
+ " sys.path.insert(0, \"..\")\n",
+ " import text_extensions_for_pandas as tp"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "# Extract Tables with Watson Discovery\n",
+ "\n",
+ "Allison connects to the [Watson Discovery](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-install) component of her firm's [IBM Cloud Pak for Data](https://www.ibm.com/products/cloud-pak-for-data) installation on their [OpenShift](https://www.openshift.com/) cluster and uploads her stack of IBM annual reports to her project. Then she uses the Watson Discovery's [Table Understanding enrichment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables) to identify tables in the PDF documents and to extract detailed information about the cells and headers that make up each table.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "You'll need two pieces of information to access your instance of Watson Discovery: An **API key** and a **service URL**. If you're using Watson Discovery on the IBM Cloud, you can find your API key and service URL in the IBM Cloud web UI. Navigate to the [resource list](https://cloud.ibm.com/resources) and click on your instance of Watson Discovery to open the management UI for your service. Then click on the \"Manage\" tab to show a page with your API key and service URL.\n",
+ "\n",
+ "The cell that follows assumes that you are using the environment variables `IBM_API_KEY` and `IBM_SERVICE_URL` to store your credentials. If you're running this notebook in Jupyter on your laptop, you can set these environment variables while starting up `jupyter notebook` or `jupyter lab`. For example:\n",
+ "``` console\n",
+ "IBM_API_KEY='' \\\n",
+ "IBM_SERVICE_URL='' \\\n",
+ " jupyter lab\n",
+ "```\n",
+ "Alternately, you can uncomment the first two lines of code below to set the `IBM_API_KEY` and `IBM_SERVICE_URL` environment variables directly. **Be careful not to store your API key in any publicly-accessible location!**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# If you need to embed your credentials inline, uncomment the following two lines and\n",
+ "# paste your credentials in the indicated locations.\n",
+ "# os.environ[\"IBM_API_KEY\"] = \"\"\n",
+ "# os.environ[\"IBM_SERVICE_URL\"] = \"\"\n",
+ "# Retrieve the API key for your Watson Discovery service instance\n",
+ "if \"IBM_API_KEY\" not in os.environ:\n",
+ " raise ValueError(\"Expected Watson Discovery api key in the environment variable 'IBM_API_KEY'\")\n",
+ "api_key = os.environ.get(\"IBM_API_KEY\") \n",
+ "# Retrieve the service URL for your Watson Discovry service instance\n",
+ "if \"IBM_SERVICE_URL\" not in os.environ:\n",
+ " raise ValueError(\"Expected Watson Discovery service URL in the environment variable 'IBM_SERVICE_URL'\")\n",
+ "service_url = os.environ.get(\"IBM_SERVICE_URL\") "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Connect to the Watson Discovery Python API\n",
+ "\n",
+ "This notebook uses the IBM Watson Python SDK to perform authentication on the IBM Cloud via the \n",
+ "`IAMAuthenticator` class. See [the IBM Watson Python SDK documentation](https://github.com/watson-developer-cloud/python-sdk#iam) for more information. \n",
+ "\n",
+ "Allison starts by using the API key and service URL from the previous cell to create an instance of the\n",
+ "Python API for Watson Discovery."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "authenticator = IAMAuthenticator(api_key)\n",
+ "version='2020-08-30'\n",
+ "discovery = DiscoveryV2(\n",
+ " version=version,\n",
+ " authenticator=authenticator\n",
+ ")\n",
+ "\n",
+ "discovery.set_service_url(service_url)\n",
+ "discovery.set_disable_ssl_verification(True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note: This notebook shows you how to run the whole usecase end to end using the Python SDK. Alternatively, Allison could use the Watson Discovery tooling to create a project, create a collection, enable the pre-trained model to apply the Table Understanding enrichment, and add the pdf documents to the collection before querying the collection as shown in [this additional notebook](./Understand_Tables_Tooling.ipynb#watson_discovery_tooling_demo)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "# Create a Project through the Watson Discovery Service API\n",
+ "\n",
+ "She creates a new Watson Discovery project:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n"
+ ]
+ }
+ ],
+ "source": [
+ "project = discovery.create_project(\n",
+ " \"TextExtensionsForPandasTableUnderstanding\",\n",
+ " \"document_retrieval\"\n",
+ " ).get_result()\n",
+ "project_id = project['project_id']"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Create a Collection through the Watson Discovery Service API\n",
+ "She then creates a new collection in her project."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n"
+ ]
+ }
+ ],
+ "source": [
+ "collection = discovery.create_collection(\n",
+ " project_id = project_id,\n",
+ " name = \"IBM-10K\"\n",
+ " ).get_result()\n",
+ "collection_id = collection['collection_id']"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Before Allison can apply the table understanding to her collection, she needs to first enable the Pre-trained-models option for her collection to get the tables annotated for her automatically using the Smart Document Understanding tool, to do so she calls following function:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def enable_pretrained_models(project_name:str, collection_name:str):\n",
+ " auth = (\"apikey\", api_key)\n",
+ " response = requests.get(f\"{service_url}/v2/datasets?version={version}\", auth=auth)\n",
+ " for dataset in response.json()['datasets']:\n",
+ " if (dataset['name'] == collection_name and \n",
+ " dataset['collections'][0]['project_name'] == project_name):\n",
+ " config_id = dataset['id']\n",
+ " configurations = requests.get \\\n",
+ " (f\"{service_url}/v2/configurations/{config_id}/converters?version={version}\", auth=auth).json()\n",
+ " for config in configurations[\"converterConfigurations\"]:\n",
+ " if config['name'] == \"ama-converter-sdu-converter\":\n",
+ " config['converterSettings']['structure_model'] = 'static'\n",
+ " headers = {'Content-type': 'application/json'}\n",
+ " response = requests.put \\\n",
+ " (f\"{service_url}/v2/configurations/{config_id}/converters?version={version}\", \\\n",
+ " auth = auth, data = json.dumps(configurations), headers = headers)\n",
+ "\n",
+ "enable_pretrained_models('TextExtensionsForPandasTableUnderstanding', 'IBM-10K')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Then she applies the Table Understanding enrichment to get detailed information about tables and table-related data within documents.\n",
+ "\n",
+ "She should apply the enrichment only to a field that contains an HTML representation of the table. That's the only way that the enrichment can read the parts of the table, such as header rows and columns, and interpret the information in the table properly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{'name': 'IBM-10K',\n",
+ " 'collection_id': '8c655b3a-0ad4-3f36-0000-017eea50450a',\n",
+ " 'description': '',\n",
+ " 'created': '2022-02-11T19:43:57.451Z',\n",
+ " 'language': 'en',\n",
+ " 'enrichments': [{'enrichment_id': '701db916-fc83-57ab-0000-000000000012',\n",
+ " 'fields': ['html']}]}"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "discovery.update_collection(\n",
+ " project_id = project_id,\n",
+ " collection_id = collection_id,\n",
+ " enrichments = [{\"enrichment_id\": \"701db916-fc83-57ab-0000-000000000012\", \"fields\": [\"html\"] }]\n",
+ " ).get_result()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Pass Documents to the Watson Discovery Service API\n",
+ "\n",
+ "Once she's created a collection and updated the collection with Table Understanding enrichment, she can upload the documents through \n",
+ "the service by invoking the [`add_document()` method](https://cloud.ibm.com/apidocs/discovery-data?code=python#adddocument)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n",
+ "/Users/monireh/opt/anaconda3/envs/text-extensions-for-pandas/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.us-south.discovery.watson.cloud.ibm.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
+ " warnings.warn(\n"
+ ]
+ }
+ ],
+ "source": [
+ "for file in glob.glob(\"../resources/IBM Annual Report/*.pdf\"):\n",
+ " with open(file,'rb') as fileinfo:\n",
+ " response = discovery.add_document(\n",
+ " project_id = project_id,\n",
+ " collection_id = collection_id,\n",
+ " file=fileinfo,\n",
+ " filename=file,\n",
+ " fileinfo='application/pdf'\n",
+ " ).get_result()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "
"
+ ],
+ "text/plain": [
+ " Yr.-to-Yr.\n",
+ " Yr.-to-Yr. Percent Change\n",
+ " Yr.-to-Yr. Adjusted for\n",
+ " 2017 2016 Percent Change Currency\n",
+ " Percent \n",
+ " \n",
+ " Change \n",
+ " \n",
+ "Total revenue $79,139 $79,919 (1.0)% (1.3)%\n",
+ "Geographies $78,793 $79,594 (1.0)% (1.4)%\n",
+ "Americas 37479 37513 -0.1 -0.6\n",
+ "Europe/Middle East/Africa 24345 24769 -1.7 -2.8\n",
+ "Asia Pacific 16970 17313 -2 -1.1"
+ ]
+ },
+ "execution_count": 13,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "revenue_2017_df = tp.io.watson.tables.make_table(table_data)\n",
+ "revenue_2017_df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ " \n",
+ "\n",
+ "The reconstructed dataframe looks good! Here's what the original table in the PDF document looked like:\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If Allison just wanted to create a DataFrame of 2016/2017 revenue figures, her task would be done. But Allison wants to reconstruct ten years of revenue by geographic region. To do that, she will need to combine information from multiple documents. For tables like this one that have multiple levels of header information, this kind of integration is easier to perform over the \"exploded\" version of the table, where each cell in the table is represented a single row containing all the corresponding header values.\n",
+ "\n",
+ "Allison passes the same table data from the 2017 report through the Text Extensions for Pandas function `make_exploded_df()` to produce the exploded represention of the table:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
"
+ ],
+ "text/plain": [
+ "column_header_texts 2016 2017 \\\n",
+ "row_header_texts_0 \n",
+ "Americas 37,513 37,479 \n",
+ "Asia Pacific 17,313 16,970 \n",
+ "Europe/Middle East/Africa 24,769 24,345 \n",
+ "Geographies $79,594 $78,793 \n",
+ "Total revenue $79,919 $79,139 \n",
+ "\n",
+ "column_header_texts Yr.-to-Yr. Percent Change Adjusted for Currency \\\n",
+ "row_header_texts_0 \n",
+ "Americas (0.6) \n",
+ "Asia Pacific (1.1) \n",
+ "Europe/Middle East/Africa (2.8) \n",
+ "Geographies (1.4)% \n",
+ "Total revenue (1.3)% \n",
+ "\n",
+ "column_header_texts Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent Change \n",
+ "row_header_texts_0 \n",
+ "Americas (0.1) \n",
+ "Asia Pacific (2.0) \n",
+ "Europe/Middle East/Africa (1.7) \n",
+ "Geographies (1.0)% \n",
+ "Total revenue (1.0)% "
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "exploded_df.pivot(index=\"row_header_texts_0\", columns=\"column_header_texts\", values=\"text\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "But because she is about to merge this DataFrame with similar data from other documents, Allison keeps the data in exploded format for now. \n",
+ "\n",
+ "Allison's next task is to write some Pandas transformations that will clean and reformat the DataFrame for each source table prior to merging them all together. She uses the 2017 report's data as a test case for creating this code. The first step is to convert the cell values in the Watson Discovery output from text to numeric values. Text Extensions for Pandas includes a more robust version of [`pandas.to_numeric()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_numeric.html) that can handle common idioms for representing currencies and percentages. Allison uses this function, called `convert_cols_to_numeric()`, to convert all the cell values to numbers. She adds a new column \"value\" to her DataFrame to hold these numbers."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
text
\n",
+ "
row_header_texts_0
\n",
+ "
column_header_texts
\n",
+ "
attributes.type
\n",
+ "
value
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
0
\n",
+ "
$79,139
\n",
+ "
Total revenue
\n",
+ "
2017
\n",
+ "
[Currency]
\n",
+ "
79139.0
\n",
+ "
\n",
+ "
\n",
+ "
1
\n",
+ "
$79,919
\n",
+ "
Total revenue
\n",
+ "
2016
\n",
+ "
[Currency]
\n",
+ "
79919.0
\n",
+ "
\n",
+ "
\n",
+ "
2
\n",
+ "
(1.0)%
\n",
+ "
Total revenue
\n",
+ "
Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C...
\n",
+ "
[Number]
\n",
+ "
-1.0
\n",
+ "
\n",
+ "
\n",
+ "
3
\n",
+ "
(1.3)%
\n",
+ "
Total revenue
\n",
+ "
Yr.-to-Yr. Percent Change Adjusted for Currency
\n",
+ "
[Number]
\n",
+ "
-1.3
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
$78,793
\n",
+ "
Geographies
\n",
+ "
2017
\n",
+ "
[Currency]
\n",
+ "
78793.0
\n",
+ "
\n",
+ "
\n",
+ "
5
\n",
+ "
$79,594
\n",
+ "
Geographies
\n",
+ "
2016
\n",
+ "
[Currency]
\n",
+ "
79594.0
\n",
+ "
\n",
+ "
\n",
+ "
6
\n",
+ "
(1.0)%
\n",
+ "
Geographies
\n",
+ "
Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C...
\n",
+ "
[Number]
\n",
+ "
-1.0
\n",
+ "
\n",
+ "
\n",
+ "
7
\n",
+ "
(1.4)%
\n",
+ "
Geographies
\n",
+ "
Yr.-to-Yr. Percent Change Adjusted for Currency
\n",
+ "
[Number]
\n",
+ "
-1.4
\n",
+ "
\n",
+ "
\n",
+ "
8
\n",
+ "
37,479
\n",
+ "
Americas
\n",
+ "
2017
\n",
+ "
[Number]
\n",
+ "
37479.0
\n",
+ "
\n",
+ "
\n",
+ "
9
\n",
+ "
37,513
\n",
+ "
Americas
\n",
+ "
2016
\n",
+ "
[Number]
\n",
+ "
37513.0
\n",
+ "
\n",
+ "
\n",
+ "
10
\n",
+ "
(0.1)
\n",
+ "
Americas
\n",
+ "
Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C...
\n",
+ "
[Number]
\n",
+ "
-0.1
\n",
+ "
\n",
+ "
\n",
+ "
11
\n",
+ "
(0.6)
\n",
+ "
Americas
\n",
+ "
Yr.-to-Yr. Percent Change Adjusted for Currency
\n",
+ "
[Number]
\n",
+ "
-0.6
\n",
+ "
\n",
+ "
\n",
+ "
12
\n",
+ "
24,345
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
2017
\n",
+ "
[Number]
\n",
+ "
24345.0
\n",
+ "
\n",
+ "
\n",
+ "
13
\n",
+ "
24,769
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
2016
\n",
+ "
[Number]
\n",
+ "
24769.0
\n",
+ "
\n",
+ "
\n",
+ "
14
\n",
+ "
(1.7)
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C...
\n",
+ "
[Number]
\n",
+ "
-1.7
\n",
+ "
\n",
+ "
\n",
+ "
15
\n",
+ "
(2.8)
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
Yr.-to-Yr. Percent Change Adjusted for Currency
\n",
+ "
[Number]
\n",
+ "
-2.8
\n",
+ "
\n",
+ "
\n",
+ "
16
\n",
+ "
16,970
\n",
+ "
Asia Pacific
\n",
+ "
2017
\n",
+ "
[Number]
\n",
+ "
16970.0
\n",
+ "
\n",
+ "
\n",
+ "
17
\n",
+ "
17,313
\n",
+ "
Asia Pacific
\n",
+ "
2016
\n",
+ "
[Number]
\n",
+ "
17313.0
\n",
+ "
\n",
+ "
\n",
+ "
18
\n",
+ "
(2.0)
\n",
+ "
Asia Pacific
\n",
+ "
Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C...
\n",
+ "
[Number]
\n",
+ "
-2.0
\n",
+ "
\n",
+ "
\n",
+ "
19
\n",
+ "
(1.1)
\n",
+ "
Asia Pacific
\n",
+ "
Yr.-to-Yr. Percent Change Adjusted for Currency
\n",
+ "
[Number]
\n",
+ "
-1.1
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " text row_header_texts_0 \\\n",
+ "0 $79,139 Total revenue \n",
+ "1 $79,919 Total revenue \n",
+ "2 (1.0)% Total revenue \n",
+ "3 (1.3)% Total revenue \n",
+ "4 $78,793 Geographies \n",
+ "5 $79,594 Geographies \n",
+ "6 (1.0)% Geographies \n",
+ "7 (1.4)% Geographies \n",
+ "8 37,479 Americas \n",
+ "9 37,513 Americas \n",
+ "10 (0.1) Americas \n",
+ "11 (0.6) Americas \n",
+ "12 24,345 Europe/Middle East/Africa \n",
+ "13 24,769 Europe/Middle East/Africa \n",
+ "14 (1.7) Europe/Middle East/Africa \n",
+ "15 (2.8) Europe/Middle East/Africa \n",
+ "16 16,970 Asia Pacific \n",
+ "17 17,313 Asia Pacific \n",
+ "18 (2.0) Asia Pacific \n",
+ "19 (1.1) Asia Pacific \n",
+ "\n",
+ " column_header_texts attributes.type value \n",
+ "0 2017 [Currency] 79139.0 \n",
+ "1 2016 [Currency] 79919.0 \n",
+ "2 Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C... [Number] -1.0 \n",
+ "3 Yr.-to-Yr. Percent Change Adjusted for Currency [Number] -1.3 \n",
+ "4 2017 [Currency] 78793.0 \n",
+ "5 2016 [Currency] 79594.0 \n",
+ "6 Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C... [Number] -1.0 \n",
+ "7 Yr.-to-Yr. Percent Change Adjusted for Currency [Number] -1.4 \n",
+ "8 2017 [Number] 37479.0 \n",
+ "9 2016 [Number] 37513.0 \n",
+ "10 Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C... [Number] -0.1 \n",
+ "11 Yr.-to-Yr. Percent Change Adjusted for Currency [Number] -0.6 \n",
+ "12 2017 [Number] 24345.0 \n",
+ "13 2016 [Number] 24769.0 \n",
+ "14 Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C... [Number] -1.7 \n",
+ "15 Yr.-to-Yr. Percent Change Adjusted for Currency [Number] -2.8 \n",
+ "16 2017 [Number] 16970.0 \n",
+ "17 2016 [Number] 17313.0 \n",
+ "18 Yr.-to-Yr. Yr.-to-Yr. Percent Change Percent C... [Number] -2.0 \n",
+ "19 Yr.-to-Yr. Percent Change Adjusted for Currency [Number] -1.1 "
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "exploded_df[\"value\"] = \\\n",
+ " tp.io.watson.tables.convert_cols_to_numeric(exploded_df[[\"text\"]])\n",
+ "exploded_df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now all the cell values have been converted to floating-point numbers, but only some of these numbers represent revenue. Looking at the 2017 data, Allison can see that the revenue numbers have 4-digit years in their column headers. So she filters the DataFrame down to just those rows with 4-digit numbers in the \"column_header_texts\" column."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
"
+ ],
+ "text/plain": [
+ " Year Region Revenue\n",
+ "0 2017 Total revenue 79139.0\n",
+ "1 2016 Total revenue 79919.0\n",
+ "4 2017 Geographies 78793.0\n",
+ "5 2016 Geographies 79594.0\n",
+ "8 2017 Americas 37479.0\n",
+ "9 2016 Americas 37513.0\n",
+ "12 2017 Europe/Middle East/Africa 24345.0\n",
+ "13 2016 Europe/Middle East/Africa 24769.0\n",
+ "16 2017 Asia Pacific 16970.0\n",
+ "17 2016 Asia Pacific 17313.0"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "rows_to_retain.rename(\n",
+ " columns={\n",
+ " \"row_header_texts_0\": \"Region\",\n",
+ " \"column_header_texts\": \"Year\",\n",
+ " \"value\": \"Revenue\"\n",
+ " })[[\"Year\", \"Region\", \"Revenue\"]]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The code from the last few cells worked to clean up the 2017 data, so Allison copies and pastes that code into a Python function:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def dataframe_for_table(table: str):\n",
+ "\n",
+ " table_data = tp.io.watson.tables.parse_response({\"tables\":[table]})\n",
+ " exploded_df, _, _ = tp.io.watson.tables.make_exploded_df(\n",
+ " table_data, col_explode_by=\"concat\")\n",
+ " \n",
+ " if \"column_header_texts\" in exploded_df.columns and \"row_header_texts_0\" in exploded_df.columns: \n",
+ " rows_to_retain = exploded_df[((exploded_df[\"column_header_texts\"].str.fullmatch(\"\\d{4}\"))\n",
+ " & (exploded_df[\"text\"].str.len() > 0)\n",
+ " )].copy()\n",
+ " rows_to_retain[\"value\"] = tp.io.watson.tables.convert_cols_to_numeric(\n",
+ " rows_to_retain[[\"text\"]])\n",
+ " rows_to_retain[\"file\"] = \"filename\"\n",
+ " return (\n",
+ " rows_to_retain.rename(columns={\n",
+ " \"row_header_texts_0\": \"Region\", \"column_header_texts\": \"Year\", \"value\": \"Revenue\"})\n",
+ " [[\"Year\", \"Region\", \"Revenue\"]]\n",
+ " )\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Then she calls that function on the Watson Discovery output from the 2017 annual report to verify that it produces the same answer. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Year
\n",
+ "
Region
\n",
+ "
Revenue
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
0
\n",
+ "
2017
\n",
+ "
Total revenue
\n",
+ "
79139.0
\n",
+ "
\n",
+ "
\n",
+ "
1
\n",
+ "
2016
\n",
+ "
Total revenue
\n",
+ "
79919.0
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
2017
\n",
+ "
Geographies
\n",
+ "
78793.0
\n",
+ "
\n",
+ "
\n",
+ "
5
\n",
+ "
2016
\n",
+ "
Geographies
\n",
+ "
79594.0
\n",
+ "
\n",
+ "
\n",
+ "
8
\n",
+ "
2017
\n",
+ "
Americas
\n",
+ "
37479.0
\n",
+ "
\n",
+ "
\n",
+ "
9
\n",
+ "
2016
\n",
+ "
Americas
\n",
+ "
37513.0
\n",
+ "
\n",
+ "
\n",
+ "
12
\n",
+ "
2017
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
24345.0
\n",
+ "
\n",
+ "
\n",
+ "
13
\n",
+ "
2016
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
24769.0
\n",
+ "
\n",
+ "
\n",
+ "
16
\n",
+ "
2017
\n",
+ "
Asia Pacific
\n",
+ "
16970.0
\n",
+ "
\n",
+ "
\n",
+ "
17
\n",
+ "
2016
\n",
+ "
Asia Pacific
\n",
+ "
17313.0
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Year Region Revenue\n",
+ "0 2017 Total revenue 79139.0\n",
+ "1 2016 Total revenue 79919.0\n",
+ "4 2017 Geographies 78793.0\n",
+ "5 2016 Geographies 79594.0\n",
+ "8 2017 Americas 37479.0\n",
+ "9 2016 Americas 37513.0\n",
+ "12 2017 Europe/Middle East/Africa 24345.0\n",
+ "13 2016 Europe/Middle East/Africa 24769.0\n",
+ "16 2017 Asia Pacific 16970.0\n",
+ "17 2016 Asia Pacific 17313.0"
+ ]
+ },
+ "execution_count": 20,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "dataframe_for_table(tables[0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Looks good! Time to run the same function over an entire stack of reports."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Allison calls her `dataframe_for_table()` function on each of the tables, then concatenates all of the resulting Pandas DataFrames into a single large DataFrame."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Year
\n",
+ "
Region
\n",
+ "
Revenue
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
0
\n",
+ "
2017
\n",
+ "
Total revenue
\n",
+ "
79139.0
\n",
+ "
\n",
+ "
\n",
+ "
1
\n",
+ "
2016
\n",
+ "
Total revenue
\n",
+ "
79919.0
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
2017
\n",
+ "
Geographies
\n",
+ "
78793.0
\n",
+ "
\n",
+ "
\n",
+ "
5
\n",
+ "
2016
\n",
+ "
Geographies
\n",
+ "
79594.0
\n",
+ "
\n",
+ "
\n",
+ "
8
\n",
+ "
2017
\n",
+ "
Americas
\n",
+ "
37479.0
\n",
+ "
\n",
+ "
\n",
+ "
...
\n",
+ "
...
\n",
+ "
...
\n",
+ "
...
\n",
+ "
\n",
+ "
\n",
+ "
17
\n",
+ "
2009
\n",
+ "
Asia Pacific
\n",
+ "
20710.0
\n",
+ "
\n",
+ "
\n",
+ "
0
\n",
+ "
2018
\n",
+ "
Total revenue
\n",
+ "
79591.0
\n",
+ "
\n",
+ "
\n",
+ "
5
\n",
+ "
2018
\n",
+ "
Americas
\n",
+ "
36994.0
\n",
+ "
\n",
+ "
\n",
+ "
9
\n",
+ "
2018
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
25491.0
\n",
+ "
\n",
+ "
\n",
+ "
13
\n",
+ "
2018
\n",
+ "
Asia Pacific
\n",
+ "
17106.0
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
170 rows × 3 columns
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Year Region Revenue\n",
+ "0 2017 Total revenue 79139.0\n",
+ "1 2016 Total revenue 79919.0\n",
+ "4 2017 Geographies 78793.0\n",
+ "5 2016 Geographies 79594.0\n",
+ "8 2017 Americas 37479.0\n",
+ ".. ... ... ...\n",
+ "17 2009 Asia Pacific 20710.0\n",
+ "0 2018 Total revenue 79591.0\n",
+ "5 2018 Americas 36994.0\n",
+ "9 2018 Europe/Middle East/Africa 25491.0\n",
+ "13 2018 Asia Pacific 17106.0\n",
+ "\n",
+ "[170 rows x 3 columns]"
+ ]
+ },
+ "execution_count": 21,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "revenue_df = pd.concat([dataframe_for_table(t) for t in tables])\n",
+ "revenue_df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Allison can see that the first four lines of this DataFrame contain total worldwide revenue; and that this total occurred\n",
+ "under different names in different documents. Allison is interested in the fine-grained revenue figures, not\n",
+ "the totals, so she needs to filter out all these rows with worldwide revenue.\n",
+ "\n",
+ "What are all the names of geographic regions that IBM annual reports have used over the last ten years?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ "
Region
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "
\n",
+ "
0
\n",
+ "
Total revenue
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
Geographies
\n",
+ "
\n",
+ "
\n",
+ "
8
\n",
+ "
Americas
\n",
+ "
\n",
+ "
\n",
+ "
12
\n",
+ "
Europe/Middle East/Africa
\n",
+ "
\n",
+ "
\n",
+ "
16
\n",
+ "
Asia Pacific
\n",
+ "
\n",
+ "
\n",
+ "
0
\n",
+ "
Total revenue:
\n",
+ "
\n",
+ "
\n",
+ "
4
\n",
+ "
Geographies:
\n",
+ "
\n",
+ "
\n",
+ "
9
\n",
+ "
2009. The increase in total expense and other ...
\n",
+ "
\n",
+ "
\n",
+ "
25
\n",
+ "
Examples of the company's investments include:
\n",
+ "
\n",
+ "
\n",
+ "
29
\n",
+ "
NaN
\n",
+ "
\n",
+ "
\n",
+ "
33
\n",
+ "
• Industry sales skills to support Smarter Planet
\n",
+ "
\n",
+ "
\n",
+ "
16
\n",
+ "
Asia Pacifi c
\n",
+ "
\n",
+ "
\n",
+ "
3
\n",
+ "
of intellectual property
\n",
+ "
\n",
+ "
\n",
+ "
6
\n",
+ "
Licensing/royalty-based fees
\n",
+ "
\n",
+ "
\n",
+ "
9
\n",
+ "
Custom development income
\n",
+ "
\n",
+ "
\n",
+ "
12
\n",
+ "
Total
\n",
+ "
\n",
+ "
\n",
+ "
3
\n",
+ "
other (income)
\n",
+ "
\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " Region\n",
+ "0 Total revenue\n",
+ "4 Geographies\n",
+ "8 Americas\n",
+ "12 Europe/Middle East/Africa\n",
+ "16 Asia Pacific\n",
+ "0 Total revenue:\n",
+ "4 Geographies:\n",
+ "9 2009. The increase in total expense and other ...\n",
+ "25 Examples of the company's investments include:\n",
+ "29 NaN\n",
+ "33 • Industry sales skills to support Smarter Planet\n",
+ "16 Asia Pacifi c\n",
+ "3 of intellectual property\n",
+ "6 Licensing/royalty-based fees\n",
+ "9 Custom development income\n",
+ "12 Total\n",
+ "3 other (income)"
+ ]
+ },
+ "execution_count": 22,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "revenue_df[[\"Region\"]].drop_duplicates()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It looks like all the worldwide revenue figures are under some variation of \"Geographies\" or \"Total revenue\". \n",
+ "Allison uses Pandas' string matching facilities to keep the rows whose \"Region\" column contains the \n",
+ "name of the regions in the world."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "
\n",
+ "Extract Structured Information from Tables in PDF Documents\n",
+ " using IBM Watson Discovery Tooling and Text Extensions for Pandas\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Introduction\n",
+ "\n",
+ "Many organizations have valuable information hidden in tables inside human-readable documents like PDF files and web pages. Table identification and extraction technology can turn this human-readable information into a format that data science tools can import and use. Text Extensions for Pandas and Watson Discovery make this process much easier.\n",
+ "\n",
+ "In this notebook, we'll follow the journey of Allison, an analyst at an investment bank. Allison's employer has assigned her to cover several different companies, one of which is IBM. As part of her analysis, Allison wants to track IBM's revenue over time, broken down by geographical region. That detailed revenue information is all there in IBM's filings with the U.S. Securities and Exchange Commission (SEC). For example, here's IBM's 2019 annual report:\n",
+ "\n",
+ "\n",
+ "\n",
+ "Did you see the table of revenue by geography? It's here, on page 39:\n",
+ "\n",
+ "\n",
+ "\n",
+ "Here's what that table looks like close up:\n",
+ "\n",
+ "\n",
+ "\n",
+ "But this particular table only gives two years' revenue figures. Allison needs to have enough data to draw a meaningful chart of revenue over time. 10 years of annual revenue figures would be a good starting point. \n",
+ "\n",
+ "Allison has a collection of IBM annual reports going back to 2009. In total, these documents contain about 1500 pages of financial information. Hidden inside those 1500 pages are the detailed revenue figures that Allison wants. She needs to find those figures, extract them from the documents, and import them into her data science tools.\n",
+ "\n",
+ "Fortunately, Allison has [Watson Discovery](https://www.ibm.com/cloud/watson-discovery), IBM's suite of tools for managing and extracting value from collections of human-readable documents.\n",
+ "\n",
+ "The cells that follow will show how Allison uses Text Extensions for Pandas and Watson Discovery to import the detailed revenue information from her PDF documents into a Pandas DataFrame...\n",
+ "\n",
+ "\n",
+ "\n",
+ "...that she then uses to generate a chart of revenue over time:\n",
+ "\n",
+ "\n",
+ "\n",
+ "But first, let's set your environment up so that you can run Allison's code yourself.\n",
+ "\n",
+ "(If you're just reading through the precomputed outputs of this notebook, you can skip ahead to the section labeled [\"Extract Tables with Watson Discovery\"](#watson_discovery))."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Environment Setup\n",
+ "\n",
+ "This notebook requires a Python 3.7 or later environment with the following packages:\n",
+ "* The dependencies listed in the [\"requirements.txt\" file for Text Extensions for Pandas](https://github.com/CODAIT/text-extensions-for-pandas/blob/master/requirements.txt)\n",
+ "* `matplotlib`\n",
+ "* `text_extensions_for_pandas`\n",
+ "\n",
+ "You can satisfy the dependency on `text_extensions_for_pandas` in either of two ways:\n",
+ "\n",
+ "* Run `pip install text_extensions_for_pandas` before running this notebook. This command adds the library to your Python environment.\n",
+ "* Run this notebook out of your local copy of the Text Extensions for Pandas project's [source tree](https://github.com/CODAIT/text-extensions-for-pandas). In this case, the notebook will use the version of Text Extensions for Pandas in your local source tree **if the package is not installed in your Python environment**.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Core Python libraries\n",
+ "import json\n",
+ "import os\n",
+ "import sys\n",
+ "import requests\n",
+ "import glob\n",
+ "from typing import *\n",
+ "import pandas as pd\n",
+ "from matplotlib import pyplot as plt\n",
+ "from IPython.display import HTML\n",
+ "from base64 import b64encode\n",
+ "\n",
+ "# IBM libraries\n",
+ "from ibm_watson import DiscoveryV2\n",
+ "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator\n",
+ "\n",
+ "# And of course we need the text_extensions_for_pandas library itself.\n",
+ "try:\n",
+ " import text_extensions_for_pandas as tp\n",
+ "except ModuleNotFoundError as e:\n",
+ " # If we're running from within the project source tree and the parent Python\n",
+ " # environment doesn't have the text_extensions_for_pandas package, use the\n",
+ " # version in the local source tree.\n",
+ " if not os.getcwd().endswith(\"notebooks\"):\n",
+ " raise e\n",
+ " if \"..\" not in sys.path:\n",
+ " sys.path.insert(0, \"..\")\n",
+ " import text_extensions_for_pandas as tp"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "# Extract Tables with Watson Discovery\n",
+ "\n",
+ "Allison connects to the [Watson Discovery](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-install) component of her firm's [IBM Cloud Pak for Data](https://www.ibm.com/products/cloud-pak-for-data) installation on their [OpenShift](https://www.openshift.com/) cluster and uploads her stack of IBM annual reports to her project. Then she uses the Watson Discovery's [Table Understanding enrichment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables) to identify tables in the PDF documents and to extract detailed information about the cells and headers that make up each table.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "# Create a Project & Collection of Documents & Apply Table Understanding using Watson Discovery Tooling\n",
+ "This notebook shows you how Allison accomplishes her whole task end to end using the Watson Discovery tooling and python API. Allison first creates a project of type `Document Retreival` named \"TextExtensionsForPandasTableUnderstanding\", then she creates a collection named `IBM-10K` and upload the PDF documents into that collection. Before Allison can apply the table understanding to her collection, she needs to first enable the pre-trained-models option for her collection to get the tables annotated for her automatically using the pre-trained Smart Document Understanding algorithm. It would take some time for all the documents to get ingested and she can query the documents."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note: Alternatively, Allison could use the Watson Discovery's Python SDK to do the same programatically as shown in this [additional notebook](./Understand_Tables_API.ipynb#watson_discovery_create_project).\n",
+ "\n",
+ "If you're using the Watson Discovery tooling, you can follow the steps in the following video and then go ahead to the section labeled [\"Query the Project\"](#watson_discovery_qury_project). Otherwise you can simply follow the steps in [Create a Project through the Watson Discovery Service API](./Understand_Tables_API.ipynb#watson_discovery_create_project) to see how you can do the same using the Python SDK.)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "