notes_enterprise_data_analysis.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:6a6e0d600a2daae8df88091263a38bd4363970485f57ca204dfa1b8bcc541217"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Enterprise Data Analysis and Visualization Study"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Source: http://vis.stanford.edu/files/2012-EnterpriseAnalysisInterviews-VAST.pdf\n",
      "\n",
      "###Summary\n",
      "* Companies use data analysis to model customer engagement, improve operations, productions, combat fraud, and make business decisions\n",
      "* Highlight pain points, challenges, barriers to adoption visual tools\n",
      "* Important visual design implications, opportunity for visual analytic tools to improve quality of analysis"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Types of Data Analysts\n",
      "* Three types: Hacker, Scripter, Application users\n",
      "* 5 level tasks: discovery, wrangling, profiling, modeling, and reporting\n",
      "* Hackers - Use python and SQL, good at SQL, programming, and stats\n",
      "* Able to acquire new data source outside of organization data warehouse (APIs)\n",
      "* Good knowledge of query languages (SQL), run jobs at scale\n",
      "* Some of them even build and maintain organization's central data warehouse and database engines\n",
      "* Most common activities:\n",
      "    * Locating Data\n",
      "    * Data integration, data wrangling\n",
      "* Tools: Database (SQL), Scripting (Python), Modeling (R - can use Python)\n",
      "* Spend a lot time in early-stage analytic activities prior to modeling\n",
      "* Visualization tools: Tableau, Excel, Powerpoint, D3, and Raphael"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Distributed Data\n",
      "* Analysts in large organization work with data stored in various repositories\n",
      "* Work with at least 3 different formats of data\n",
      "    * Spreadsheet in shared file system (excel)\n",
      "    * Account data stored in relational database (SQL)\n",
      "    * Log files in HDFS (Hadoop distributed file system)\n",
      "* Involve integrating data from multiple sources\n",
      "* Much data work in different warehouses, in different schemas\n",
      "* They work on this integration themselves (hackers)\n",
      "\n",
      "###Analysis\n",
      "* Serve many different departments (marketing, business development, sales, operations, and design teams)\n",
      "* Translate high-level business questions to low-level analytic task\n",
      "* Typically generate reports in the form of summary statistics, charts or recommendations\n",
      "* Share static reports using powerpoints etc\n",
      "* Results shared via email, shared file system, or group meetings\n",
      "\n",
      "###Collaboration\n",
      "* Most analysts report they rarely interact with other analysts\n",
      "* \"Working on a team is the exception in my experience\"\n",
      "* 4 types of resources: data, scripts, results, and documentation\n",
      "* Analysts typically do not share scripts with each other\n",
      "* Difference between product code and analysis code\n",
      "* Analysts often share results of their findings\n",
      "* Most reports are static are do not allow others to modify\n",
      "* Why analysts do not share code?\n",
      "    * Due to diversity of languages the analysts used\n",
      "    * e.g Mathematica, Perl, Python etc\n",
      "    * Writing the script themselves is less time-consuming than reading someone else code\n",
      "* Be wary of reusing code, as if you blindly reuse it, you might miss obvious things which are important\n",
      "* Analysts has general attitude that code and data were experimental, ad-hoc, throwaway. So they spend less time writing modular, reusable, and parametrizable code. Rarely includes documentation\n",
      "* A lot of work did not validate useful hypothesis, end up discarding the code"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "###Challenges in Analysis Process\n",
      "* Discover\n",
      "    * Get data to complete an analysis (data set online, getting from MySQL database, or getting an excel spreadsheet)\n",
      "    * Finding relevant data across different databases and files is very timing consuming\n",
      "    * Hard to know where the data is -  no proper central wiki. Best way is to ask people.\n",
      "> It is really hard to know where the data is. We have all the\n",
      "data, but there is no huge schema where we can say this\n",
      "data is here and this variable is there. It may be written\n",
      "but the wiki is very stale: pointers don\u2019t point to the right\n",
      "place and it changes really fast. The best thing you can\n",
      "learn working here is whom to ask, because in their head\n",
      "a lot of people know a lot of stuff. It\u2019s more like folklore.\n",
      "Knowledge is transmitted as you join.\n",
      "\n",
      "* Wrangle\n",
      "    * Put the data into a desired format\n",
      "    * Parsing fields from log files, integrating data from multiple sources into single file\n",
      "    * Need to manipulate the date before using\n",
      "    * Parsing text files, manipulating data layout, integrating multiple data sources\n",
      "    * Often time consuming and tedious\n",
      "> I spend more than half of my time integrating, cleansing\n",
      "and transforming data without doing any actual analysis.\n",
      "Most of the time I\u2019m lucky if I get to do any analysis. Most\n",
      "of the time once you transform the data you just do an average...\n",
      "the insights can be scarily obvious. It\u2019s fun when\n",
      "you get to do something somewhat analytical.\n",
      "    * Big issue: processing semi-structed data\n",
      "    * Common example: Parsing log files - requires writing complex regular expression to extract data into elevant fields\n",
      "    * Difficulty: integrating data from multiple sources\n",
      "    * Have to migrate all data sets into same data process framework\n",
      "    * Spend most of her time integrating data together\n",
      "    * Funnel analysis: sequence of actions users took\n",
      "> One analyst at a web\n",
      "company investigated the sequence of actions users took before converting\n",
      "to a paying customer, upgrading their accounts, or canceling\n",
      "their accounts. The source data set was a log of user activities on the\n",
      "website, with each entry corresponding to a single activity by a single\n",
      "user. The analysts needed to group activities not only by user, but also\n",
      "by event time, where the time was conditional on other events in the\n",
      "log (i.e., prior to closest conversion). These types of queries in SQL\n",
      "often involve nested subqueries. Similar subqueries are necessary to\n",
      "write filters such as \u201cdelete all users who never upgraded their account.\u201d\n",
      "\n",
      "\n",
      "* Profile\n",
      "    * Verify quality of data and suitability for analysis task - inspecting outliers etc\n",
      "    * Missing data, erroneous values, outliers\n",
      "    * Non-heterogeneous data in a column (different data types)\n",
      "    * Use visualization and statistical routines to detect errors in data\n",
      "    * Some prefer to use statistical routines instead of visualization to find outliers\n",
      "    \n",
      "* Model\n",
      "    * Model data for prediction or summarization (sumamry stats, regression models, clustering and classification)\n",
      "    * Feature Selection: Biggest difficulty in constructing a model\n",
      "> In practice right now the biggest differentiator is feature\n",
      "selection: knowing what columns to pay attention to and\n",
      "how to sensibly transform them. Do you take the log of\n",
      "these, do you combine these two? A lot of work is just\n",
      "finding what the units of the columns should be.\n",
      "    * Scale: existing tool did not scale with size of their data sets\n",
      "    * Run distributed jobs over multiple machines\n",
      "    * Limited by type of analysis they could run because useful models, algorithms do not have available parallelized implementations\n",
      "> it is difficult to \u201ctake powerful\n",
      "algorithms that work on medium data and make them pluggable\n",
      "in the big data stack.\u201d\n",
      "    * Can use sampling but hard to do without introducing bias\n",
      ">[G]raphical representation is at best two or three dimensional.\n",
      "Three dimensions won\u2019t tell me very much about\n",
      "how 300 variables interact.\n",
      "\n",
      "\n",
      "* Report\n",
      "    * Prepare report of analysis\n",
      "    * Most common challenges: communicating assumptions and building interactive reports\n",
      "    * Reports too inflexible and did not allow interactive verification\n",
      "    * Most tedious part: moving data between tools and warehouses\n",
      "    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Future Trends\n",
      "* Availability of Public Data\n",
      "    * Ingesting public data: crawl and scrape websites, web APIs\n",
      "    * Release data in formats which are hard to process (PDF)\n",
      ">An analyst at a large hedge fund noted that their organization\u2019s ability\n",
      "to make use of publicly-available but poorly-structured data was their\n",
      "primary advantage over competitors.\n",
      "    * Increasing number of data-marts (InfoChimps.org) - make public data more accessible\n",
      "\n",
      "* Rise of Hadoop\n",
      "    * Market for Hadoop softward will increase tremendously\n",
      "* Growing demand for Hacker Analysts\n",
      "    * Analysts need to learn to program - IT is for shipping products, not help analysts run experiments\n",
      "    * Increasing scale of data requires organization to perform in-database analytics\n",
      "    * Need to adapt by writing complex SQL or Map-Reduce code\n",
      "> Diversity is pretty important. A generalist is more valuable\n",
      "than a specialist. A specialist isn\u2019t fluid enough. We look\n",
      "for pretty broad skills and data passion. If you are passionate\n",
      "about it you\u2019ll jump into whatever tool you need to. If\n",
      "it\u2019s in X, I\u2019ll go jump in X.\n",
      "* Analysis teams are growing\n"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}