Merge pull request #37 from lyft/merge_20190529a

Merge from master

CONTRIBUTING.md

@@ -294,7 +294,17 @@ python setup.py build_sphinx
 
 ### Flask server
 
-Make sure your machine meets the [OS dependencies](https://superset.incubator.apache.org/installation.html#os-dependencies) before following these steps.
+#### OS Dependencies
+
+Make sure your machine meets the [OS dependencies](https://superset.incubator.apache.org/installation.html#os-dependencies) before following these steps. 
+
+Developers should use a virtualenv. 
+
+```
+pip install virtualenv
+```
+
+Then proceed with:
 
 ```bash
 # Create a virtual environemnt and activate it (recommended)
@@ -304,10 +314,11 @@ source venv/bin/activate
 # Install external dependencies
 pip install -r requirements.txt
 pip install -r requirements-dev.txt
+
 # Install Superset in editable (development) mode
 pip install -e .
 
-# Create an admin user
+# Create an admin user in your metadata database
 fabmanager create-admin --app superset
 
 # Initialize the database
@@ -319,11 +330,10 @@ superset init
 # Load some data to play with
 superset load_examples
 
-# Start the Flask dev web server from inside the `superset` dir at port 8088
+# Start the Flask dev web server from inside your virtualenv.
 # Note that your page may not have css at this point.
 # See instructions below how to build the front-end assets.
-cd superset
-FLASK_ENV=development flask run -p 8088 --with-threads --reload --debugger
+FLASK_ENV=development superset run -p 8088 --with-threads --reload --debugger
 ```
 
 #### Logging to the browser console
@@ -355,7 +365,14 @@ app.logger.info(form_data)
 
 Frontend assets (JavaScript, CSS, and images) must be compiled in order to properly display the web UI. The `superset/assets` directory contains all NPM-managed front end assets. Note that there are additional frontend assets bundled with Flask-Appbuilder (e.g. jQuery and bootstrap); these are not managed by NPM, and may be phased out in the future.
 
-First, be sure you are using recent versions of NodeJS and npm. Using [nvm](https://github.com/creationix/nvm) to manage them is recommended.
+#### nvm and node
+
+First, be sure you are using recent versions of NodeJS and npm. Using [nvm](https://github.com/creationix/nvm) to manage them is recommended. Check the docs at the link to be sure, but at the time of writing the following would install nvm and node:
+
+```bash
+curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh | bash
+nvm install node
+```
 
 #### Prerequisite
 
@@ -396,6 +413,12 @@ npm run dev
 npm run prod
 ```
 
+If you run this service from somewhere other than your local machine, you may need to add hostname value to webpack.config.js at .devServer.public specifying the endpoint at which you will access the app. For example: myhost:9001. For convenience you may want to install webpack, webpack-cli and webpack-dev-server globally so that you can run them directly:
+
+```bash
+npm install --global webpack webpack-cli webpack-dev-server
+```
+
 #### Updating NPM packages
 
 Use npm in the prescribed way, making sure that
@@ -517,14 +540,15 @@ superset db upgrade
 superset init
 superset load_test_users
 superset load_examples
-superset runserver
+superset run --port 8081
 ```
 
 Run Cypress tests:
 
 ```bash
-cd /superset/superset/assets
+cd superset/assets
 npm run build
+npm run install-cypress
 npm run cypress run
 
 # run tests from a specific file
@@ -534,6 +558,8 @@ npm run cypress run -- --spec cypress/integration/explore/link.test.js
 npm run cypress run -- --spec cypress/integration/dashboard/index.test.js --config video=true
 ```
 
+See [`superset/assets/cypress_build.sh`](https://github.com/apache/incubator-superset/blob/master/superset/assets/cypress_build.sh).
+
 ## Translating
 
 We use [Babel](http://babel.pocoo.org/en/latest/) to translate Superset. In Python files, we import the magic `_` function using:

README.md

@@ -171,6 +171,7 @@ the world know they are using Superset. Join our growing community!
  1. [Airbnb](https://github.com/airbnb)
  1. [Airboxlab](https://foobot.io)
  1. [Aktia Bank plc](https://www.aktia.com)
+ 1. [American Express](https://www.americanexpress.com)
  1. [Amino](https://amino.com)
  1. [Apollo GraphQL](https://www.apollographql.com/)
  1. [Ascendica Development](http://ascendicadevelopment.com)

contrib/docker/README.md

@@ -0,0 +1,58 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Getting Start with Superset using Docker
+
+Docker is an easy way to get stated with Superset.
+
+## Initializing Database
+
+To initialize the database with a user and example charts, dashboards and datasets run:
+
+```bash
+SUPERSET_LOAD_EXAMPLES=yes docker-compose run --rm superset ./docker-init.sh
+```
+
+This may take a minute.
+
+## Normal Operation
+
+To run the container, simply run:
+
+```bash
+docker-compose up
+```
+
+After several minutes for superset initialization to finish, you can open a browser and view [`http://localhost:8088`](http://localhost:8088) 
+to start your journey.
+
+## Developing
+
+While running, the container server will reload on modification of the superset python and javascript source code.
+Don't forget to reload the page to take the new frontend into account though.
+
+## Production
+
+It is also possible to run Superset in non-development mode: in the `docker-compose.yml` file remove
+the volumes needed for development and change the variable `SUPERSET_ENV` to `production`.
+
+## Resource Constraints
+
+If you are attempting to build on a Mac and it exits with 137 you need to increase your docker resources.
+OSX instructions: https://docs.docker.com/docker-for-mac/#advanced (Search for memory)

docs/installation.rst

@@ -91,6 +91,8 @@ OSX instructions: https://docs.docker.com/docker-for-mac/#advanced (Search for m
 
 Or if you're curious and want to install superset from bottom up, then go ahead.
 
+See also `contrib/docker/README.md <https://github.com/apache/incubator-superset/blob/master/contrib/docker/README.md>`_
+
 OS dependencies
 ---------------
 
@@ -121,7 +123,13 @@ that the required dependencies are installed: ::
     sudo yum upgrade python-setuptools
     sudo yum install gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel libsasl2-devel openldap-devel
 
-**OSX**, system python is not recommended. brew's python also ships with pip  ::
+**Mac OS X** If possible, you should upgrade to the latest version of OS X as issues are more likely to be resolved for that version. 
+You *will likely need* the latest version of XCode available for your installed version of OS X. You should also install
+the XCode command line tools: ::
+
+    xcode-select --install
+
+System python is not recommended. Homebrew's python also ships with pip: ::
 
     brew install pkg-config libffi openssl python
     env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography==2.4.2
@@ -184,8 +192,7 @@ Follow these few simple steps to install Superset.::
     superset init
 
     # To start a development web server on port 8088, use -p to bind to another port
-    flask run -p 8080 --with-threads --reload --debugger
-
+    superset run -p 8080 --with-threads --reload --debugger
 
 After installation, you should be able to point your browser to the right
 hostname:port `http://localhost:8088 <http://localhost:8088>`_, login using
@@ -219,10 +226,8 @@ Refer to the
 `Gunicorn documentation <https://docs.gunicorn.org/en/stable/design.html>`_
 for more information.
 
-Note that *gunicorn* does not
-work on Windows so the `superset runserver` command is not expected to work
-in that context. Also, note that the development web
-server (`superset runserver -d`) is not intended for production use.
+Note that the development web
+server (`superset run` or `flask run`) is not intended for production use.
 
 If not using gunicorn, you may want to disable the use of flask-compress
 by setting `ENABLE_FLASK_COMPRESS = False` in your `superset_config.py`
@@ -387,6 +392,12 @@ Here's a list of some of the recommended packages.
 |  Pinot        | ``pip install pinotdb``             | ``pinot+http://controller:5436/``               |
 |               |                                     | ``query?server=http://controller:5983/``        |
 +---------------+-------------------------------------+-------------------------------------------------+
+|  Apache Drill |                                     | For the REST API:``                             |
+|               |                                     | ``drill+sadrill://``                            |
+|               |                                     | For JDBC                                        |
+|               |                                     | ``drill+jdbc://``                               |
++---------------+-------------------------------------+-------------------------------------------------+
+
 
 Note that many other databases are supported, the main criteria being the
 existence of a functional SqlAlchemy dialect and Python driver. Googling
@@ -444,6 +455,31 @@ Required environment variables: ::
 
 See `Teradata SQLAlchemy <https://github.com/Teradata/sqlalchemy-teradata>`_.
 
+Apache Drill
+---------
+At the time of writing, the SQLAlchemy Dialect is not available on pypi and must be downloaded here:
+`SQLAlchemy Drill <https://github.com/JohnOmernik/sqlalchemy-drill>`_
+
+Alternatively, you can install it completely from the command line as follows: ::
+
+    git clone https://github.com/JohnOmernik/sqlalchemy-drill
+    cd sqlalchemy-drill
+    python3 setup.py install
+
+Once that is done, you can connect to Drill in two ways, either via the REST interface or by JDBC.  If you are connecting via JDBC, you must have the
+Drill JDBC Driver installed.
+
+The basic connection string for Drill looks like this ::
+
+    drill+sadrill://{username}:{password}@{host}:{port}/{storage_plugin}?use_ssl=True
+
+If you are using JDBC to connect to Drill, the connection string looks like this: ::
+
+    drill+jdbc://{username}:{password}@{host}:{port}/{storage_plugin}
+
+For a complete tutorial about how to use Apache Drill with Superset, see this tutorial:
+`Visualize Anything with Superset and Drill <http://thedataist.com/visualize-anything-with-superset-and-drill/>`_
+
 Caching
 -------
 
@@ -905,7 +941,13 @@ To allow scheduled queries, add the following to your `config.py`:
                     # this is where the error message is shown
                     'container': 'end_date',
                 },
-            ]
+            ],
+            # link to the scheduler; this example links to an Airflow pipeline
+            # that uses the query id and the output table as its name
+            'linkback': (
+                'https://airflow.example.com/admin/airflow/tree?'
+                'dag_id=query_${id}_${extra_json.schedule_info.output_table}'
+            ),
         },
     }
 

requirements-dev.txt

@@ -16,22 +16,22 @@
 #
 coverage==4.5.3
 flake8-commas==2.0.0
-flake8-import-order==0.18
+flake8-import-order==0.18.1
 flake8-mypy==17.8.0
-flake8-quotes==1.0.0
-flake8==3.6.0
-flask-cors==3.0.6
-ipdb==0.11
+flake8-quotes==2.0.1
+flake8==3.7.7
+flask-cors==3.0.7
+ipdb==0.12
 mypy==0.670
-mysqlclient==1.3.13
+mysqlclient==1.4.2.post1
 nose==1.3.7
-pip-tools==3.5.0
+pip-tools==3.7.0
 psycopg2-binary==2.7.5
-pycodestyle==2.4.0
+pycodestyle==2.5.0
 pyhive==0.6.1
 pylint==1.9.2
 python-dotenv==0.10.1
 redis==2.10.6
 statsd==3.3.0
 thrift==0.11.0
-tox==3.5.3
+tox==3.11.1

requirements.txt

@@ -2,9 +2,8 @@
 # This file is autogenerated by pip-compile
 # To update, run:
 #
-#    pip-compile --output-file requirements.txt setup.py
+#    pip-compile --output-file=requirements.txt setup.py
 #
-
 alembic==1.0.0            # via flask-migrate
 amqp==2.3.2               # via kombu
 apispec[yaml]==1.2.0      # via flask-appbuilder
@@ -43,7 +42,7 @@ humanize==0.5.1
 idna==2.6
 isodate==0.6.0
 itsdangerous==0.24        # via flask
-jinja2==2.10              # via flask, flask-babel
+jinja2==2.10.1            # via flask, flask-babel
 jsonschema==3.0.1         # via flask-appbuilder
 kombu==4.2.1              # via celery
 mako==1.0.7               # via alembic
@@ -64,20 +63,21 @@ pydruid==0.5.2
 pyjwt==1.7.1              # via flask-appbuilder, flask-jwt-extended
 pyrsistent==0.14.11       # via jsonschema
 python-dateutil==2.6.1
+python-dotenv==0.10.1
 python-editor==1.0.3      # via alembic
 python-geohash==0.8.5
 python3-openid==3.1.0     # via flask-openid
 pytz==2018.5              # via babel, celery, pandas
-pyyaml==3.13
-requests==2.20.0
+pyyaml==5.1
+requests==2.22.0
 retry==0.9.2
 selenium==3.141.0
 simplejson==3.15.0
-sqlalchemy-utils==0.32.21
+six==1.11.0               # via bleach, cryptography, flask-jwt-extended, flask-talisman, isodate, jsonschema, pathlib2, polyline, prison, pydruid, pyrsistent, python-dateutil, sqlalchemy-utils, wtforms-json
+sqlalchemy-utils==0.33.11
 sqlalchemy==1.3.1
 sqlparse==0.2.4
-unicodecsv==0.14.1
-urllib3==1.22             # via requests, selenium
+urllib3==1.24.3           # via requests, selenium
 vine==1.1.4               # via amqp
 webencodings==0.5.1       # via bleach
 werkzeug==0.14.1          # via flask, flask-jwt-extended

setup.py

@@ -95,16 +95,16 @@ def get_git_sha():
         'polyline',
         'pydruid>=0.5.2',
         'python-dateutil',
+        'python-dotenv',
         'python-geohash',
-        'pyyaml>=3.13',
-        'requests>=2.20.0',
+        'pyyaml>=5.1',
+        'requests>=2.22.0',
         'retry>=0.9.2',
         'selenium>=3.141.0',
         'simplejson>=3.15.0',
         'sqlalchemy>=1.3.1,<2.0',
-        'sqlalchemy-utils',
+        'sqlalchemy-utils>=0.33.2',
         'sqlparse',
-        'unicodecsv',
         'wtforms-json',
     ],
     extras_require={

superset/__init__.py

@@ -230,7 +230,9 @@ def is_feature_enabled(feature):
 if conf.get('ENABLE_FLASK_COMPRESS'):
     Compress(app)
 
-Talisman(app, content_security_policy=None)
+if app.config['TALISMAN_ENABLED']:
+    talisman_config = app.config.get('TALISMAN_CONFIG')
+    Talisman(app, **talisman_config)
 
 # Hook that provides administrators a handle on the Flask APP
 # after initialization

superset/assets/cypress/integration/dashboard/save.js

@@ -20,6 +20,8 @@ import readResponseBlob from '../../utils/readResponseBlob';
 import { WORLD_HEALTH_DASHBOARD } from './dashboard.helper';
 
 export default () => describe('save', () => {
+  Cypress.config('chromeWebSecurity', false);
+
   let dashboardId;
   let boxplotChartId;