Merge master into lyft/master

CONTRIBUTING.md

@@ -319,7 +319,7 @@ pip install -r requirements-dev.txt
 pip install -e .
 
 # Create an admin user in your metadata database
-fabmanager create-admin --app superset
+flask fab create-admin
 
 # Initialize the database
 superset db upgrade
@@ -336,6 +336,9 @@ superset load_examples
 FLASK_ENV=development superset run -p 8088 --with-threads --reload --debugger
 ```
 
+If you have made changes to the FAB-managed templates, which are not built the same way as the newer, React-powered front-end assets, you need to start the app without the `--with-threads` argument like so:
+`FLASK_ENV=development superset run -p 8088 --reload --debugger`
+
 #### Logging to the browser console
 
 This feature is only available on Python 3. When debugging your application, you can have the server logs sent directly to the browser console using the [ConsoleLog](https://github.com/betodealmeida/consolelog) package. You need to mutate the app, by adding the following to your `config.py` or `superset_config.py`:
@@ -594,7 +597,7 @@ LANGUAGES = {
 ### Extracting new strings for translation
 
 ```bash
-fabmanager babel-extract --target superset/translations --output superset/translations/messages.pot --config superset/translations/babel.cfg -k _ -k __ -k t -k tn -k tct
+flask fab babel-extract --target superset/translations --output superset/translations/messages.pot --config superset/translations/babel.cfg -k _ -k __ -k t -k tn -k tct
 ```
 
 You can then translate the strings gathered in files located under
@@ -607,7 +610,7 @@ For the translations to take effect:
 ```bash
 # In the case of JS translation, we need to convert the PO file into a JSON file, and we need the global download of the npm package po2json.
 npm install -g po2json
-fabmanager babel-compile --target superset/translations
+flask fab babel-compile --target superset/translations
 # Convert the en PO file into a JSON file
 po2json -d superset -f jed1.x superset/translations/en/LC_MESSAGES/messages.po superset/translations/en/LC_MESSAGES/messages.json
 ```

README.md

@@ -94,29 +94,8 @@ Superset speaks many SQL dialects through SQLAlchemy, a Python
 ORM that is compatible with
 [most common databases](https://docs.sqlalchemy.org/en/rel_1_2/core/engines.html).
 
-Superset can be used to visualize data out of most databases:
-* MySQL
-* Postgres
-* Vertica
-* Oracle
-* Microsoft SQL Server
-* SQLite
-* Greenplum
-* Firebird
-* MariaDB
-* Sybase
-* IBM DB2
-* Exasol
-* MonetDB
-* Snowflake
-* Redshift
-* Clickhouse
-* Apache Kylin
-* Apache Drill
-* Google BigQuery
-* **more!** look for the availability of a SQLAlchemy dialect for your database
-  to find out whether it will work with Superset
-
+A list of currently supported SQL databases can be found
+[here](https://superset.incubator.apache.org/#databases).
 
 Apache Druid (Incubating)!
 ------
@@ -168,6 +147,7 @@ Who uses Apache Superset (incubating)?
 Here's a list of organizations who have taken the time to send a PR to let
 the world know they are using Superset. Join our growing community!
 
+ 1. [6play](https://www.6play.fr)
  1. [AiHello](https://www.aihello.com)
  1. [Airbnb](https://github.com/airbnb)
  1. [Airboxlab](https://foobot.io)
@@ -193,6 +173,7 @@ the world know they are using Superset. Join our growing community!
  1. [Grassroot](https://www.grassrootinstitute.org/)
  1. [Hostnfly](https://www.hostnfly.com/)
  1. [HuiShouBao](http://www.huishoubao.com/)
+ 1. [Intercom](https://www.intercom.com/)
  1. [jampp](https://jampp.com/)
  1. [Konfío](http://konfio.mx)
  1. [Kuaishou](https://www.kuaishou.com/)
@@ -214,6 +195,7 @@ the world know they are using Superset. Join our growing community!
  1. [Steamroot](https://streamroot.io/)
  1. [Showmax](https://tech.showmax.com)
  1. [Tails.com](https://tails.com)
+ 1. [Tenable](https://www.tenable.com)
  1. [THEICONIC](http://theiconic.com.au/)
  1. [Tobii](http://www.tobii.com/)
  1. [Tooploox](https://www.tooploox.com/)

UPDATING.md

@@ -23,6 +23,10 @@ assists people when migrating to a new version.
 
 ## Next Version
 
+* [7667](https://github.com/apache/incubator-superset/pull/7667): a change to
+make all Unix timestamp (which by definition are in UTC) comparisons refer
+to a timestamp in UTC as opposed to local time.
+
 * [5451](https://github.com/apache/incubator-superset/pull/5451): a change
 which adds missing non-nullable fields to the `datasources` table. Depending on
 the integrity of the data, manual intervention may be required.
@@ -33,7 +37,7 @@ which adds missing non-nullable fields and uniqueness constraints to the
 manual intervention may be required.
 * `fabmanager` command line is deprecated since Flask-AppBuilder 2.0.0, use
 the new `flask fab <command>` integrated with *Flask cli*.
-* `SUPERSET_UPDATE_PERMS` environment variable was replaced by 
+* `SUPERSET_UPDATE_PERMS` environment variable was replaced by
 `FAB_UPDATE_PERMS` config boolean key. To disable automatic
 creation of permissions set `FAB_UPDATE_PERMS = False` on config.
 * [5453](https://github.com/apache/incubator-superset/pull/5453): a change

contrib/docker/docker-init.sh

@@ -18,7 +18,8 @@
 set -ex
 
 # Create an admin user (you will be prompted to set username, first and last name before setting a password)
-fabmanager create-admin --app superset
+export FLASK_APP=superset:app
+flask fab create-admin
 
 # Initialize the database
 superset db upgrade

docs/index.rst

@@ -93,6 +93,38 @@ Features
 - Integration with most SQL-speaking RDBMS through SQLAlchemy
 - Deep integration with Druid.io
 
+Databases
+---------
+
+The following RDBMS are currently suppored:
+
+- `Amazon Athena <https://aws.amazon.com/athena/>`_
+- `Amazon Redshift <https://aws.amazon.com/redshift/>`_
+- `Apache Drill <https://drill.apache.org/>`_
+- `Apache Druid <http://druid.io/>`_
+- `Apache Hive <https://hive.apache.org/>`_
+- `Apache Impala <https://impala.apache.org/>`_
+- `Apache Kylin <http://kylin.apache.org/>`_
+- `Apache Pinot <https://pinot.incubator.apache.org/>`_
+- `Apache Spark SQL <https://spark.apache.org/sql/>`_
+- `BigQuery <https://cloud.google.com/bigquery/>`_
+- `ClickHouse <https://clickhouse.yandex/>`_
+- `Google Sheets <https://www.google.com/sheets/about/>`_
+- `Greenplum <https://greenplum.org/>`_
+- `IBM Db2 <https://www.ibm.com/analytics/db2/>`_
+- `MySQL <https://www.mysql.com/>`_
+- `Oracle <https://www.oracle.com/database/>`_
+- `PostgreSQL <https://www.postgresql.org/>`_
+- `Presto <http://prestodb.github.io/>`_
+- `Snowflake <https://www.snowflake.com/>`_
+- `SQLite <https://www.sqlite.org/>`_
+- `SQL Server <https://www.microsoft.com/en-us/sql-server/>`_
+- `Teradata <https://www.teradata.com/>`_
+- `Vertica <https://www.vertica.com/>`_
+
+Other database engines with a proper DB-API driver and SQLAlchemy dialect should
+be supported as well.
+
 Screenshots
 -----------
 
@@ -135,4 +167,3 @@ Indices and tables
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
-

docs/installation.rst

@@ -123,7 +123,7 @@ that the required dependencies are installed: ::
     sudo yum upgrade python-setuptools
     sudo yum install gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel libsasl2-devel openldap-devel
 
-**Mac OS X** If possible, you should upgrade to the latest version of OS X as issues are more likely to be resolved for that version. 
+**Mac OS X** If possible, you should upgrade to the latest version of OS X as issues are more likely to be resolved for that version.
 You *will likely need* the latest version of XCode available for your installed version of OS X. You should also install
 the XCode command line tools: ::
 
@@ -346,58 +346,60 @@ connect to the databases you want to access through Superset.
 
 Here's a list of some of the recommended packages.
 
-+---------------+-------------------------------------+-------------------------------------------------+
-| database      | pypi package                        | SQLAlchemy URI prefix                           |
-+===============+=====================================+=================================================+
-|  MySQL        | ``pip install mysqlclient``         | ``mysql://``                                    |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Postgres     | ``pip install psycopg2``            | ``postgresql+psycopg2://``                      |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Presto       | ``pip install pyhive``              | ``presto://``                                   |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Hive         | ``pip install pyhive``              | ``hive://``                                     |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Oracle       | ``pip install cx_Oracle``           | ``oracle://``                                   |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  sqlite       |                                     | ``sqlite://``                                   |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Snowflake    | ``pip install snowflake-sqlalchemy``| ``snowflake://``                                |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Redshift     | ``pip install sqlalchemy-redshift`` | ``redshift+psycopg2://``                        |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  MSSQL        | ``pip install pymssql``             | ``mssql://``                                    |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Impala       | ``pip install impyla``              | ``impala://``                                   |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  SparkSQL     | ``pip install pyhive``              | ``jdbc+hive://``                                |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Greenplum    | ``pip install psycopg2``            | ``postgresql+psycopg2://``                      |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Athena       | ``pip install "PyAthenaJDBC>1.0.9"``| ``awsathena+jdbc://``                           |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Athena       | ``pip install "PyAthena>1.2.0"``    | ``awsathena+rest://``                           |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Vertica      | ``pip install                       |  ``vertica+vertica_python://``                  |
-|               | sqlalchemy-vertica-python``         |                                                 |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  ClickHouse   | ``pip install                       | ``clickhouse://``                               |
-|               | sqlalchemy-clickhouse``             |                                                 |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Kylin        | ``pip install kylinpy``             | ``kylin://``                                    |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  BigQuery     | ``pip install pybigquery``          | ``bigquery://``                                 |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Teradata     | ``pip install sqlalchemy-teradata`` | ``teradata://``                                 |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Pinot        | ``pip install pinotdb``             | ``pinot+http://controller:5436/``               |
-|               |                                     | ``query?server=http://controller:5983/``        |
-+---------------+-------------------------------------+-------------------------------------------------+
-|  Apache Drill |                                     | For the REST API:``                             |
-|               |                                     | ``drill+sadrill://``                            |
-|               |                                     | For JDBC                                        |
-|               |                                     | ``drill+jdbc://``                               |
-+---------------+-------------------------------------+-------------------------------------------------+
-
++------------------+---------------------------------------+-------------------------------------------------+
+| database         | pypi package                          | SQLAlchemy URI prefix                           |
++==================+=======================================+=================================================+
+| Amazon Athena    | ``pip install "PyAthenaJDBC>1.0.9"``  | ``awsathena+jdbc://``                           |
++------------------+---------------------------------------+-------------------------------------------------+
+| Amazon Athena    | ``pip install "PyAthena>1.2.0"``      | ``awsathena+rest://``                           |
++------------------+---------------------------------------+-------------------------------------------------+
+| Amazon Redshift  | ``pip install sqlalchemy-redshift``   | ``redshift+psycopg2://``                        |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Drill     | ``pip install sqlalchemy-drill``      | For the REST API:``                             |
+|                  |                                       | ``drill+sadrill://``                            |
+|                  |                                       | For JDBC                                        |
+|                  |                                       | ``drill+jdbc://``                               |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Druid     | ``pip install pyduid``                | ``druid://``                                    |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Hive      | ``pip install pyhive``                | ``hive://``                                     |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Impala    | ``pip install impyla``                | ``impala://``                                   |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Kylin     | ``pip install kylinpy``               | ``kylin://``                                    |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Pinot     | ``pip install pinotdb``               | ``pinot+http://CONTROLLER:5436/``               |
+|                  |                                       | ``query?server=http://CONTROLLER:5983/``        |
++------------------+---------------------------------------+-------------------------------------------------+
+| Apache Spark SQL | ``pip install pyhive``                | ``jdbc+hive://``                                |
++------------------+---------------------------------------+-------------------------------------------------+
+| BigQuery         | ``pip install pybigquery``            | ``bigquery://``                                 |
++------------------+---------------------------------------+-------------------------------------------------+
+| ClickHouse       | ``pip install sqlalchemy-clickhouse`` |                                                 |
++------------------+---------------------------------------+-------------------------------------------------+
+| Google Sheets    | ``pip install gsheetsdb``             | ``gsheets://`                                   |
++------------------+---------------------------------------+-------------------------------------------------+
+| IBM Db2          | ``pip install ibm_db_sa``             | ``db2+ibm_db://``                               |
++------------------+---------------------------------------+-------------------------------------------------+
+| MySQL            | ``pip install mysqlclient``           | ``mysql://``                                    |
++------------------+---------------------------------------+-------------------------------------------------+
+| Oracle           | ``pip install cx_Oracle``             | ``oracle://``                                   |
++------------------+---------------------------------------+-------------------------------------------------+
+| PostgreSQL       | ``pip install psycopg2``              | ``postgresql+psycopg2://``                      |
++------------------+---------------------------------------+-------------------------------------------------+
+| Presto           | ``pip install pyhive``                | ``presto://``                                   |
++------------------+---------------------------------------+-------------------------------------------------+
+| Snowflake        | ``pip install snowflake-sqlalchemy``  | ``snowflake://``                                |
++------------------+---------------------------------------+-------------------------------------------------+
+| SQLite           |                                       | ``sqlite://``                                   |
++------------------+---------------------------------------+-------------------------------------------------+
+| SQL Server       | ``pip install pymssql``               | ``mssql://``                                    |
++------------------+---------------------------------------+-------------------------------------------------+
+| Teradata         | ``pip install sqlalchemy-teradata``   | ``teradata://``                                 |
++------------------+---------------------------------------+-------------------------------------------------+
+| Vertica          | ``pip install                         |  ``vertica+vertica_python://``                  |
+|                  | sqlalchemy-vertica-python``           |                                                 |
++------------------+---------------------------------------+-------------------------------------------------+
 
 Note that many other databases are supported, the main criteria being the
 existence of a functional SqlAlchemy dialect and Python driver. Googling
@@ -691,10 +693,6 @@ users are active on the platform.
 
 Celery Tasks
 ------------
-On large analytic databases, it's common to run background jobs, reports
-and/or queries that execute for minutes or hours. In certain cases, we need
-to support long running tasks that execute beyond the typical web request's
-timeout (30-60 seconds).
 
 On large analytic databases, it's common to run queries that
 execute for minutes or hours.
@@ -952,7 +950,7 @@ To allow scheduled queries, add the following to your `config.py`:
     }
 
 This feature flag is based on [react-jsonschema-form](https://github.com/mozilla-services/react-jsonschema-form),
-and will add a button called "Schedule Query" to SQL Lab. When the button is 
+and will add a button called "Schedule Query" to SQL Lab. When the button is
 clicked, a modal will show up where the user can add the metadata required for
 scheduling the query.