Rebase from master (#6660)

* [SIP-3] Scheduled email reports for Slices / Dashboards (#5294) * [scheduled reports] Add support for scheduled reports * Scheduled email reports for slice and dashboard visualization (attachment or inline) * Scheduled email reports for slice data (CSV attachment on inline table) * Each schedule has a list of recipients (all of them can receive a single mail, or separate mails) * All outgoing mails can have a mandatory bcc - for audit purposes. * Each dashboard/slice can have multiple schedules. In addition, this PR also makes a few minor improvements to the celery infrastructure. * Create a common celery app * Added more celery annotations for the tasks * Introduced celery beat * Update docs about concurrency / pools * [scheduled reports] - Debug mode for scheduled emails * [scheduled reports] - Ability to send test mails * [scheduled reports] - Test email functionality - minor improvements * [scheduled reports] - Rebase with master. Minor fixes * [scheduled reports] - Add warning messages * [scheduled reports] - flake8 * [scheduled reports] - fix rebase * [scheduled reports] - fix rebase * [scheduled reports] - fix flake8 * [scheduled reports] Rebase in prep for merge * Fixed alembic tree after rebase * Updated requirements to latest version of packages (and tested) * Removed py2 stuff * [scheduled reports] - fix flake8 * [scheduled reports] - address review comments * [scheduled reports] - rebase with master * Typo: Fixed link (#6087) * Update contributing.md with latest local dev instructions (#6513) * Fix malformed table in docs/visualization.rst (#6409) * Update requests version (#6510) * Update requests version * Run pip-compile --output-file requirements.txt * [warm] Enforcing consistent form-data (#6531) * Avoid resetting margin to 0 (#6536) * Avoid resetting margin to 0 * Fixing margin for pie chart * Refactor teradata to new time_grain_functions spec (#6539) * Refactor teradata to new time_grain_functions spec * Add test for time_grain_functions * Add docker files to gitignore (#6507) * add docker files to gitignore * Update .gitignore * adding in a dependency version to fix an error with Flask CLI (#6547) * Fix string value displaying NaN (#6534) * Documentation Correction to use http.server for Python3 (#6549) * Remove note about snowflake-sqlalchemy stable version regression (#6398) * filter_values documentation fix (#5977) * Pass security manager to QUERY_LOGGER (#6548) * Pass security manager to log_query * Fix lint * Increase size of column `name` in table `ab_view_meu` (#6532) * Create migration script * Use batch operation for ALTER COLUMN * fix addr_str format bug (#6551) * Fix deck.gl Polygon not show (#6545) * Fix deck.gl Polygon not show * Replace "//" to "// " * Secure unsecured views and prevent regressions (#6553) * Secure views and prevent regressions * Force POST on shortner * Fix tests * [cosmetic] add css no-wrap on CRUD's last modified (#6522) * [RfC] Fix URL too long (#6519) When a URL gets too long (usually happens with controls that allow free form text), it creates an array of bugs. * bug when creating a short URL, because the POST request's referrer field is too long * a bug when saving a chart (when the modal shows up) also because of the referrer being too long Some work has been done in the past to mitigate this, and I'm unclear if it's some sort of regression because of the Flask upgrade or some browser change, or whether these bugs have always been around. This is a request for comment on an approach that works. Let me know if you can think of better ideas as to how to manage this. My current solution looks for 8000+ characters URLs and shrinks them to only `/superset/explore/?URL_IS_TOO_LONG_TO_SHARE&form_data=...` and we only keep the formData keys for `datasource` and `viz_type`. Not super elegant but does the trick. * minor, auto tune debug mode when use flask_env (#6550) * minor, auto tune debug mode when use flask_env * update changes * Make owner a m2m relation on datasources (#6544) * Make owner a m2m relation on datasources * Fix pylint * Make migration work in mysql & sqlite * Fix multiple db_migrations heads (#6560) * [SIP-14] remove dependency on yarn in favor of npm (#6541) * [SIP-14] remove dependency on yarn in favor of npm Find more details here #6217 * Add core-js to dev deps * lint * fix cypress * break down cypress_build.sh * fix cypress * Use 'npm ci' in place of 'npm install' * Bump npm req to >=6.5.0 * Enforcing npm@'>=6.5.0' * Using latest lts of nvm * Make cache settings global * remove uneeded 'npm run test' as 'npm run cover' takes care of that * Prefix with 'time' * Trying to upgrade babel-eslint instead of downgrading * upgrading babel-polyfill to '@babel/polyfill' * Moving to babel 7 * remove unused packages * Make boto3/botocore installation optional (#6540) * Make boto3 installation optional * pylinting * [SQL Lab] Allow running multiple statements (#6112) * Allow running multiple statements from SQL Lab * fix tests * More tests * merge heads * fix heads * Adding missing slash (#6567) * [refactor] moving some datasource-related code to the frontend (#5769) * [refactor] moving some datasource-related code to the frontend * fix js tests * fix tests * fix test * json_iso_dttm_ser use for TableViz (#6563) * fix #6530 and add some other chinese translation (#6569) * Fix: updated required cryptography version to 2.4.2 to resolve #6509 (#6516) * Fix: updated required cryptography version to 2.4.2 to resolve #6509 * Fix: updated cryptography version to 2.4.2 to resolve #6509 * Fix: updated cryptography version to 2.4.2 to resolve #6509 * Revert "Fix: updated required cryptography version to 2.4.2 to resolve #6509" This reverts commit b57b08c. * Fix: updated cryptography version to 2.4.2 * Fix multi/dual when no right y axis (#6571) * [design] use angle icons instead of carets for expandable panels (#6564) * [design] use angle icons instead of carets for expandable panels Also moving to the right to conform to the material design specs * Fix cypress test * fixing issue #6572 with Oracle date handling (#6580) * fix Oracle engine specs for dates issue #6572 * fix Oracle engine specs for dates issue #6572 * fix Oracle engine specs for dates issue #6572, removing comment * ng a trailing space * Change margin for slice description in charts for the dashboard view (#6575) Changed margin for the slices description in dashboard view, previously the margin was 5px for all the sides but for improve the view just apply margin for top and bottom. * [bugfix] moving from reactable to reactable-arc fork (#6576) * Using batch_op in db migration 0b1f1ab473c0 (#6581) As I needed to downgrade from db migration 0b1f1ab473c0, I realized I needed to use batch_op against SQLLite. * Finish move to babel 7 (#6573) * Finish move to babel 7 * Bump jest to 23.6.0 * Address comments * Enhance Docker (#6504) Refactored deprecated functionalities: - Used 'celery worker' command instead of 'superset worker' which is deprecated since 0.26.0 - Used 'gunicorn' command instead of 'superset runserver' * add chinese translate (#6592) * Fix bug: some word not translate in js. (#6598) delete the 'null,' in messages.json, so that word can be translate in js. * shift labels down along y-axis (#6596) * Fix 'Uncaught TypeError: Cannot read property 'value' of undefined' #6556 (#6574) closes #6556 * added get_data ; * added cache logic * added cache logic * added cache key logic * add missing part * revised the metric part * added to_dict * fix lint * fix lint * fix lint * fix lint * fix lint * fix flake8 * fix invalid name * fix invalid name * fix invalid name * enable worldcloid * working version * rebase * fixed expore link issue * add endpoint * added migration script WIP * fixed lint * remove migration script * fix flake8 * fix tesT * fix tesT * fix tesT * fix typo * fixed metric format * fixed metric format * fixed metric format * fixed tests * fixed test * fixed druid test * fixed more tests * fixed more tests * fix test for viz * fixed js tests * fixed ts test * fixed ts test * fixed cypress test * fixed the metric format in filterViz
apache · Jan 11, 2019 · 36be03b · 36be03b
1 parent 22d04d4
commit 36be03b
Show file tree

Hide file tree

Showing 113 changed files with 30,658 additions and 14,237 deletions.
diff --git a/.gitignore b/.gitignore
@@ -44,3 +44,19 @@ yarn-error.log
 *.iml
 venv
 @eaDir/
+
+# Docker
+./Dockerfile
+./docker-build.sh
+./docker-compose.yml
+./docker-entrypoint.sh
+./docker-init.sh
+
+# Test data
+celery_results.sqlite
+celerybeat-schedule
+celerydb.sqlite
+celerybeat.pid
+geckodriver.log
+ghostdriver.log
+testCSV.csv
diff --git a/.travis.yml b/.travis.yml
@@ -5,42 +5,24 @@ jobs:
     - language: python
       python: 3.6
       env: TOXENV=cypress-dashboard
-      cache:
-        pip: true
-        yarn: true
-        directories:
-          - ~/.npm
-          - ~/.cache
       services:
         - redis-server
       before_install:
-        - nvm install 8.9
+        - nvm install 10.14.2
     - language: python
       python: 3.6
       env: TOXENV=cypress-explore
-      cache:
-        pip: true
-        yarn: true
-        directories:
-          - ~/.npm
-          - ~/.cache
       services:
         - redis-server
       before_install:
-        - nvm install 8.9
+        - nvm install 10.14.2
     - language: python
       python: 3.6
       env: TOXENV=cypress-sqllab
-      cache:
-        pip: true
-        yarn: true
-        directories:
-          - ~/.npm
-          - ~/.cache
       services:
         - redis-server
       before_install:
-        - nvm install 8.9
+        - nvm install 10.14.2
     - language: python
       python: 3.6
       env: TOXENV=py36-mysql
@@ -51,17 +33,10 @@ jobs:
         - mysql -u root -e "DROP DATABASE IF EXISTS superset; CREATE DATABASE superset DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci"
         - mysql -u root -e "CREATE USER 'mysqluser'@'localhost' IDENTIFIED BY 'mysqluserpassword';"
         - mysql -u root -e "GRANT ALL ON superset.* TO 'mysqluser'@'localhost';"
-    - language: node_js
-      node_js: 8.9
-      cache:
-        yarn: true
+    - language: python
+      env: TOXENV=javascript
       before_install:
-        - cd superset/assets
-      install:
-        - yarn install --frozen-lockfile
-      script:
-        - npm run lint
-        - npm run cover
+        - nvm install 10.14.2
     - language: python
       python: 3.6
       env: TOXENV=flake8
@@ -89,3 +64,8 @@ script:
   - tox
 after_success:
   - codecov
+cache:
+  pip: true
+  directories:
+    - ~/.npm
+    - ~/.cache
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -98,9 +98,9 @@ meets these guidelines:
 7.  If you are asked to update your pull request with some changes there's
     no need to create a new one. Push your changes to the same branch.
 
-## Local development
+## Setup Local Environment for Development
 
-First, [fork the repository on GitHub](https://help.github.com/articles/about-forks/), then clone it. You can clone the main repository directly instead, but you won't be able to send pull requests.
+First, [fork the repository on GitHub](https://help.github.com/articles/about-forks/), then clone it. You can clone the main repository directly, but you won't be able to send pull requests.
 
 ```bash
 git clone git@github.com:your-username/incubator-superset.git
@@ -153,7 +153,8 @@ and start a simple web server so we can check out the docs in a browser:
 
 ```bash
 cd docs/_build/html
-python -m SimpleHTTPServer
+python -m http.server # Python2 users should use SimpleHTTPServer
+
 ```
 
 This will start a small Python web server listening on port 8000. Point your
@@ -222,17 +223,19 @@ superset init
 # Load some data to play with
 superset load_examples
 
-# Start the Flask dev web server from inside the `superset` dir (but see below for frontend asset compilation)
+# Start the Flask dev web server from inside the `superset` dir at port 8088
+# Note that your page may not have css at this point.
+# See instructions below how to build the front-end assets.
 cd superset
-flask run -p 8080 --with-threads --reload --debugger
+FLASK_ENV=development flask run -p 8088 --with-threads --reload --debugger
 ```
 
 #### Logging to the browser console
 
 This feature is only available on Python 3. When debugging your application, you can have the server logs sent directly to the browser console:
 
 ```bash
-superset runserver -d --console-log
+FLASK_ENV=development flask run -p 8088 --with-threads --reload --debugger --console-log
 ```
 
 You can log anything to the browser console, including objects:
@@ -243,41 +246,29 @@ app.logger.error('An exception occurred!')
 app.logger.info(form_data)
 ```
 
-### Frontend assets
+### Frontend Assets
 
 Frontend assets (JavaScript, CSS, and images) must be compiled in order to properly display the web UI. The `superset/assets` directory contains all NPM-managed front end assets. Note that there are additional frontend assets bundled with Flask-Appbuilder (e.g. jQuery and bootstrap); these are not managed by NPM, and may be phased out in the future.
 
 First, be sure you are using recent versions of NodeJS and npm. Using [nvm](https://github.com/creationix/nvm) to manage them is recommended.
 
+#### Prerequisite
+
+#### Installing Dependencies
+
 Install third-party dependencies listed in `package.json`:
 
 ```bash
 # From the root of the repository
 cd superset/assets
 
-# If needed, install yarn, a replacement for `npm install`
-npm install -g yarn
-
-# Install dependencies
-yarn
+# Install dependencies from `package-lock.json`
+npm ci
 ```
 
-Finally, to compile frontend assets, run any of the following commands.
-
-```bash
-# Start a watcher that recompiles your assets as you modify them (reload your browser to see changes)
-npm run dev
+#### Building
 
-# Compile the Javascript and CSS in production/optimized mode for official releases
-npm run prod
-
-# Copy a conf file from the frontend to the backend
-npm run sync-backend
-```
-
-#### Webpack dev server
-
-Alternatively, you can run the Webpack dev server, which runs on port 9000 and proxies non-asset requests to the Flask server on port 8088. After pointing your browser to it, updates to asset sources will be reflected in-browser without a refresh.
+You can run the Webpack dev server (in a separate terminal from Flask), which runs on port 9000 and proxies non-asset requests to the Flask server on port 8088. After pointing your browser to it, updates to asset sources will be reflected in-browser without a refresh.
 
 ```bash
 # Run the dev server
@@ -290,9 +281,24 @@ npm run dev-server -- --port=9001
 npm run dev-server -- --supersetPort=8081
 ```
 
-#### Upgrading NPM packages
+Alternatively you can use one of the following commands.
+
+```bash
+# Start a watcher that recompiles your assets as you modify them (but have to manually reload your browser to see changes.)
+npm run dev
+
+# Compile the Javascript and CSS in production/optimized mode for official releases
+npm run prod
+
+# Copy a conf file from the frontend to the backend
+npm run sync-backend
+```
 
-After adding or upgrading an NPM package by changing `package.json`, you must run `yarn install`, which will regenerate the `yarn.lock` file. Then, be sure to commit the new `yarn.lock` so that other users' builds are reproducible. See [the Yarn docs](https://yarnpkg.com/blog/2016/11/24/lockfiles-for-all/) for more information.
+#### Updating NPM packages
+
+Use npm in the prescribed way, making sure that
+`superset/assets/package-lock.json` is updated according to `npm`-prescribed
+best practices.
 
 #### Feature flags
 
@@ -309,20 +315,35 @@ export enum FeatureFlag {
 }
 ```
 
+## Linting
+
+Lint the project with:
+
+```bash
+# for python
+tox -e flake8
+
+# for javascript
+cd superset/assets
+npm ci
+npm run lint
+```
+
 ## Testing
 
-All tests are carried out in [tox](http://tox.readthedocs.io/en/latest/index.html)
-a standardized testing framework mostly for Python (though we also used it for Javascript).
+### Python Testing
+
+All python tests are carried out in [tox](http://tox.readthedocs.io/en/latest/index.html)
+a standardized testing framework.
 All python tests can be run with any of the tox [environments](http://tox.readthedocs.io/en/latest/example/basic.html#a-simple-tox-ini-default-environments), via,
 
 ```bash
 tox -e <environment>
 ```
 
-i.e.,
+For example,
 
 ```bash
-tox -e py27
 tox -e py36
 ```
 
@@ -342,17 +363,16 @@ Note that the test environment uses a temporary directory for defining the
 SQLite databases which will be cleared each time before the group of test
 commands are invoked.
 
-### JavaScript testing
+### JavaScript Testing
 
 We use [Jest](https://jestjs.io/) and [Enzyme](http://airbnb.io/enzyme/) to test Javascript. Tests can be run with:
 
 ```bash
-cd superset/assets/spec
-npm install
+cd superset/assets
 npm run test
 ```
 
-### Integration testing
+### Integration Testing
 
 We use [Cypress](https://www.cypress.io/) for integration tests. Tests can be run by `tox -e cypress`. To open Cypress and explore tests first setup and run test server:
 
@@ -373,18 +393,6 @@ npm run build
 npm run cypress run
 ```
 
-### Linting
-
-Lint the project with:
-
-```bash
-# for python
-tox -e flake8
-
-# for javascript
-tox -e eslint
-```
-
 ## Translating
 
 We use [Babel](http://babel.pocoo.org/en/latest/) to translate Superset. In Python files, we import the magic `_` function using:
@@ -399,7 +407,7 @@ At runtime, the `_` function will return the translation of the given string for
 In JavaScript, the technique is similar: we import `t` (simple translation), `tn` (translation containing a number).
 
 ```javascript
-import {t, tn } from '@superset-ui/translation';
+import { t, tn } from '@superset-ui/translation';
 ```
 
 ### Enabling language selection

diff --git a/UPDATING.md b/UPDATING.md
@@ -3,6 +3,16 @@
 This file documents any backwards-incompatible changes in Superset and
 assists people when migrating to a new version.
 
+## Superset 0.31.0
+* boto3 / botocore was removed from the dependency list. If you use s3
+as a place to store your SQL Lab result set or Hive uploads, you may
+have to rely on an alternate requirements.txt file to install those
+dependencies.
+* From 0.31.0 onwards, we recommend not using the npm package `yarn` in
+favor of good old `npm install`. While yarn should still work just fine,
+you should probably align to guarantee builds similar to the ones we
+use in testing and across the community in general.
+
 ## Superset 0.29.0
 * India was removed from the "Country Map" visualization as the geojson
   file included in the package was very large

diff --git a/contrib/docker/Dockerfile b/contrib/docker/Dockerfile
@@ -25,12 +25,6 @@ RUN apt-get install -y vim less postgresql-client redis-tools
 RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \
     && apt-get install -y nodejs
 
-# https://yarnpkg.com/lang/en/docs/install/#debian-stable
-RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
-    && echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list \
-    && apt-get update \
-    && apt-get install -y yarn
-
 WORKDIR /home/superset
 
 COPY requirements.txt .
@@ -48,11 +42,10 @@ ENV PATH=/home/superset/superset/bin:$PATH \
     PYTHONPATH=/home/superset/superset/:$PYTHONPATH
 
 RUN cd superset/assets \
-    && yarn --non-interactive --frozen-lockfile --link-duplicates \
-    && yarn run sync-backend \
-    && yarn run build \
-    && rm -rf node_modules \
-    && yarn cache clean
+    && npm ci \
+    && npm run sync-backend \
+    && npm run build \
+    && rm -rf node_modules
 
 COPY contrib/docker/docker-init.sh .
 COPY contrib/docker/docker-entrypoint.sh /entrypoint.sh

diff --git a/contrib/docker/docker-compose.yml b/contrib/docker/docker-compose.yml
@@ -7,6 +7,7 @@ services:
       - 6379:6379
     volumes:
       - redis:/data
+
   postgres:
     image: postgres:10
     restart: unless-stopped
@@ -18,6 +19,7 @@ services:
       - 5432:5432
     volumes:
       - postgres:/var/lib/postgresql/data
+
   superset:
     build:
       context: ../../
@@ -44,6 +46,7 @@ services:
       - ./superset_config.py:/home/superset/superset/superset_config.py
       # this is needed for development, remove with SUPERSET_ENV=production
       - ../../superset:/home/superset/superset
+
 volumes:
   postgres:
     external: false