Add CUSTOM_TEMPLATE_PROCESSOR config #9376
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dpgaspar
reviewed
Mar 26, 2020
dpgaspar
reviewed
Mar 26, 2020
dpgaspar
reviewed
Mar 26, 2020
craig-rueda
requested changes
Mar 26, 2020
tests/superset_test_config.py
Outdated
| return str(offset_day) | ||
|
|
||
|
|
||
| class CustomPrestoTemplateProcessor(PrestoTemplateProcessor): |
There was a problem hiding this comment.
Pull this out into its own file / test?
dandanhub
force-pushed
the
add_custom_temp_mrg
branch
from
37d1ef8
to
937e1f1
Compare
dandanhub
force-pushed
the
add_custom_temp_mrg
branch
from
937e1f1
to
d8a56f3
Compare
|
@dpgaspar @craig-rueda would you please have another review on the PR? Thanks! |
craig-rueda
requested changes
Apr 3, 2020
superset/config.py
Outdated
| # dictionary, which means the existing keys get overwritten by the content of this | ||
| # dictionary. The customized addons don't necessarily need to use jinjia templating | ||
| # language. This allows you to define custom logic to process macro template. | ||
| CUSTOM_TEMPLATE_PROCESSOR = {} # type: Dict[str, BaseTemplateProcessor] |
There was a problem hiding this comment.
Nit: naming here -- CUSTOM_TEMPLATE_PROCESSOR should be something like CUSTOM_TEMPLATE_PROCESSOR_MAP or even just CUSTOM_TEMPLATE_PROCESSORS, as there can potentially be several processors configured via this config property.
dandanhub
force-pushed
the
add_custom_temp_mrg
branch
from
d8a56f3
to
18fc322
Compare
|
@craig-rueda I addressed the comment. Would you mind having another review? |
craig-rueda
approved these changes
Apr 3, 2020
dpgaspar
approved these changes
Apr 5, 2020
|
Looks good, but some conflicts need to be solved |
dandanhub
force-pushed
the
add_custom_temp_mrg
branch
from
187fcce
to
8e63194
Compare
Codecov Report
@@ Coverage Diff @@
## master #9376 +/- ##
=======================================
Coverage 59.01% 59.01%
=======================================
Files 383 383
Lines 12191 12191
Branches 3014 3014
=======================================
Hits 7194 7194
Misses 4813 4813
Partials 184 184 Continue to review full report at Codecov.
|
|
Thanks @dpgaspar resolved the conflict. If you can help merge the PR to upstream, that would be much appreciated. |
craig-rueda
added a commit
that referenced
this pull request
Apr 22, 2020
* improve helm chart * add Apache license headers * fix: suburst chart when secondary metric is defined (#9343) * refactor: remove settooltip (#9332) * [config] Fixing GET_FEATURE_FLAGS_FUNC example (#9339) * [config] Fixing GET_FEATURE_FLAGS_FUNC example * Adding typing for GET_FEATURE_FLAGS_FUNC * Update config.py * feat: [SQLLAB] add checkbox to control autocomplete (#9338) * [SQLLAB] add checkbox to control autocomplete * autocomplete -> autocompleteEnabled * fix defaultProps * fix spec * Cal heatmap controls migration (#9355) * controls migrated * linting * migrating unique EventFlow controls (#9358) * migrating unique controls * Lint ✨ * Treemap controls migration (#9352) * Treemap * linting * feat: [explore] don't save filters inherited from a dashboard (#9340) * feat: [explore] don't save filters inherited from a dashboard When navigating to explore from a dashboard context, the current dashboard filter(s) are passed along to explore so that the context is kept. So say you're filtering on "country=Romania", in your dashboard and pivot to explore, that filter is still there and keep on exploring. Now a common issue is that you'll want to make some tweak to your chart that are unrelated to the filter, say toggling the legend off for instance, and then save it. Now you back to your dashboard and even though you started with an "all countries" dashboard, with a global filter on country, now that one chart is stuck on "Romania". Typically you notice this when filtering on something else, say "Italy" and then that one chart now has two mutually exclusive filters, and show "No data". Now, the fix is to flag the filter as "extra" (that's the not-so-good internal name we use for these inherited filters) and make it clear that that specific filter is special and won't be saved when saving the chart. * fix build * [charts] Refactor API using SIP-35 (#9329) * [charts] Refactor charts API using SIP-35 * [charts] Fix, copy pasta * [charts] simplify * Add global install of webpack and webpack-cli to docker-compose (#9310) * [dataset] columns and metrics API (nested) (#9268) * [dataset] columns and metrics API (nested) * [dataset] tests and validation * [datasets] Fix, revert list field name to database_name * Adding requirements-local.txt support (#9357) * Adding requirements-local.txt support * Reverting package-lock.json * Migrate unique Heatmap controls (#9360) * Migrate Heatmap controls * Lint * fix: don't parseFloat when the *already numeric* value ends in a decimal point (#9351) * [sip-15] Fixing typo in docstring (#9377) * [explore view] fix long query issue from Run in SQL LAB Button (#9345) * [explore view] fix long query issue from Run in SQL LAB Button * SQL Lab page needs to take the post form data, too * fix variable names * updated payload dict, rename hidden form Co-authored-by: Jesse Yang <jesse.yang@airbnb.com> * fix a typo in set prop value (#9383) * Migrating unique DirectedForce controls (#9359) * migrating controls * Linting ✨ * oops... meant to pull this out in a conflict resolution. * Update MANIFEST.in (#9261) Include superset-frontent/package.json in package because setup.py requires it * fix: [dashboard] add row padding (#9381) * Removing WordCloud controls from CONTRIBUTING.md (#9391) * [dashboard] handle markdown error (#9350) * [dashboard] handle markdown error * localize error message, fix review comments. * [cache] Cleaning up viz/cache logic (#9375) Co-authored-by: John Bodley <john.bodley@airbnb.com> * migrating controls 🎛 (#9386) * Migrating unique Table controls (#9388) * Migrating unique BoxPlot controls (#9392) * upgrade to react-bootstrap v0.33.1 (#9372) * upgrade to react-bootstrap v0.33.1 * fix test * simplify proptypes Co-Authored-By: Evan Rusackas <evan@preset.io> * simplify more proptypes Co-Authored-By: Evan Rusackas <evan@preset.io> Co-authored-by: Evan Rusackas <evan@preset.io> * migrating unique controls (#9374) * Migrating horizon controls (#9368) * migrating unique controls * linting ✨ * build: use manifest hooks for dev server proxy and fix hot reload for charts (#9333) * Use manifest hooks for dev server proxy * Rewrite dashboard/App.jsx to supress Redux error in hot reload * Update ChartRenderer to allow hot realod in Explore * Fix hot reload in dashboars as well * Revert changes to ChartRenderer.jsx Will submit in another PR. * Clean up * migrating controls (#9373) * [dataset] New, export API endpoint (#9367) * [dataset] New, export API endpoint * Fix, lint * [charts] New, bulk delete API endpoint (#9387) * [charts] New, bulk delete API endpoint * [charts] Fix, typos * [charts] Fix wrong model name * [sip-15] fix messaging (#9397) * Mirgrating unique Partition chart controls (#9385) * mirgrating controls * specific import * feat: add SSL certificate validation for Druid (#9396) * feat: add SSL certificate feature * Address comments * don't mutate extras * Address comments and add polish * Add further polish * [fix] Fixing cache key inconsistencies (#9382) Co-authored-by: John Bodley <john.bodley@airbnb.com> * add initjob conf + secret * [dependency] Fix, Bump FAB to 2.3.1 (#9413) * [mypy] Enforcing typing for charts (#9411) Co-authored-by: John Bodley <john.bodley@airbnb.com> * add missing headers * [dashboards] Fix, API update slug uniqueness refusing empty string (#9417) * [dashboards] Fix, API update slug uniqueness refusing empty string * [dashboards] tests * Build: fix hot reload for charts (#9400) * Fix hot reload for charts * Override tsc options for plugins Plugins will sometimes have their own `tsconfig.json`, which will interfere with the dev server config if configured inappropriately. This change makes sure `tsc` (and `ts-loader`) always compiles to `esnext` modules, so we don't get "Uncaught ReferenceError: exports is not defined" error. * [fix] allow force refresh for No Results chart (#9408) * [fix] dashboard filter indicator no showing single number value (#9401) * fix pagination for list views (#9425) * feat(visualization): add timeRangeFixed to Big Number w/ Trendlines (#9341) * feat: add timeRangeFixed and increase font size for Big Number chart * Update package-lock.json * [fix]some translation not work better (#9428) * Add check for SSL certificate and add form validators (#9436) * Add check for server_cert falsy and add form validators * Address comments * migrating unique controls (#9446) * Enforcing linting of LESS (#9356) * linting some LESS * enforce linting * moving font binaries out of stylesheets to fonts folder. * renaming font specimen sheet directory. * Migrating unique bigNumber(total) controls (#9440) * migrating options controls * lint ✨ * baby steps... * shared BigNumber controls file * capitalization matters * Migrating unique NVD3 viz controls (#9445) * donut control * labels outside * linting * x_log_scale, pie_label_type * linting * line_charts, line_charts_2 * marker_labels * marker_line_labels, marker_lines, markers * prefix_metric_with_slice_name, range_labels, ranges * stacked_style * Revert "stacked_style" This reverts commit aae2d418b8f1950fbe32248b62e50a12fbf186bd. * stacked_style take 2... * Revert "stacked_style take 2..." This reverts commit 38b7155f50143684c92a67b2a9fa16f65c269d35. * Migrating NVD3 Area's stacked_style config (#9454) * Migrating NVD3 Area's stacked_style config * test mods * lint * nonsense change * undoing nonsense. Come on GitHub, WORK. * Eslint prefer-object-spread (#9466) * The rule. * The result * [mypy] Enforcing typing for some modules (#9416) Co-authored-by: John Bodley <john.bodley@airbnb.com> * chore: bump black to 19.10b0 and mypy to 0.770 (#9378) * Bump black to 19.10b0 * Upgrade mypy to 0.770 * Update how inline type is defined * release: Add support for ZSH in RELEASING and add tagging instructions (#9451) * Add support for ZSH in RELEASING and add tagging instructions * fail if unsupported shell * Added dremio (#9472) * [query] New, readonly API (#9403) * [query] New, readonly API * Update tests/queries/api_tests.py Co-Authored-By: Ville Brofeldt <33317356+villebro@users.noreply.github.com> * [mypy] Enforcing typing for superset.examples (#9469) Co-authored-by: John Bodley <john.bodley@airbnb.com> * Migrating shared DeckGL controls (#9455) * Missed a couple unique controls! * new shared module, with filter_nulls export. * autozoom * dimension * nixing unneeded import * stroke_width * js_columns * remaining js_* controls * cleaning up unused stuff * legend_format * exports for prior commit * legend_position * line_column * fill_color_picker * filled, stroked, stroke_color_picker * grid_size, extruded * ... more gridsize * viewport * point_radius_fixed, spatial * multiplier, line_type, line_width * reverse_long_lat * lint ✨ * [Doc] Update installation doc for Dremio (#9464) * update installation doc for Dremio Customers fail to install pyodbc properly which on linux systems requires you to install unixodbc-devel package. Update the instructions to make it clear. * fix new line * [mypy] Enforcing typing for superset.dashboards (#9418) * [mypy] Enforcing typing for superset.dashboards * Make return types equal on all commands * Make return types equal on all commands * [dashboard] address comments same return type on commands * lint * lint * [datasets] Add strict type annotation (#9437) * [datasets] Add strict type annotation * Fix refresh endpoint * Improve logic on update * Filter owners select by text input (#9337) * filter owners select by text input * use rison * fix backend owners filter logic * use fullname, not username on owners inputs * fix some tests * fixing tests * deterministic tests * appease linter * add back search by username * more comprehensive filter test * add clarifying text * formatting... * Handle empty dataframes in TableViz (#9480) TableViz fails to display empty dataframes returning an error like: "None of [Index(['project', 'count'], dtype='object')] are in the [columns]" The behavior has been observed while testing 0.36.0rc3 with Druid datasources. issue: #9468 * [sqllab] Add CUSTOM_TEMPLATE_PROCESSOR config (#9376) Co-authored-by: Dandan Shi <dshi@dropbox.com> * [tests] refactor, change datasets and charts to it's own folder (#9487) * pylint: accept specific 2 character names by default (#9460) * lint: accept 2 letter names by default * Address review comments * Remove e and d from good-names * Upgrade table chart plugin to 0.12.13 (#9493) * feat: change default time range in sql lab explore (#9486) Change from "100 years go" to "no filter". 100 years ago is basically equivalent to no filter, but has implications on how x-axis is plotted on certain charts (e.g. Big Number). * fix: add explore control tabOverride at the section level (#9495) * [query] Migrate api v1 query to new location (#9479) * [query] Migrate api v1 query to new location * Improved errors and tests * [query] nit and revert undesired change * [query] lint * [query] revert nan change * Update superset/queries/api.py Co-Authored-By: Ville Brofeldt <33317356+villebro@users.noreply.github.com> * Update tests/queries/api_tests.py Co-Authored-By: Ville Brofeldt <33317356+villebro@users.noreply.github.com> * change endpoint location to charts Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com> * Bump FAB to 2.3.2 (#9491) * [dashboards] Fix, update dashboard owners not propagating to charts o… (#9484) * [dashboards] Fix, update dashboard owners not propagating to charts owners * Add tests * Fix tests * better naming * [dashboards] New, tittle and slug OR filter (#9435) * [dashboards] New, tittle and slug OR filter * Update requirements, because of prison bump * Tests * Fix tests * Avoid like filter on empty string value * merge master brings strict typing to the table * fix: add lineWidth to Shared_DeckGL.jsx (#9496) * fix: add lineWidth to Shared_DeckGL.jsx * linting * apply changes * opps again .... license * feat: Add post processing to QueryObject (#9427) * Add post processing to QueryObject * Simplify sort signature and require explicit sort order * Add new operations and unit tests * linting * Address comments * Simplify test method names * Address comments * Linting * remove unnecessary logic * Apply strict whitelisting to all getattr calls * Add checking of rolling_type_options and add/improve docs * Add visualization flow to the CTA queries (#9370) * Visualize after running Ctas query (#4) * Visualize after running Ctas query. Stub for table viz Work on the viz Add JS functionality for Ctas viz Add test for the table viz flow Fix test Add js test for the ctas viz * Fix tests * Resolve comments * Leverate tmpSchema in the query object * Fix i18n string Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com> * Bump sqlalchemy and dremio deps (#9503) * [charts] New, custom filter for name OR description (#9492) * [charts] New, custom filter for name OR description * Improve test * docker node_modules in its own volume (#9509) * [Dashboard] new listview filters & emotion infra (#9462) * [Dashboard] listview filters to match new design * use title_or_slug filter * use ?. operator * move components to components folder * chore: auto label issues based on the template chosen (#9507) * chore: auto label issues based on the template chosen * quoted stuffs * control to turn off table bar-chart backgrounds (#9500) * control to turn off table bar-chart backgrounds * flipping enable to disable, to support existing charts * lint ✨ * Turn ON to enable (reversing logic) * bumping big number chart * ling ✨ * Another attempt to fix a viz.py bug (#9522) * Another attempt to fix a viz.py bug * Add types to new ddtm attributes * deprecate groupby controls in query_obj (#9366) * Deprecate groupby from query_obj * Fix query_object bug * Fix histogram * Remove groupby from legacy druid connector and fix first batch of unit tests * Deprecate some unnecessary tests + fix a few others * Address comments * hide SIP-38 changes behind feature flag * Break out further SIP-38 related tests * Reslove test errors * Add feature flag to QueryContext * Resolve tests and bad rebase * Backport recent changes from viz.py and fix broken DeckGL charts * Fix bad rebase * backport #9522 and address comments * [dashboard] Fix, improve test for custom filter (#9521) * [api] refactor, remove unnecessary code, using command pattern now (#9520) * Importing validators module from superset-ui (#9465) * importing... this is going to break CI, but there it is. * adding validator package. * pluralization - maybe it should be plural, but whatevs * updating validators to match new exports * removing old validators * lintage ✨ * validateNumeric -> validateNumber * using legacyValidateInteger, legacyValidateNumber * linting ✨ * removing extraneous package lock file * Fix typo in viz.py (#9533) * [Build] Add Github workflows (#9517) As an replacement to Travis CI. * Disabling recording in Cypress tests (#9535) * Disabling recording in Cypress tests * Removing --group --ci-build-id * Run CI on all pushes / PR's (#9538) * [list views] add work-break css for table layouts (#9537) * [thumbnails] API and celery task for dashboards and charts (#8947) * [copy] fix: Row Level Security get_rls_filters func SELECT statement (#9541) * fix: Row Level Security get_rls_filters func SELECT statement * More general RowLevelSecurityTests case to avoid improper ids matching * Make email parsing more robust (#9523) * Make email parsing more robust * Address comment Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com> * docs: point our README the maintained Docker image (#9536) Our README now points to a not-so-well-maintain, community-contributed Docker image. The docker image we link to here is managed and supported by actual committers under the Preset org. * Bump copyright notice (#9539) * Ditching travis config in favor of Github Actions (#9548) * Bump @superset-ui/legacy-plugin-chart-table to 0.12.14 (#9549) * [dashboard] New, add statsd metrics to the API (#9519) * [fix] warm up cache error handling (#9560) Co-authored-by: John Bodley <john.bodley@airbnb.com> * [fix] dashboard filter scope bug (#9550) * [Build] Collect frontend code coverage from Cypress tests (#9555) * build: collect code coverage from Cypress Collect frontend code coverage reports from Cypress tests and add proper tagging for all tests. * Fix bash script lint error from shellcheck * Revert Cypress to 4.3.0 to see if it fixes a failing test * chore: Migrate unique FilterBox controls from controls.jsx (#9566) * Migrate FilterBox controls from controls.jsx * Linting * chore: Add OpenAPI docs to /api/v1/chart/data EP (#9556) * Add OpenAPI docs to /api/v1/chart/data EP * Add missing fields to QueryObject, fix linting and unit test errors * Fix unit test errors * abc * Fix errors uncovered by schema validation and add unit test for invalid payload * Add schema for response * Remove unnecessary pylint disable * [pypi] Include compiled translations on Pypi pkg (#9567) * doc: Add changelog for 0.36.0 (#9570) * Migrating shared NVD3 controls to new module (#9525) * proto module * caught a missed 'freq' unique control * line_interpolation * linting * showLegend * show_controls * xAxisLabel * bottomMargin * x_ticks_layout * missed one * x_axis_format * yLogScale * y_axis_bounds * linting * nixing yarn lock * x_axis_showminmax * xAxisShowminmax control * richTooltip * linting, syntax fix * show_bar_value, bar_stacked * reduceXticks, yaxislabel * left_margin, max_bubble_size, y_axis_showminmax * show_labels * send_time_range, y_axis_2_format, show_markers, order_bars * nixing commented imports * fake controls * looking up actual controls for comparison. * adding key to test setup * controls inventory * apache junk * lint ✨ * ignore null controls * fixing goofed up spread operation for xAxisFormat config * lint ✨ * fixes for errors caused by <hr> element in filterbox controls * fixing filter controls for 'druid_time_origin', 'granularity', 'granularity_sqla', 'time_grain_sqla' * getControlsInventory -> getControlsForVizType * further renaming of chartControlsInventory - > getControlsForVizType Co-authored-by: David Aaron Suddjian <aasuddjian@gmail.com> * fix: Add deprecated fields to QueryObject schema (#9579) * fix: Add deprecated fields to QueryObject schema * linting * [mypy] Enforcing typing for superset.migrations (#9583) Co-authored-by: John Bodley <john.bodley@airbnb.com> * Re-enable the AnnotationLayerModelView read API (#9443) * Re-enable the AnnotationLayerModelView read API * Fix CI * [config] Enable dashboard bootstrap payload reduction by default (#9585) * [docs] fix, elastic.co elasticsearch product location (#9598) * [fix] Fixing issue with Jinja filter_value (#9582) Co-authored-by: John Bodley <john.bodley@airbnb.com> * [Build] moves prettier check to separate script (#9591) * [Build] moves prettier check to separate script * rename step: eslint -> lint * Add documentation build to Github Actions (#9569) * Add documentation build to Github Actions * Update requirements for documentation builds * Minor optimization - only install requirements for documentation in documentation job * Add raw number/integer option to d3 number formats (#9562) * Bump superset-ui-connection to 0.12.22 (#9602) * [charts] feat: add statsd to charts api (#9571) * add statsd to charts api * update test for charts api * [charts] add statsd asserts wrapper * [charts] update api test * removed white space * [tests] Helper script to run single tests (#9547) * [charts] adds new filters ui (#9530) * [charts] adds new filters ui * move null check to be more visible * better filter lists and async filter functionality * [SQL Lab] Lock result set controls to be always visible (#9587) * improve helm chart * add Apache license headers * add initjob conf + secret * add missing headers * apply changes * opps again .... license Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com> Co-authored-by: Krist Wongsuphasawat <krist.wongz@gmail.com> Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com> Co-authored-by: ʈᵃᵢ <tdupreetan@gmail.com> Co-authored-by: Evan Rusackas <evan@preset.io> Co-authored-by: Maxime Beauchemin <maximebeauchemin@gmail.com> Co-authored-by: Daniel Vaz Gaspar <danielvazgaspar@gmail.com> Co-authored-by: Will Barrett <will@preset.io> Co-authored-by: Craig Rueda <craig@craigrueda.com> Co-authored-by: Grace Guo <grace.guo@airbnb.com> Co-authored-by: Jesse Yang <jesse.yang@airbnb.com> Co-authored-by: Alexander Mancevice <amancevice@cargometrics.com> Co-authored-by: John Bodley <john.bodley@airbnb.com> Co-authored-by: David Aaron Suddjian <1858430+suddjian@users.noreply.github.com> Co-authored-by: venter <venter.zhu@outlook.com> Co-authored-by: Naren <narendrans@users.noreply.github.com> Co-authored-by: Luca Toscano <elukey@users.noreply.github.com> Co-authored-by: dandanhub <dandanshi@live.cn> Co-authored-by: Dandan Shi <dshi@dropbox.com> Co-authored-by: Bogdan <b.kyryliuk@gmail.com> Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com> Co-authored-by: Octavian <octavian.corlade@gmail.com> Co-authored-by: Aliaksei Kushniarevich <axelet3@gmail.com> Co-authored-by: David Aaron Suddjian <aasuddjian@gmail.com> Co-authored-by: Erik Ritter <erik.ritter@airbnb.com> Co-authored-by: Lily Kuang <lily@preset.io>
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CATEGORY
Choose one
SUMMARY
Superset by default only supports templating using the Jinja templating language which allows for using macros in your SQL code.
The PR is to introduce CUSTOM_TEMPLATE_PROCESSOR config to allow customizing macros template processors for various database engine. A good example of usage is to support processor that treats $MACRO as macro, as opposed to {{ JINJIA_TEMPLATE }}.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TEST PLAN
ADDITIONAL INFORMATION
REVIEWERS
@bkyryliuk