Skip to content

Commit

Permalink
feat(business-types): initial implementation of SIP-78 (#18794)
Browse files Browse the repository at this point in the history 
* add BUSINESS_TYPE_ADDONS to config with example callback

* Removing uneeded whitespace

* [Work in progress] Modifying cidr function to allow for  single ip and adding port outline

* Added test REST endpoint, added some more ports

I've thrown in a test.py script as well that will try to connect to the
business_type endpoint.

* Moving code from config.py into the business api

Very simple api is exposed that will allow someone to call a checkport
endpoint and get back a response.

* Removing commented out bits.

* Adding fucntion dict back to the config

* Moving business_type endpoint to charts

* Adding schema for get endpoint

* Removing imports, updating docstring, fixing typo

Just some small changes as described in the title.  I've updated the
test.py as well so it functions with the endpoint changes.

* Adding translation dict

* Fixing ops

* Adding check for list

* Modifying changes to add quotes where needed

Also changed BusinessTypeResponse to resp.

* Adding in some code to call the filter config

If a column starts with "cidr_" it will call the code in config.py to
try to translate the filter.  Nothing is changed in the JSON being
executed, just some information is dumped to console.

* Porting Ryan's changes

* Adding migration script (as per Ryan's PR)

* Fixing typo

* Prettier fixes

* [CLDN-1043] Adding rough version of filter changes for business types

* fix down migration

* Fixing bugs after merge

* adding functionality to appy filters in back end

* Fixing linting issues

* fix down revision

* Changing conversion callback to handle multiple values at once

* Adding string representation of values

* Code cleanup plus fixing debouce to only be called once for each entry

* Removing non needed logginh

* Changing operator list to use sting values

* Using text value operators

* Removing clear operator call

* Moving business type endpoints

* fix down revision

* Adding port functions

* update migration

* fix bad rebase and add ff

* implement validator

* dont add invalid values to response

* [CLDN-1205] Added a new exception type for a business type translation error. Added the error message in the display_value field within the business type response. Modified the IP and Port business types to populate the error message field in the response if an error occurs

* [CLDN-1205] Added meaningful error message for port translation errors

* Removing status field from businesstype Response and adding in error message

* [CLDN-1205] Added check to make sure the port business type is within the valid range of ports, if it is not, it will populate the error message

* [CLDN-1205] Fixed the if statement that checks to see if the string_value is in the valid range of port numbers. It did not corrently verify this before now.

* [CLDN-1205] Fixed an error where it was trying to use string_value in <= statements. I just casted string_value to an integer if it is numeric, which allows <= operators to be used on it

* [CLDN-1207] Added unit tests for the cidr_func and port_translation_func functions which are located in /superset/config.py

* [CLDN-1207] removed the assertRaises line as it does not work with the cidr_func and port_translation_func functions

* [CLDN-1207] Added the skeleton of the test_cidr_translate_filter_func unit test, still need to update what the expected response from the function will be.

* [CLDN-1207] Added the remainder of the back-end unit tests for the business types

* [CLDN-1207] Fixed the syntax error which caused the test_cidr_translate_filter_func_NOT_IN_double unit test to fail

* [CLDN-1207] Removed the logging that was added for debugging purposes

* [CLDN-1207] Formatted the commands_tests.py file to make it nicer to look at/read through

* [CLDN-1207] Fixed the code so that it conformed to the pylint requirements (i.e., pylint no longer complains about the code in commands_tests.py)

* [CLDN-1207] Modified some of the docstrings so they made better use of the 100 character per line, line limit

* [CLDN-1207] Added the beginnings of the unit tests for the
business types API

* [CLDN-1207] Added a comment to the top of the commands_tests.py file explaining how to run the unit tests. This prevents the next person who tries to run them from having to waste time trying the different forms of testing that Superset supports (e.g., pytest, tox, etc.)

* [CLDN-1207] Added a grammar fix to the comments describing how to run the unit tests

* [CLDN-1207] Modified the description of the business_type API endpoints as they did not represent what the API was actually doing

* [CLDN-1207] Added further instructions on how to run the unit tests that are within the business_type/api_tests.py file

* add request validation

* disable request if business type missing

* [CLDN-1207] Unit tests for the business type API are now working, however, they need to be modified to make use of @mock as we don't want to have to run the server to be able to run the unit tests

* Removing businesss types deffinitons from config

* Adding select to only show valid business types

* Fixed Enzyme tests

* Added scalfolding for selecting filter dropdown

* Adding intigration tests

* fix revision

* fix typos and unnecessary requests

* break out useBusinessTypes

* Added front-end RTL unit tests for the business type API endpoint

* Fixed error from unit tests

* Added a unit test to ensure the operator list is updated after a business type API response is received

* Removing elect compoenet for business types

* Adding feature flag and allowing saving when no business type present

* fixing useEffect hooks

* Adding feature flag to model

* Changing behavior such that an empty string returns a default response

* add form validation

* Modified comments in unit test as command to run test has changed

* Modified comments in unit test as filename to run test has changed

* Modified the api_tests.py file to conform to the linting requirements

* Changed the name of one of the tests to reflect what the test is actually testing

* Added cypress back to the package.json

* Added informative comments

* Updated comments in files as well as removed imports which were not being used

* Changes made by npm run prettier

* Fixed spelling mistakes

* Updated models.py to remove placeholder comments used in development

* Added feature flag mocking in unit test

* Fixing open api failure

* Fixing business types to pass unit tests

* Reverting unsafe connections back to false

* Removing print statement

* Adding business tpye to export test

* setting default feature flag to false for business type

* Reverting pre commit

* Reverting pre commit and running pre commit

* Reverting pre commit and running pre commit

* Fixing formatting

* Adding license

* Fixing Linting

* Protecting api enpoints

* updating model

* Fixing code path when business type exists

* Linting

* Linting

* Fixing linting

* Fixing spelling

* Fixing schemas

* Fixing app import

* fixing item render

* Added RTL test to make sure business type operator list is updated after API response

* Fixing linting

* fix migration

* Changing unit tests

* Fixing import and DB migration after rebase

* Renaming to advanced types

* Fixing Linting

* More renaming

* Removing uneeded change

* Fixing linting and test errors

* Removing unused imports

* linting

* Adding more detailed name for migration

* Moving files to plugins

* more renaming

* Fixing schema name

* Disabling feature flag that should not be enabled by default

* Adding extra cehck

* NameChange

* formatting

* Fixing equals check

* Moveing all advanced type classes and types to one file, and converting tests to functional

* Adding advanced type to test and fix linitng

Co-authored-by: Ville Brofeldt <ville.v.brofeldt@gmail.com>
Co-authored-by: Dan Parent <daniel.parent@cse-cst.gc.ca>
Co-authored-by: GITHUB_USERNAME <EMAIL>
Co-authored-by: cccs-Dustin <96579982+cccs-Dustin@users.noreply.github.com>
  • Loading branch information
@cccs-RyanS @villebro
@dparent1 @cccs-Dustin
4 people authored May 16, 2022
1 parent 2650131 commit ddc01ea
Show file tree
Hide file tree
Showing 36 changed files with 2,095 additions and 104 deletions.
12 changes: 12 additions & 0 deletions docs/static/resources/openapi.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3205,6 +3205,12 @@
"owners": {
"$ref": "#/components/schemas/DashboardRestApi.get_list.User2"
},
"advanced_data_type": {
"maxLength": 255,
"minLength": 1,
"nullable": true,
"type": "string"
},
"position_json": {
"nullable": true,
"type": "string"
Expand Down Expand Up @@ -4245,6 +4251,12 @@
"nullable": true,
"type": "string"
},
"advanced_data_type": {
"maxLength": 255,
"minLength": 1,
"nullable": true,
"type": "string"
},
"uuid": {
"format": "uuid",
"nullable": true,
Expand Down
1 change: 1 addition & 0 deletions superset-frontend/packages/superset-ui-core/src/utils/featureFlags.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export enum FeatureFlag {
ALERT_REPORTS = 'ALERT_REPORTS',
CLIENT_CACHE = 'CLIENT_CACHE',
DYNAMIC_PLUGINS = 'DYNAMIC_PLUGINS',
ENABLE_ADVANCED_DATA_TYPES = 'ENABLE_ADVANCED_DATA_TYPES',
SCHEDULED_QUERIES = 'SCHEDULED_QUERIES',
SQL_VALIDATORS_BY_ENGINE = 'SQL_VALIDATORS_BY_ENGINE',
ESTIMATE_QUERY_COST = 'ESTIMATE_QUERY_COST',
Expand Down
249 changes: 178 additions & 71 deletions superset-frontend/src/components/Datasource/DatasourceEditor.jsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -173,22 +173,46 @@ function ColumnCollectionTable({
return (
<CollectionTable
collection={columns}
tableColumns={[
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]}
sortColumns={[
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]}
tableColumns={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? [
'column_name',
'advanced_data_type',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
: [
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
}
sortColumns={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? [
'column_name',
'advanced_data_type',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
: [
'column_name',
'type',
'is_dttm',
'main_dttm_col',
'filterable',
'groupby',
]
}
allowDeletes
allowAddItem={allowAddItem}
itemGenerator={itemGenerator}
Expand Down Expand Up @@ -243,6 +267,20 @@ function ColumnCollectionTable({
}
/>
)}
{isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES) ? (
<Field
fieldKey="advanced_data_type"
label={t('Advanced data type')}
control={
<TextControl
controlId="advanced_data_type"
placeholder={t('Advanced Data type')}
/>
}
/>
) : (
<></>
)}
<Field
fieldKey="python_date_format"
label={t('Datetime format')}
Expand Down Expand Up @@ -300,62 +338,131 @@ function ColumnCollectionTable({
</Fieldset>
</FormContainer>
}
columnLabels={{
column_name: t('Column'),
type: t('Data type'),
groupby: t('Is dimension'),
is_dttm: t('Is temporal'),
main_dttm_col: t('Default datetime'),
filterable: t('Is filterable'),
}}
columnLabels={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? {
column_name: t('Column'),
advanced_data_type: t('Advanced data type'),
type: t('Data type'),
groupby: t('Is dimension'),
is_dttm: t('Is temporal'),
main_dttm_col: t('Default datetime'),
filterable: t('Is filterable'),
}
: {
column_name: t('Column'),
type: t('Data type'),
groupby: t('Is dimension'),
is_dttm: t('Is temporal'),
main_dttm_col: t('Default datetime'),
filterable: t('Is filterable'),
}
}
onChange={onColumnsChange}
itemRenderers={{
column_name: (v, onItemChange, _, record) =>
editableColumnName ? (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
<TextControl value={v} onChange={onItemChange} />
</StyledLabelWrapper>
) : (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
{v}
</StyledLabelWrapper>
),
main_dttm_col: (value, _onItemChange, _label, record) => {
const checked = datasource.main_dttm_col === record.column_name;
const disabled = !columns.find(
column => column.column_name === record.column_name,
).is_dttm;
return (
<Radio
data-test={`radio-default-dttm-${record.column_name}`}
checked={checked}
disabled={disabled}
onChange={() =>
onDatasourceChange({
...datasource,
main_dttm_col: record.column_name,
})
}
/>
);
},
type: d => (d ? <Label>{d}</Label> : null),
is_dttm: checkboxGenerator,
filterable: checkboxGenerator,
groupby: checkboxGenerator,
}}
itemRenderers={
isFeatureEnabled(FeatureFlag.ENABLE_ADVANCED_DATA_TYPES)
? {
column_name: (v, onItemChange, _, record) =>
editableColumnName ? (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
<EditableTitle
canEdit
title={v}
onSaveTitle={onItemChange}
/>
</StyledLabelWrapper>
) : (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
{v}
</StyledLabelWrapper>
),
main_dttm_col: (value, _onItemChange, _label, record) => {
const checked = datasource.main_dttm_col === record.column_name;
const disabled = !columns.find(
column => column.column_name === record.column_name,
).is_dttm;
return (
<Radio
data-test={`radio-default-dttm-${record.column_name}`}
checked={checked}
disabled={disabled}
onChange={() =>
onDatasourceChange({
...datasource,
main_dttm_col: record.column_name,
})
}
/>
);
},
type: d => (d ? <Label>{d}</Label> : null),
advanced_data_type: d => (
<Label onChange={onColumnsChange}>{d}</Label>
),
is_dttm: checkboxGenerator,
filterable: checkboxGenerator,
groupby: checkboxGenerator,
}
: {
column_name: (v, onItemChange, _, record) =>
editableColumnName ? (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
<TextControl value={v} onChange={onItemChange} />
</StyledLabelWrapper>
) : (
<StyledLabelWrapper>
{record.is_certified && (
<CertifiedBadge
certifiedBy={record.certified_by}
details={record.certification_details}
/>
)}
{v}
</StyledLabelWrapper>
),
main_dttm_col: (value, _onItemChange, _label, record) => {
const checked = datasource.main_dttm_col === record.column_name;
const disabled = !columns.find(
column => column.column_name === record.column_name,
).is_dttm;
return (
<Radio
data-test={`radio-default-dttm-${record.column_name}`}
checked={checked}
disabled={disabled}
onChange={() =>
onDatasourceChange({
...datasource,
main_dttm_col: record.column_name,
})
}
/>
);
},
type: d => (d ? <Label>{d}</Label> : null),
is_dttm: checkboxGenerator,
filterable: checkboxGenerator,
groupby: checkboxGenerator,
}
}
/>
);
}
Expand Down
9 changes: 8 additions & 1 deletion ...t-frontend/src/explore/components/controls/FilterControl/AdhocFilterEditPopover/index.jsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ export default class AdhocFilterEditPopover extends React.Component {
this.onMouseMove = this.onMouseMove.bind(this);
this.onMouseUp = this.onMouseUp.bind(this);
this.onAdhocFilterChange = this.onAdhocFilterChange.bind(this);
this.setSimpleTabIsValid = this.setSimpleTabIsValid.bind(this);
this.adjustHeight = this.adjustHeight.bind(this);
this.onTabChange = this.onTabChange.bind(this);

Expand All @@ -106,6 +107,7 @@ export default class AdhocFilterEditPopover extends React.Component {
width: POPOVER_INITIAL_WIDTH,
height: POPOVER_INITIAL_HEIGHT,
activeKey: this.props?.adhocFilter?.expressionType || 'SIMPLE',
isSimpleTabValid: true,
};

this.popoverContentRef = React.createRef();
Expand All @@ -124,6 +126,10 @@ export default class AdhocFilterEditPopover extends React.Component {
this.setState({ adhocFilter });
}

setSimpleTabIsValid(isValid) {
this.setState({ isSimpleTabValid: isValid });
}

onSave() {
this.props.onChange(this.state.adhocFilter);
this.props.onClose();
Expand Down Expand Up @@ -214,6 +220,7 @@ export default class AdhocFilterEditPopover extends React.Component {
onHeightChange={this.adjustHeight}
partitionColumn={partitionColumn}
popoverRef={this.popoverContentRef.current}
validHandler={this.setSimpleTabIsValid}
/>
</ErrorBoundary>
</Tabs.TabPane>
Expand Down Expand Up @@ -252,7 +259,7 @@ export default class AdhocFilterEditPopover extends React.Component {
</Button>
<Button
data-test="adhoc-filter-edit-popover-save-button"
disabled={!stateIsValid}
disabled={!stateIsValid || !this.state.isSimpleTabValid}
buttonStyle={
hasUnsavedChanges && stateIsValid ? 'primary' : 'default'
}
Expand Down
Loading

6 comments on commit ddc01ea

@jbenguira
Copy link

@jbenguira jbenguira commented on ddc01ea May 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This caused a breaking change because of advanced_data_type was not declared in my config.py file after upgrading to the latest docker image for the tag master.

I have added imports and vars related to advanced_data_type to fix the issue.

But it would be nice to implement it in a non breaking way for next changes 😊

@villebro
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jbenguira can you explain in more detail what the error was and what the workaround was? I tried to reproduce but I can't see any problems in the config.py file.

@jbenguira
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! After upgrading here is the error I was seeing this:

ERROR:superset.app:Failed to create app
superset_worker | Traceback (most recent call last):
superset_worker | File "/app/superset/app.py", line 37, in create_app
superset_worker | app_initializer.init_app()
superset_worker | File "/app/superset/initialization/init.py", line 460, in init_app
superset_worker | self.init_app_in_ctx()
superset_worker | File "/app/superset/initialization/init.py", line 410, in init_app_in_ctx
superset_worker | self.configure_data_sources()
superset_worker | File "/app/superset/initialization/init.py", line 476, in configure_data_sources
superset_worker | ConnectorRegistry.register_sources(module_datasource_map)
superset_worker | File "/app/superset/connectors/connector_registry.py", line 42, in register_sources
superset_worker | module_obj = import(module_name, fromlist=class_names)
superset_worker | File "/app/superset/connectors/sqla/init.py", line 17, in
superset_worker | from . import models, views
superset_worker | File "/app/superset/connectors/sqla/models.py", line 137, in
superset_worker | ADVANCED_DATA_TYPES = config["ADVANCED_DATA_TYPES"]
superset_worker | KeyError: 'ADVANCED_DATA_TYPES'

I was unable to start Superset, it was crashing in a loop, after searching in the source code of Superset I noticed that recent change (this commit)

for me the fix was to add those lines

//at the top
from superset.advanced_data_type.plugins.internet_address import internet_address
from superset.advanced_data_type.plugins.internet_port import internet_port
from superset.advanced_data_type.types import AdvancedDataType

//in default features
"ENABLE_ADVANCED_DATA_TYPES": False,

//below
ADVANCED_DATA_TYPES: Dict[str, AdvancedDataType] = {
"internet_address": internet_address,
"port": internet_port,
}

After that I was able to restart superset

@villebro
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't fully follow - those lines are already featured in config.py. How are you running Superset? It appears you're somehow mixing different cut points of the codebase.

@jbenguira
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm running superset using docker and when I start a new superset instance I have to customize the config.py
So I have created a config.py file mounted as a volume in the docker image

then later, I have upgraded to the latest docker image, but the config.py changed in a breaking way
is there another way to do this?

@villebro
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you should add any customizations to the file superset_config.py, not to config.py directly. If youre running docker-compose, check the file docker/pythonpath_dev/superset_config_local.examplewhich contains info on how to create asuperset_config.pyfordocker-compose`.

Please sign in to comment.