Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add MotherDuck DB engine spec #24934

Merged
merged 4 commits into from
Aug 9, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ def get_git_sha() -> str:
"dremio": ["sqlalchemy-dremio>=1.1.5, <1.3"],
"drill": ["sqlalchemy-drill==0.1.dev"],
"druid": ["pydruid>=0.6.5,<0.7"],
"duckdb": ["duckdb-engine==0.8.0"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@betodealmeida should this be >=0.8.0,<1.0.0?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@john-bodley could be, but in general I prefer to pin 0.x versions instead of using a range, since in theory the API is not sable and 0.8.1 could introduce breaking changes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On that note, 0.8.1 is out, and seems to follow semver
https://github.com/duckdb/duckdb/releases/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could go either way on this... trying to rationalize a preference.

If we pin the version, and there's an issue (potentially security) then we might have a fire drill on our hands.

If we use a range, and something breaks, we might have a bug on our hands.

By the "lesser evil" metric, I think this might put the range idea ahead in my mind by about the width of a frog hair, as long as they stick to semver ;)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I know they were planning to change how databases are returned by their dialect, so maybe we keep this frozen until they hit 1.0?

Also, this is an optional dependency, people can always install a newer version if they want.

"dynamodb": ["pydynamodb>=0.4.2"],
"solr": ["sqlalchemy-solr >= 0.2.0"],
"elasticsearch": ["elasticsearch-dbapi>=0.2.9, <0.3.0"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,17 +38,13 @@ const SqlAlchemyTab = ({
testInProgress?: boolean;
children?: ReactNode;
}) => {
let fallbackDocsUrl;
let fallbackDisplayText;
if (SupersetText) {
Copy link
Member Author

@betodealmeida betodealmeida Aug 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SupersetText is either a module or {}, and both are truthy.

const loadModule = () => {
try {
// eslint-disable-next-line global-require, import/no-unresolved
return require('../../../superset_text') || {};
} catch (e) {
return {};
}
};

fallbackDocsUrl =
SupersetText.DB_MODAL_SQLALCHEMY_FORM?.SQLALCHEMY_DOCS_URL;
fallbackDisplayText =
SupersetText.DB_MODAL_SQLALCHEMY_FORM?.SQLALCHEMY_DISPLAY_TEXT;
} else {
fallbackDocsUrl = 'https://docs.sqlalchemy.org/en/13/core/engines.html';
fallbackDisplayText = 'SQLAlchemy docs';
}
const fallbackDocsUrl =
SupersetText?.DB_MODAL_SQLALCHEMY_FORM?.SQLALCHEMY_DOCS_URL ||
'https://docs.sqlalchemy.org/en/13/core/engines.html';
const fallbackDisplayText =
SupersetText?.DB_MODAL_SQLALCHEMY_FORM?.SQLALCHEMY_DISPLAY_TEXT ||
'SQLAlchemy docs';

return (
<>
<StyledInputContainer>
Expand Down Expand Up @@ -82,9 +78,10 @@ const SqlAlchemyTab = ({
data-test="sqlalchemy-uri-input"
value={db?.sqlalchemy_uri || ''}
autoComplete="off"
placeholder={t(
'dialect+driver://username:password@host:port/database',
)}
placeholder={
db?.sqlalchemy_uri_placeholder ||
t('dialect+driver://username:password@host:port/database')
}
onChange={onInputChange}
/>
</div>
Expand Down
10 changes: 8 additions & 2 deletions superset-frontend/src/features/databases/DatabaseModal/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -946,8 +946,13 @@ const DatabaseModal: FunctionComponent<DatabaseModalProps> = ({
const selectedDbModel = availableDbs?.databases.filter(
(db: DatabaseObject) => db.name === database_name,
)[0];
const { engine, parameters, engine_information, default_driver } =
selectedDbModel;
const {
engine,
parameters,
engine_information,
default_driver,
sqlalchemy_uri_placeholder,
} = selectedDbModel;
const isDynamic = parameters !== undefined;
setDB({
type: ActionType.dbSelected,
Expand All @@ -959,6 +964,7 @@ const DatabaseModal: FunctionComponent<DatabaseModalProps> = ({
: CONFIGURATION_METHOD.SQLALCHEMY_URI,
engine_information,
driver: default_driver,
sqlalchemy_uri_placeholder,
},
});

Expand Down
4 changes: 4 additions & 0 deletions superset/databases/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -1293,6 +1293,9 @@ def available(self) -> Response:
type: array
items:
type: string
sqlalchemy_uri_placeholder:
description: Placeholder for the SQLAlchemy URI
type: string
default_driver:
description: Default driver for the engine
type: string
Expand Down Expand Up @@ -1330,6 +1333,7 @@ def available(self) -> Response:
"name": engine_spec.engine_name,
"engine": engine_spec.engine,
"available_drivers": sorted(drivers),
"sqlalchemy_uri_placeholder": engine_spec.sqlalchemy_uri_placeholder,
"preferred": engine_spec.engine_name in preferred_databases,
"engine_information": engine_spec.get_public_information(),
}
Expand Down
11 changes: 6 additions & 5 deletions superset/db_engine_specs/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,12 @@ class BaseEngineSpec: # pylint: disable=too-many-public-methods
engine_aliases: set[str] = set()
drivers: dict[str, str] = {}
default_driver: str | None = None

# placeholder with the SQLAlchemy URI template
sqlalchemy_uri_placeholder = (
"engine+driver://user:password@host:port/dbname[?key=value&key=value...]"
)

disable_ssh_tunneling = False

_date_trunc_functions: dict[str, str] = {}
Expand Down Expand Up @@ -1958,11 +1964,6 @@ class BasicParametersMixin:
# recommended driver name for the DB engine spec
default_driver = ""

# placeholder with the SQLAlchemy URI template
sqlalchemy_uri_placeholder = (
Copy link
Member Author

@betodealmeida betodealmeida Aug 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving this from the "dynamic form" mixin to the base DB engine spec.

"engine+driver://user:password@host:port/dbname[?key=value&key=value...]"
)

# query parameter to enable encryption in the database connection
# for Postgres this would be `{"sslmode": "verify-ca"}`, eg.
encryption_parameters: dict[str, str] = {}
Expand Down
7 changes: 7 additions & 0 deletions superset/db_engine_specs/duckdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,10 @@ def get_table_names(
cls, database: Database, inspector: Inspector, schema: str | None
) -> set[str]:
return set(inspector.get_table_names(schema))


class MotherDuckEngineSpec(DuckDBEngineSpec):
engine = "duckdb"
engine_name = "MotherDuck"

sqlalchemy_uri_placeholder = "duckdb:///md:{SERVICE_TOKEN}@{database_name}"