forked from getredash/redash
-
Notifications
You must be signed in to change notification settings - Fork 21
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Schema Viewer Drawer (getredash#3291)
* Process extra column metadata for a few sql-based data sources. * Add Table and Column metadata tables. * Periodically update table and column schema tables in a celery task. * Fetching schema returns data from table and column metadata tables. * Add tests for backend changes. * Front-end shows extra table metadata and uses new schema response. * Delete datasource schema data when deleting a data source. * Process and store data source schema when a data source is first created or after a migration. * Tables should have a unique name per datasource. * Addressing review comments. * Update migration file for mixins. * Appease PEP8 * Upgrade migration file for rebase. * Cascade delete. * Adding org_id * Remove redundant column and table prefixes. * Non-existing tables and columns should be filtered out on the server side not client side. * Fetching table samples should be optional and should happen in a separate task per table. * Allow users to force a schema refresh. * Use updated_at to help prune old schema metadata periodically. * Using settings.SCHEMAS_REFRESH_QUEUE * fix for getredash#2426 test * more stable test_interactive_new * Closes #927, #928: Schema refresh improvements. * Closes #934, #935: Remove type from schema browser and don't show empty example column in schema drawer (#936) * Speed up schema fetch requests with fewer postgres queries. * Add column metadata to Athena glue processing. * Fix bug assuming 'metadata' exists for every table. * Closes #939: Persisted, existing table metadata should be updated. * Sample processing should be rate-limited. * Add cli command for refreshing data samples. * Schema refreshes should not overwrite column 'example' field. * refresh_samples() should filter tables_to_sample on the datasource's id being sampled * Correctly wrap long text in schema drawer. Schema Improvements Part 2: Add data source config options. Adding BigQuery schema drawer with data types and samples. Add empty migration to replace the removed schedule_until migration Add merge migration. Fix spacing issue with data scanned value in query execution metadata. Increase schema refresh timeout. Remove old migrations. Co-authored-by: Alison <github@bankofknowledge.net> Co-authored-by: Jannis Leidel <jannis@leidel.info>
- Loading branch information
Showing
55 changed files
with
2,736 additions
and
237 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| import { some } from "lodash"; | ||
| import React from "react"; | ||
| import PropTypes from "prop-types"; | ||
| import Drawer from "antd/lib/drawer"; | ||
| import Table from "antd/lib/table"; | ||
|
|
||
| import { DataSourceMetadata, Query } from "@/components/proptypes"; | ||
|
|
||
| function textWrapRenderer(text) { | ||
| return <div style={{ wordWrap: "break-word", wordBreak: "break-all" }}>{text}</div>; | ||
| } | ||
|
|
||
| export default class SchemaData extends React.PureComponent { | ||
| static propTypes = { | ||
| show: PropTypes.bool.isRequired, | ||
| onClose: PropTypes.func.isRequired, | ||
| tableName: PropTypes.string, | ||
| tableDescription: PropTypes.string, | ||
| tableMetadata: PropTypes.arrayOf(DataSourceMetadata), | ||
| sampleQueries: PropTypes.arrayOf(Query), | ||
| }; | ||
|
|
||
| static defaultProps = { | ||
| tableName: "", | ||
| tableDescription: "", | ||
| tableMetadata: [], | ||
| sampleQueries: [], | ||
| }; | ||
|
|
||
| render() { | ||
| const tableDataColumns = [ | ||
| { | ||
| title: "Metadata", | ||
| dataIndex: "metadata", | ||
| width: 400, | ||
| key: "metadata", | ||
| }, | ||
| { | ||
| title: "Value", | ||
| dataIndex: "value", | ||
| width: 400, | ||
| key: "value", | ||
| render: text => { | ||
| if (typeof text === "string") { | ||
| return text; | ||
| } | ||
| return ( | ||
| <ul style={{ margin: 0, paddingLeft: "15px" }}> | ||
| {Object.values(text).map(query => ( | ||
| <li> | ||
| <a target="_blank" rel="noopener noreferrer" href={`queries/${query.id}/source`}> | ||
| {query.name} | ||
| </a> | ||
| </li> | ||
| ))} | ||
| </ul> | ||
| ); | ||
| }, | ||
| }, | ||
| ]; | ||
|
|
||
| const columnDataColumns = [ | ||
| { | ||
| title: "Column Name", | ||
| dataIndex: "name", | ||
| width: 400, | ||
| key: "name", | ||
| render: textWrapRenderer, | ||
| }, | ||
| { | ||
| title: "Column Type", | ||
| dataIndex: "type", | ||
| width: 400, | ||
| key: "type", | ||
| render: textWrapRenderer, | ||
| }, | ||
| ]; | ||
|
|
||
| const hasDescription = some(this.props.tableMetadata, columnMetadata => columnMetadata.description); | ||
|
|
||
| const hasExample = some(this.props.tableMetadata, columnMetadata => columnMetadata.example); | ||
|
|
||
| if (hasDescription) { | ||
| columnDataColumns.push({ | ||
| title: "Description", | ||
| dataIndex: "description", | ||
| width: 400, | ||
| key: "description", | ||
| render: textWrapRenderer, | ||
| }); | ||
| } | ||
|
|
||
| if (hasExample) { | ||
| columnDataColumns.push({ | ||
| title: "Example", | ||
| dataIndex: "example", | ||
| width: 400, | ||
| key: "example", | ||
| render: textWrapRenderer, | ||
| }); | ||
| } | ||
| const tableData = [ | ||
| { | ||
| metadata: "Table Description", | ||
| value: this.props.tableDescription || "N/A", | ||
| key: "description", | ||
| }, | ||
| { | ||
| metadata: "Sample Usage", | ||
| value: this.props.sampleQueries.length > 0 ? this.props.sampleQueries : "N/A", | ||
| key: "sample", | ||
| }, | ||
| ]; | ||
|
|
||
| return ( | ||
| <Drawer closable={false} placement="bottom" height={500} onClose={this.props.onClose} visible={this.props.show}> | ||
| <h4 style={{ margin: 0 }}>{this.props.tableName}</h4> | ||
| <hr /> | ||
| <h5>Table Data</h5> | ||
| <Table | ||
| dataSource={tableData} | ||
| pagination={false} | ||
| scroll={{ y: 350 }} | ||
| size="small" | ||
| showHeader={false} | ||
| columns={tableDataColumns} | ||
| /> | ||
| <br /> | ||
| <h5>Column Data</h5> | ||
| <Table | ||
| dataSource={this.props.tableMetadata} | ||
| pagination={false} | ||
| scroll={{ y: 175 }} | ||
| size="small" | ||
| rowKey="id" | ||
| columns={columnDataColumns} | ||
| /> | ||
| </Drawer> | ||
| ); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.