Adding YAML Import-Export for Datasources to CLI

[fabianmenges]

Summary

Adding YAML import and export for datasources, which includes (SqlAlchemy) Databases and Druid Clusters to the Superset CLI.

Description

I added the core of the Import/Export logic to the ImportMixin mix-in class. It heavily relies on SqlAlchemy to determine the schema of the YAML and the relationship of objects. Specifically it uses unique constraints to identify and update existing elements. This required me to add unique constraints to existing tables, but I'm pretty confident that this should not cause major issues with existing installations since I added them in the "spirit" of the current design.

In addition to the SqlAlchemy relationships the main object hierarchy needs to be defined by configuring the export_parent and export_children attributes appropriately (documented in code).

The unit test covers only basic importing exporting (it was liftet from the existing pickle import/export) and should probably be extended to cover more edge cases.

You can export databases, druid clusters, tables, datasources from the UI:

Possible Future Projects/Improvements

• Add YAML import/export for Slices and Dashboards.
• Adding support to import/export individual database / tables / metrics, etc. as root element of a YAML file.
• Support of importing multiple YAML files at the same time
• Support to split exports into smaller files
• It would also be nice to use the YAML import for the example datasets.
• Adding YAML import to the Web UI and/or API

Re-created #2993 This PR was getting too old for travis.
Context: https://groups.google.com/forum/#!topic/airbnb_superset/GeWZs42_NyA

[mistercrunch]

Looks like travis isn't kicking in, any idea why? I noticed that before only on your PRs, bad luck or something related to your workflow?

[fabianmenges]

Apparently travis says 'abuse detected'. I'm contacting travis support and CC you.

[mistercrunch]

In the meantime you can probably branch off your branch, amend the commit with a different commit message to get a different SHA, and open a new PR off of it referencing this one. Might work...

[fabianmenges]

Finally got picked up by Travis.

[yamyamyuo]

Hi @mistercrunch , I am very interested in this feature

Possible Future Projects/Improvements:
Add YAML import/export for Slices.

is there any plans for that?

[mistercrunch]

@yamyamyuo note that it's possible to do most of this using the ORM directly. Could be good to have a code generation feature as an export. Exporting would generate code that uses the ORM to create the slice and/or dashboard.

Check out this file for an example of how we generate the examples:
https://github.com/apache/incubator-superset/blob/master/superset/data/__init__.py#L79

Note that of course this isn't a public API, and we won't provide guarantees around forward compatibility, though it shouldn't change much.

[yamyamyuo]

@mistercrunch I have a deeper look at superset. I found the security part is very detailed but also complex.

1. How did Airbnb manage the user's authorization? Pick some people as admin and grant perm when new users come? How did the admin know every user's role? Shall I grant the Finance group datasource to a marketing employee?
2. Would it be too annoying for the admin to add/delete/edit all the roles and add the roles to each users? Is there an automatic way or more efficient way?
3. Does Airbnb deploy Superset to separate environment? Such as development, staging and production environment. Does the analyst try to create the slice/dashboard in development environment and when it success, do that process again in production environment? How did the analyst create same slice/dashboard in more than one environment? We can't just copy and paste since slice cannot import/export. Did they repeat the same process in different env?

There are so many questions🙈I am really appreciated if you can share your experience or advice!