A python quasi-client (and tools) that interacts with DataHub GraphQL endpoints.
Requires python >=3.9
pip install git+https://github.com/makenotion/datahub-tools
Three environment variables are required:
- DATAHUB_GMS_URL - e.g. "https://your_business.acryl.io/gms"
- DATAHUB_GRAPHQL_URL - e.g. "https://your_business.acryl.io/api/graphql"
- DATAHUB_GMS_TOKEN - Access token generated from https://your_business.acryl.io/settings/tokens
Make sure you have your environment variables set (see above)
Sample some of your DataHub entities
from datahub_tools.client import get_datahub_entities
from datahub_tools.classes import DHEntity
entities: list[DHEntity] = get_datahub_entities(limit=5)
Take the first entity and apply some changes
from datahub_tools.client import update_dataset_description, emit_metadata
entity = entities[0]
# update the table's description
update_dataset_description(
resource_urn=entity.urn,
description="The new description for this entity"
)
# update the description for a column within the table
update_field_descriptions(
resource_urn=entity.urn,
field_descriptions={"my_column": "A new description for this column"}
)
# set the custom properties
emit_metadata(metadata={'foo': 'bar'}, resource_urn=entity.urn)
This package was not created with the intent of open-sourcing (rather just for internal use at Notion) and so it does not contain wrappers around everything endpoint. There is a lot missing before reaching feature parity but hopefully still helpful.
The goals of the functions were to provide simple and easy programmatic access to common DataHub operations
- emit metadata
- fetch entities (datasets)
- fetch datahub users
- fetch datahub groups
- remove/set user/group owners
- remove/set tags
Users are encouraged to setup logging in advance as many steps and communications are logged to INFO.
Contributions are welcome, but please add tests for new code. Testing is setup using pytest and orchestrated with tox:
pyenv shell 3.9.16 3.10.9 3.11.1
pip install -U pip setuptools tox
tox
pyenv shell -
brew install pre-commit
pre-commit install
The pre-commit hooks will run automatically upon a commit
Contact Evelyn Wou <evelyn[at]makenotion.com> Originally authored by Ada Draginda