Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

♻️ Refactor current code to use Python SDK #27

Merged
merged 86 commits into from
Jul 25, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
7405aa8
check for valid pat
nsenno-dbr Jun 8, 2023
fb31a1b
Merge branch 'main' into python-sdk
renardeinside Jun 8, 2023
8444b26
Merge branch 'main' into python-sdk
renardeinside Jun 9, 2023
65c8ef6
implement authconfig
renardeinside Jun 9, 2023
f2cedbe
get local workspace groups
nsenno-dbr Jun 14, 2023
6802bd0
save workbook
nsenno-dbr Jun 14, 2023
b4a3ad8
Merge branch 'python-sdk' of github.com:databricks/UC-Upgrade into py…
nsenno-dbr Jun 14, 2023
5532df3
remove scratches from repo
renardeinside Jul 17, 2023
291527c
fix linters
renardeinside Jul 17, 2023
5d60560
introduce structure
renardeinside Jul 17, 2023
8bd905d
add clear toolkit instructions
renardeinside Jul 17, 2023
5554f65
fix linter errors
renardeinside Jul 17, 2023
60a08c5
upgrade versions
renardeinside Jul 17, 2023
b4a70b4
Merge remote-tracking branch 'origin/main' into python-sdk
renardeinside Jul 17, 2023
8e6a1b6
fix exclusions
renardeinside Jul 17, 2023
0e38e8b
add runtime imports
renardeinside Jul 17, 2023
1937b30
add nice formatting and docs to the migration notebook
Jul 17, 2023
4240408
add verifications
renardeinside Jul 17, 2023
d893fc8
fix
renardeinside Jul 17, 2023
4c09c11
add creds
renardeinside Jul 17, 2023
0feb6d8
minor fixes
renardeinside Jul 19, 2023
79007c3
finish migration to hatch
renardeinside Jul 19, 2023
5471c0a
Merge remote-tracking branch 'origin/main' into python-sdk
renardeinside Jul 19, 2023
2b3560a
add hatch caching
renardeinside Jul 19, 2023
c610a2d
move logger to mixins
renardeinside Jul 19, 2023
68e5d5f
move logger to mixins
renardeinside Jul 19, 2023
1314a72
move ruff comment to the top
renardeinside Jul 19, 2023
b960141
exclude notebooks from ruff checks
renardeinside Jul 19, 2023
9750b02
fix readme
renardeinside Jul 19, 2023
02963c8
fix readme
renardeinside Jul 19, 2023
bbac931
fix readme
renardeinside Jul 19, 2023
83cd99c
improve dev docs
renardeinside Jul 19, 2023
304502f
remove context-related methods
renardeinside Jul 19, 2023
8b942de
provide installation mechanism for package in the notebook
renardeinside Jul 19, 2023
2986db2
fix imports
renardeinside Jul 19, 2023
aac5b0f
rename function
renardeinside Jul 19, 2023
06d99af
minor fixes
renardeinside Jul 19, 2023
744e31d
minor fixes
renardeinside Jul 19, 2023
0eb8ddd
fix imports
renardeinside Jul 19, 2023
52303d2
fix order
renardeinside Jul 19, 2023
7335fae
add path
renardeinside Jul 19, 2023
f97bf64
add path
renardeinside Jul 19, 2023
9458fb9
provide a new way to introduce dependencies
renardeinside Jul 19, 2023
81671dc
fix imports
renardeinside Jul 19, 2023
085a748
disable reloads
renardeinside Jul 19, 2023
4f5c382
add paths
renardeinside Jul 19, 2023
2794849
add builder and installator
renardeinside Jul 19, 2023
3073a94
add hatch installation
renardeinside Jul 19, 2023
74f1f07
add hatch installation
renardeinside Jul 19, 2023
f7aa254
fix usages
renardeinside Jul 19, 2023
f1d07c5
remove notebooks
renardeinside Jul 19, 2023
7259bd0
introduce tests
renardeinside Jul 20, 2023
e18ef85
fix readme
renardeinside Jul 20, 2023
7a48061
fix readme
renardeinside Jul 20, 2023
d48509c
save first e2e
renardeinside Jul 20, 2023
b1a2a72
add mc tests
renardeinside Jul 20, 2023
c6be9d4
add unit testing
renardeinside Jul 20, 2023
80235f0
add new mocking logic
renardeinside Jul 20, 2023
feb6d65
add config loader
renardeinside Jul 20, 2023
c681978
lint
renardeinside Jul 20, 2023
81cd637
silence pyspark broadcast warnings
renardeinside Jul 20, 2023
cec0213
fix str enum issues
renardeinside Jul 20, 2023
3cd8579
fix text messages
renardeinside Jul 20, 2023
2c613e0
add spark mock
renardeinside Jul 21, 2023
c9ea222
fix lint
renardeinside Jul 21, 2023
cb8bb50
fix strenum
renardeinside Jul 21, 2023
3d08568
add session adapter
renardeinside Jul 21, 2023
f45bf57
align methods to managers
renardeinside Jul 21, 2023
676fc6d
add crud ops for temp groups
renardeinside Jul 21, 2023
1f6cedd
remove temp dirs
renardeinside Jul 21, 2023
bf7bb00
add functionality to apply new permissions to temp groups
renardeinside Jul 24, 2023
2eaa103
apply linter
renardeinside Jul 24, 2023
96c1d34
reformat readme
renardeinside Jul 24, 2023
1b6e7ba
lint
renardeinside Jul 24, 2023
c2e2794
remove args
renardeinside Jul 24, 2023
93e1f03
add group migration logic
renardeinside Jul 25, 2023
7c920c1
add e2e tests and bugfixes
renardeinside Jul 25, 2023
33a35cf
update readme
renardeinside Jul 25, 2023
232873e
improve readme
renardeinside Jul 25, 2023
48f8992
fix readme and e2e tests
renardeinside Jul 25, 2023
3c4668e
include default actions into the integration tests
renardeinside Jul 25, 2023
1230b3c
add verifications to e2e tests
renardeinside Jul 25, 2023
54e2d2f
add e2e tests for permissions
renardeinside Jul 25, 2023
47798cb
lint
renardeinside Jul 25, 2023
d20462e
remove outdated comments
renardeinside Jul 25, 2023
cb1fbdd
finalize entitlements and roles
renardeinside Jul 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ Recommended VM types are:
> Please note that you **don't** need to use `poetry` inside notebooks or in the Databricks workspace.
> It's only introduced to simplify local development.

- This package uses the Databricks Python SDK for [Authentication](https://github.com/databricks/databricks-sdk-py#authentication) to your Databricks Workspace

### Details of package installation

Since the package itself is managed with `poetry`, to re-use it inside the notebooks we're doing the following:
Expand Down
24 changes: 19 additions & 5 deletions uc_upgrade/group_migration.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,18 @@
from pyspark.sql import session
from pyspark.sql.functions import array_contains, col, collect_set, lit
from pyspark.sql.types import MapType, StringType, StructField, StructType

from databricks.sdk import WorkspaceClient
from databricks.sdk.core import DatabricksError

import logging

# Initialize logger
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

client = WorkspaceClient()


class GroupMigration:
def __init__(
Expand Down Expand Up @@ -83,11 +89,19 @@ def __init__(

# Check if we should automatically generate list, and do it immediately.
# Implementers Note: Could change this section to a lazy calculation by setting groupL to nil or some sentinel value and adding checks before use.
res = requests.get(f"{self.workspace_url}/api/2.0/preview/scim/v2/Me", headers=self.headers)
# logger.info(res.text)
if res.status_code == 403:
logger.error("token not valid.")
return

try:
client.current_user.me()
except DatabricksError as e:

error_message, = e.args

if error_message == "Invalid access token.":
logger.error("token not valid.")
return
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HariGS-DB do we want to return here or should we raise the full exception?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think its just a step to test if token is valid. we are doing this in the init hence.

else:
raise e

if autoGenerateList:
logger.info(
"autoGenerateList parameter is set to TRUE. Ignoring groupL parameter and instead will automatically generate list of migraiton groups."
Expand Down