Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Databricks - move init scripts off DBFS #8916

Closed
tgravescs opened this issue Aug 2, 2023 · 7 comments · Fixed by #8961
Closed

[BUG] Databricks - move init scripts off DBFS #8916

tgravescs opened this issue Aug 2, 2023 · 7 comments · Fixed by #8961
Assignees
@NvTimLiu
Labels
bug Something isn't working build Related to CI / CD or cleanly building documentation Improvements or additions to documentation

Comments

@tgravescs
Copy link
Collaborator

Describe the bug
init scripts in DBFS will be deprecated sept 1 2023. Update docs and move any internal testing/dev environments off of dbfs for init scripts.
@tgravescs tgravescs added bug Something isn't working ? - Needs Triage Need team to review and classify labels Aug 2, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Aug 8, 2023
@pxLi pxLi assigned NvTimLiu and unassigned pxLi Aug 9, 2023
@pxLi pxLi added documentation Improvements or additions to documentation build Related to CI / CD or cleanly building labels Aug 9, 2023
@pxLi
Copy link
Collaborator

pxLi commented Aug 9, 2023 •
edited
Loading

cc @NvTimLiu to help with this one, thanks
image

per the doc, Databricks recommends storing all cluster-scoped init scripts using workspace files.
https://docs.databricks.com/en/files/workspace-init-scripts.html
https://docs.databricks.com/en/init-scripts/cluster-scoped.html#configure-a-cluster-scoped-init-script-using-the-ui (we can check if any automated way to do this)

NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Aug 9, 2023
@NvTimLiu
Save Databricks init scripts in the workspace
808879f 
To close: NVIDIA#8916

Init scripts in DBFS will be deprecated sept 1 2023.

Move Databricks init scripts in the workspace

Signed-off-by: Tim Liu <timl@nvidia.com>
NvTimLiu added a commit to NvTimLiu/spark-rapids that referenced this issue Aug 9, 2023
@NvTimLiu
Save Databricks init scripts in the workspace [databricks]
eb1bd47 
To close: NVIDIA#8916

Init scripts in DBFS will be deprecated sept 1 2023.

Move Databricks init scripts in the workspace

Signed-off-by: Tim Liu <timl@nvidia.com>
@NvTimLiu NvTimLiu mentioned this issue Aug 9, 2023
Merged
@sameerz
Copy link
Collaborator

sameerz commented Aug 11, 2023

Update docs ...

Per @viadea our documentation is updated.

@viadea
Copy link
Collaborator

viadea commented Aug 11, 2023

Yes it is updated by @nvliyuan in #8221

pxLi pushed a commit that referenced this issue Aug 25, 2023
@NvTimLiu
Save Databricks init scripts in the workspace [databricks] (#8961)
69ac83c 
* Save Databricks init scripts in the workspace [databricks]

To close: #8916

Init scripts in DBFS will be deprecated sept 1 2023.

Move Databricks init scripts in the workspace

Signed-off-by: Tim Liu <timl@nvidia.com>

* Change Databricks init scripts to workspace from dbfs for docs

Signed-off-by: Tim Liu <timl@nvidia.com>

* Reuse init scripts path

Signed-off-by: Tim Liu <timl@nvidia.com>

* Update copyright

Signed-off-by: Tim Liu <timl@nvidia.com>

* Move uploading and cleaning Databricks workspace initscripts into the common functions

Signed-off-by: Tim Liu <timl@nvidia.com>

---------

Signed-off-by: Tim Liu <timl@nvidia.com>
@pxLi pxLi reopened this Aug 25, 2023
@pxLi
Copy link
Collaborator

pxLi commented Aug 25, 2023

merged #8961. lets close this ticket after all internal usages have been updated @NvTimLiu thanks~

@NvTimLiu
Copy link
Collaborator

I will remove all our internal deprecated Jenkinfiles pointing init scripts to DB workspace storage this week, thx~

@mattahrens
Copy link
Collaborator

Can this be closed?

@NvTimLiu
Copy link
Collaborator

Yes, the fixed PR has been merged on the GitHub, Let me close it

There are some CI script update in internal GitLab repo, which I'll file another issue to follow up

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NvTimLiu NvTimLiu

Labels
bug Something isn't working build Related to CI / CD or cleanly building documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Save Databricks init scripts in the workspace [databricks] NvTimLiu/spark-rapids
6 participants
@tgravescs @mattahrens @sameerz @pxLi @viadea @NvTimLiu