Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: use service connectors to authenticate secrets stores. #2154

Merged
merged 7 commits into from
Dec 19, 2023
Merged

Feature: use service connectors to authenticate secrets stores. #2154

merged 7 commits into from
Dec 19, 2023

Conversation

stefannica
Copy link
Contributor

@stefannica stefannica commented Dec 14, 2023 •
edited
Loading

Describe changes

Reuse the AWS/GCP/Azure Service Connector functionality to authenticate the AWS/GCP/Azure Secrets Store instead of relying on the limited configuration currently available in the Secrets Store implementations.

  • move connector type and resource type to a global scope where they can be imported without side-effects
  • add auth_method and auth_config attributes to AWS/GCP/Azure secrets store and use them to initialize an in-memory service connector
  • protect the secrets store back-end client initialization with a mutex to account for multi-threading in the ZenML server context
  • implement all service connector related glue in a separate base secrets store class, for maximum reusability
  • keep backwards compatibility with previous configuration attributes
  • update helm chart with new options

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • If I have added an integration, I have updated the integrations table and the corresponding website section.
  • I have added tests to cover my changes.
  • I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@github-actions github-actions bot added internal To filter out internal PRs and issues enhancement New feature or request labels Dec 14, 2023
@stefannica stefannica force-pushed the feature/use-connectors-for-secrets-stores branch from a244079 to fe55e50 Compare December 15, 2023 18:53
@github-actions GitHub Actions
Copy link
Contributor

E2E template updates in examples/e2e have been pushed.

@stefannica stefannica force-pushed the feature/use-connectors-for-secrets-stores branch from 2ca188c to c797ccf Compare December 15, 2023 21:51
@stefannica stefannica marked this pull request as ready for review December 15, 2023 21:51
@github-actions GitHub Actions
Copy link
Contributor

NLP template updates in examples/nlp-case have been pushed.

@stefannica stefannica force-pushed the feature/use-connectors-for-secrets-stores branch from c1d573f to b685e0b Compare December 18, 2023 08:37
wjayesh
wjayesh reviewed Dec 19, 2023
View reviewed changes
Copy link
Contributor

@wjayesh wjayesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to see connectors being used for secrets; less duplication and cleaner code. Thanks!

src/zenml/zen_stores/secrets_stores/service_connector_secrets_store.py Outdated Show resolved Hide resolved
src/zenml/zen_stores/secrets_stores/service_connector_secrets_store.py Show resolved Hide resolved
src/zenml/zen_stores/secrets_stores/aws_secrets_store.py Outdated Show resolved Hide resolved
src/zenml/zen_stores/secrets_stores/aws_secrets_store.py Outdated Show resolved Hide resolved
src/zenml/zen_stores/secrets_stores/aws_secrets_store.py Show resolved Hide resolved
src/zenml/zen_stores/secrets_stores/azure_secrets_store.py Show resolved Hide resolved
src/zenml/zen_stores/schemas/code_repository_schemas.py Show resolved Hide resolved
@stefannica @wjayesh
Apply suggestions from code review
1c2ad8a 
Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>
@github-actions GitHub Actions
Copy link
Contributor

E2E template updates in examples/e2e have been pushed.

wjayesh
wjayesh approved these changes Dec 19, 2023
View reviewed changes
Copy link
Contributor

@wjayesh wjayesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🛷!

@stefannica stefannica merged commit 3ba010d into develop Dec 19, 2023
5 checks passed
@stefannica stefannica deleted the feature/use-connectors-for-secrets-stores branch December 19, 2023 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wjayesh wjayesh wjayesh approved these changes

@safoinme safoinme Awaiting requested review from safoinme

Assignees
No one assigned
Labels
enhancement New feature or request internal To filter out internal PRs and issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@stefannica @wjayesh @actions-user