implement abstractstagingclient for azure blob storage #1218
Conversation
jklegar
requested review from
davidheryanto,
khorshuheng,
pyalex,
woop and
zhilingc
as code owners
|
/kind feature |
jklegar
force-pushed
the
jacob_azure_blob
branch
from
ab77383
to
3eaf3a0
Compare
sdk/python/feast/loaders/file.py
Outdated
| @@ -76,10 +78,10 @@ def export_source_to_staging_location( | |||
| os.path.join(source_uri.netloc, source_uri.path) | |||
| ) | |||
| else: | |||
| # gs, s3 file provided as a source. | |||
| # gs, s3, azure blob file provided as a source. | |||
| assert source_uri.hostname is not None | |||
| return get_staging_client(source_uri.scheme).list_files( | |||
There was a problem hiding this comment.
seems like the Config object isn't available to be passed to get_staging_client here, so is there a better way to pass in the required parameters (account name and access key) for the Azure client than requiring them to be passed in in the config?
jklegar
changed the title
|
I've done some basic manual testing, not sure if there's more that should be done outside of end-to-end Azure tests - what is the case for S3/GCS? |
| AZURE_BLOB_ACCOUNT_NAME: Optional[str] = None | ||
|
|
||
| #: Account access key for Azure blob storage_client | ||
| AZURE_BLOB_ACCOUNT_ACCESS_KEY: Optional[str] = None |
There was a problem hiding this comment.
It's not possible for this configuration to be read from environmental variables by the azure client?
they are covered by e2e tests |
Signed-off-by: Jacob Klegar <jacob@tecton.ai>
Signed-off-by: Jacob Klegar <jacob@tecton.ai>
Signed-off-by: Jacob Klegar <jacob@tecton.ai>
jklegar
force-pushed
the
jacob_azure_blob
branch
from
928fa3a
to
1a9a10a
Compare
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jklegar, woop The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: Jacob Klegar jacob@tecton.ai
What this PR does / why we need it: Supports Azure Blob Storage as an object store and batch data source as part of #1181
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?: