Add support for mounting custom volumes (cloud credentials) #212
Conversation
GeorgesLorre
force-pushed
the
feature/mount-extra-volumes
branch
from
f2c7c1c
to
729979e
Compare
|
There seem to be a lot of file changes with formatting / renaming. Is this from the Pandas PR? |
GeorgesLorre
force-pushed
the
feature/service-build
branch
from
bba7ef5
to
e7397ba
Compare
GeorgesLorre
requested review from
RobbeSneyders,
ChristiaensBert and
PhilippeMoussalli
should be solved (had to rebase) |
| # example for GCP, this mounts your application default credentials to the docker container | ||
| "$HOME/.config/gcloud/application_default_credentials.json:/root/.config/gcloud/application_default_credentials.json:ro" | ||
| # exaple for AWS, this mounts your current credentials to the docker container | ||
| "$HOME/.aws/credentials:/root/.aws/credentials:ro" |
There was a problem hiding this comment.
Maybe check with the Azure heroes if there's a similar way to it for Azure?
| @@ -106,7 +117,7 @@ def _generate_spec(self, pipeline: Pipeline) -> dict: | |||
| command = ["--metadata", json.dumps(asdict(metadata))] | |||
|
|
|||
| # add in and out manifest paths to command | |||
| command.extend(["--output_manifest_path", f"{path}/manifest.txt"]) | |||
| command.extend(["--output_manifest_path", f"{path}/manifest.json"]) | |||
There was a problem hiding this comment.
This still overwrites the manifest for every component. We should still address this, but can be taken up in a separate PR.
There was a problem hiding this comment.
will fix in another PR
|
|
||
| extra_volumes = [ | ||
| # example for GCP, this mounts your application default credentials to the docker container | ||
| "$HOME/.config/gcloud/application_default_credentials.json:/root/.config/gcloud/application_default_credentials.json:ro" |
There was a problem hiding this comment.
Should the user know the correct within the container themselves or should we provide some additional docs on it. When using an SA, is the application_default_credentials.json still a correct path to mount to?
docs/pipeline.md
Outdated
| extra_volumes = [ | ||
| # example for GCP, this mounts your application default credentials to the docker container | ||
| "$HOME/.config/gcloud/application_default_credentials.json:/root/.config/gcloud/application_default_credentials.json:ro" | ||
| # exaple for AWS, this mounts your current credentials to the docker container |
There was a problem hiding this comment.
| # exaple for AWS, this mounts your current credentials to the docker container | |
| # example for AWS, this mounts your current credentials to the docker container |
| Args: | ||
| pipeline: the pipeline to compile | ||
| output_path: the path where to save the docker-compose spec | ||
| extra_volumes: a list of extra volumes (using the Short syntax: |
There was a problem hiding this comment.
Add some information about the format of the extra_volumes. e.g. list of "source:target" strings with absolute paths where target is the credentials folder/file path specified by the used cloud provider/service
GeorgesLorre
force-pushed
the
feature/mount-extra-volumes
branch
from
8b7ab75
to
5ab4125
Compare
No description provided.