[DataComp] Update pipeline name, remove DockerCompiler

[NielsRogge]

This PR updates the pipeline name of DataComp, and makes sure it can run fine both locally and on GCP.

Switching between local vs GCP is currently done as follows:

• I need to manually comment out the base path I don't want to use
• it's either running fondant run pipeline:pipeline --local or python pipeline.py

[NielsRogge]

The cluster image embeddings component won't be used for now, which is why it is removed here.

[PhilippeMoussalli]

Switching between local vs GCP is currently done as follows:

• I need to manually comment out the base path I don't want to use

Isn't this just changing the base path in PipelineConfigs? you can use a local base path or a remote one for the local runner. For the remote runner, it's always the remote one.

• it's either running fondant run pipeline:pipeline --local or python pipeline.py

I think you can keep the interface and code of the pipeline as is, for now only fondant run pipeline:pipeline --local running the pipeline with kubeflow using a fondant run should be soon available with this PR #291 (@GeorgesLorre can you confirm?).

[NielsRogge]

Isn't this just changing the base path in PipelineConfigs? you can use a local base path or a remote one for the local runner. For the remote runner, it's always the remote one.

Yes but as I'm switching regularly between both, it's handy to specify both a local and a remote base path rather than having to type it everytime. cc @GeorgesLorre

[RobbeSneyders]

Thanks @NielsRogge!

[NielsRogge]

Feel free to approve :)

[GeorgesLorre]

Isn't this just changing the base path in PipelineConfigs? you can use a local base path or a remote one for the local runner. For the remote runner, it's always the remote one.

Yes but as I'm switching regularly between both, it's handy to specify both a local and a remote base path rather than having to type it everytime. cc @GeorgesLorre

I'm working on making the interface for docker and kfp more similar (TBD).

But you can alter your pipeline.py already to make this easier:

if __name__ == "__main__":
    # assuming kfp run
    client = Client(host=PipelineConfigs.HOST)
    # override the base_path
    pipeline.base_path = "some/remote/path"
    client.compile_and_run(pipeline=pipeline)

So if you call your pipeline.py it will assume it is a kfp run and override the base_path and if you use fondant run pipeline:pipeline --local it should not run the code under the if statement.