Augment DockerRunner to support running from a fondant Pipeline

[GeorgesLorre]

No description provided.

[GeorgesLorre]

  E   TypeError: unsupported operand type(s) for |: 'type' and 'type'

Thanks python 3.8...

[RobbeSneyders]

Thanks @GeorgesLorre! The general approach looks good!

[RobbeSneyders]

Shouldn't this be _run?

[RobbeSneyders]

Do we need to expose this? I would move this to a temp file or hidden directory.

[PhilippeMoussalli]

Wouldn't this make it more difficult to run the pipeline again from the same component spec to follow the "compile once, run many" approach (well in this case it would be "run once, run many")?

Unless this is temporary until we introduce a general compiler that we can start from as proposed here https://github.com/orgs/ml6team/projects/1/views/8?filterQuery=&pane=issue&itemId=44594106

[RobbeSneyders]

I would only hide this in the Runner. Users can still explicitly compile using the compile command or Compiler class.

[GeorgesLorre]

With expose you mean the file and not the argument ?

[RobbeSneyders]

Both

[GeorgesLorre]

updated

[mrchtr]

I know super picky, but can we use a common yaml file extension within Fondant. Either yml or yaml?
btw: accordingly to the docker compose docs the new preferred default file for docker compose is called compose.yaml

[GeorgesLorre]

Fair point ( can we add this to precommit ?). I will make the changes

[PhilippeMoussalli]

_ No description provided. _

Are we planning on doing the same for kfp and Vertex?

[GeorgesLorre]

_ No description provided. _

Are we planning on doing the same for kfp and Vertex?

Yes the idea is to make the Runner classes prime user interfaces (next to the cli). Now you need to import a compiler and runner and and compile first before running. The idea here is that the user would only need 1 class that is easy to use.