Granular compiler tests

[PhilippeMoussalli]

Addresses https://github.com/ml6team/fondant-use-cases/issues/46

For each compiler we now have a method get_pipeline_configs() to fetch the essential information about a pipeline (component, arguments, dependencies, hardware, ...) from a compiled specification file. In this way, we no longer face issues with having to modify the specification files after every small modification to the framework. It also makes it easier to call the desired attribute without having to dive deep into the schema of the compiler.

[GeorgesLorre]

Nice changes, will definitely make testing more easy.

I do wonder if we can't do more with these ComponentConfigs ? Now they are a bit of an afterthought and only used for testing. Maybe we can also use them to configure the components for a runner ?

[PhilippeMoussalli]

Nice changes, will definitely make testing more easy.

I do wonder if we can't do more with these ComponentConfigs ? Now they are a bit of an afterthought and only used for testing. Maybe we can also use them to configure the components for a runner ?

Thinking about it now, they seem to very similar to the componentOp main difference is that the componentOp defines static configs whereas the ComponentConifgs include some attributes similar to the ComponentOp with some additional dynamic configs that can only be inferred at compile time (context, volumes, context, ...).
How would you want to use them for a runner?

[RobbeSneyders]

Not sure yet how to use them, but if we only use them for testing, I would remove them from the core Fondant code. We can either

• Move them to our test files
• Create a testing.py module if we think it can help users test Fondant pipelines as well. Connexion example

[GeorgesLorre]

I think we are all feeling the same thing: This is really clean but we only use it for testing right now so either:

• use it for more (like configuring a compiler/runner)
• or make it test specific only (and then maybe indeed move it from the core)

[PhilippeMoussalli]

Thanks for the suggestions, moved them over to a separate testing.py