Is it possible to specify different executors for solids in a single pipeline?

[lgblkb]

Basically, I have a pipeline that have both short and long-running solids. Currently, the whole pipeline is configured to be executed using dagster-celery workers since long-running solids are CPU intensive and thus benefit from parallelism. However, there are also a multitude of simple short-running solids that I don't feel comfortable being executed by celery workers.

Therefore, I want to execute simple solids with in-process executor, while leaving long-running heavy solids to dagster-celery workers. Is there any possibility to do that kind of magic wizardry?) If not, what other options do I have? Thank you guys, you are awesome!

[alangenfeld]

We do not yet offer that type of dynamic flexibility out of the box. It is technically possible to implement a custom Executor that does this, but would not be easy. Another thing to consider is merging short-running solids together so that the celery overhead is less impactful.

[lgblkb]

I see. Thanks for reply.