Dynamic task passing to a state machine

[pevogam]

Hi all,

Is there a plan to support adding more tasks to the requested list of an already used state machine and it its workers? This could be useful for instance if another process or coroutine generates more tests in parallel with a running test runner. Some elaborate tests from Cartesian configuration for instance could take quite some time to parse and one could for instance start the parallel test runs while passing further requested tasks to the requested list using some locks but so far it seems the state machine lock is internal and for internal use only.

Besides this, it would be better if the state machine takes care of its own locking and the management of the various stage lists for tasks so the current internalization of the asyncio lock is in fact better as long as one has a property or a way to modify at least the requested task list after the state machine construction.

What are your thoughts about this?

[willianrampazzo]

Hi all,

Is there a plan to support adding more tasks to the requested list of an already used state machine and it its workers? This could be useful for instance if another process or coroutine generates more tests in parallel with a running test runner. Some elaborate tests from Cartesian configuration for instance could take quite some time to parse and one could for instance start the parallel test runs while passing further requested tasks to the requested list using some locks but so far it seems the state machine lock is internal and for internal use only.

This is an interesting idea. I'm planning on writing a blueprint about an extension of the current dependency graph created inside the state machine. The idea is to create the dependency graph outside the state machine and let it handle the graph.

Thinking about your question, if a task creates an independent flow of new tasks, I don't see a reason that prevents adding it to the state machine.

So, I don't think we have concrete plans to add this feature, but I may say it is a feasible feature.

Besides this, it would be better if the state machine takes care of its own locking and the management of the various stage lists for tasks so the current internalization of the asyncio lock is in fact better as long as one has a property or a way to modify at least the requested task list after the state machine construction.

I understand your point. This would require a kind of refactor/redesign of the current state machine we have. I'm not against it, right now what prevents us from doing that is probably time and priorities.

[pevogam]

Hi all,
Is there a plan to support adding more tasks to the requested list of an already used state machine and it its workers? This could be useful for instance if another process or coroutine generates more tests in parallel with a running test runner. Some elaborate tests from Cartesian configuration for instance could take quite some time to parse and one could for instance start the parallel test runs while passing further requested tasks to the requested list using some locks but so far it seems the state machine lock is internal and for internal use only.

This is an interesting idea. I'm planning on writing a blueprint about an extension of the current dependency graph created inside the state machine. The idea is to create the dependency graph outside the state machine and let it handle the graph.

This is exactly what we do at the moment - we parse a graph, dump some charts of the dependency structure, and traverse it with parallel workers. The reason I was asking about a state machine where one could add tasks is because we could integrate our code more tightly with yours if we could reuse the state machine and "queue" new tasks to it. Currently we have to create a new state machine for each region of parallelizable tests within the graph.

[pevogam]

Thanks to @richtja work in 6cb1c2f this feature is now available and makes integration with the state much easier as mentioned above.