Event driven data-store deltas #3938

dwsutherland · 2020-11-12T02:31:54Z

These changes close #3746, closes #3921, closes #4030
Partially satisfies cylc/cylc-ui#530

At present the data-store deltas are only partially event driven, including:

Node creation (task/familiy proxies and edges).
Job creation and deltas.

A few points on current state and intentions:

Task proxy deltas are not created on event.
The job pool (job_pool.py) is separate from the data-store manager because it is event driven, but after converting the data-store to an event driven approach, this is no longer necessary.

Some updates and these deltas should still be batched for application and publishing (for now) to ensure relational consistency, summary updates are less frequent, and UI/UIS is not overwhelmed. i.e. if you have a family of 100 tasks, and each individual state change constitutes a recalculation of the family state and a publishing of these, then you could end up with hundreds if not thousands of group-state/summary and state deltas to calculate and publish in a few seconds (but batched this would only come to a handful if that).

To do:

Separate the batched task fields delta into separate event driven deltas:
- state
- held
- outputs (separate to capture individual output changes)
- prerequisites (separate to capture individual output changes)
- Clock trigger, external trigger, xtrigger changes (separate for each? just a dump for all in one proxy field ATM)
- Job addition updates
Combine/merge the event-driven job pool (job_pool.py) into the data-store manager.

Most the new delta event triggers will be from within the task_pool.py and task_events_mgr.py

Possible future work:

Publish the deltas individually
Create a data-store at the UIS off these published events, which will decouple the n-windows at both ends.

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Appropriate tests are included (unit and/or functional).
Appropriate change log entry included.
No documentation update required.
No dependency changes.

dwsutherland · 2020-12-18T02:44:15Z

Ok, ready for review..

However, there are a couple things that I might change:

One
The delta calls have been spread through most the event/task/job managers.
I think I want to change this to pass the data_store_mgr into the TaskProxy and objects created therein (i.e. state, outputs, prerequisites)..
Reasons to do this:

We may not be catching all the delta events (i.e. I may have missed places where the outputs are reset)
The way it is now makes maintenance/changes harder.. If other locations for state/output/other changes are added, data-store delta calls might be missed...

Reason(s) not to:

We may be moving away from using a pool of TaskProxys in the future anyway...
...others?

Thoughts?

Two
Some fields of the workflow element should be delta driven:
cylc/cylc-ui#543

hjoliver · 2020-12-21T01:52:20Z

I've been playing with this branch using cylc tui - it's great to see the ui responding properly now (particularly to succeeded events) 🎉

However, I'm seeing this error from the UI Server with Cylc UI:

[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'PbTaskProxy.outputs' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' t
ype if you intend to send raw bytes. 
2020-12-21 14:48:53,393 tornado.application ERROR    Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncI
OMainLoop object at 0x7f22128d2a58>>, <Task finished coro=<WorkflowsManager.update() done, defined at /home/oliverh/cylc/cylc-uiserver/cylc/uiserver/workflows_mgr.py:227> e
xception=DecodeError('Error parsing message')>)
Traceback (most recent call last):
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/home/oliverh/cylc/cylc-uiserver/venv/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
    future.result()
  File "/home/oliverh/cylc/cylc-uiserver/cylc/uiserver/workflows_mgr.py", line 255, in update
    await self._connect(wid, flow)
  File "/home/oliverh/cylc/cylc-uiserver/cylc/uiserver/workflows_mgr.py", line 202, in _connect
    flow
  File "/home/oliverh/cylc/cylc-uiserver/cylc/uiserver/data_store_mgr.py", line 134, in sync_workflow
    await self.entire_workflow_update(ids=[w_id])
  File "/home/oliverh/cylc/cylc-uiserver/cylc/uiserver/data_store_mgr.py", line 320, in entire_workflow_update
    pb_data.ParseFromString(result)
google.protobuf.message.DecodeError: Error parsing message
[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'PbTaskProxy.outputs' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' t
ype if you intend to send raw bytes.

hjoliver · 2020-12-21T02:44:24Z

Seems to be a genuine error. It still happens after recreating my environments. It occurs after starting the first flow on the back end.

dwsutherland · 2020-12-21T02:47:19Z

I'll have a look. But, on the surface, it looks like the UI Server hasn't picked up the protobuf generated module changes.. But you mentioned you recreated your virtual envs.. So, not sure yet

hjoliver · 2020-12-21T03:00:50Z

I did pip install . from cylc-flow with your branch checked out, into cylc-uiserver.

hjoliver · 2020-12-21T03:07:17Z

Also checked the correct data_messages_pb2.py is installed in the cylc-uiserver venv.

cylc/flow/network/schema.py

dwsutherland · 2020-12-21T03:41:11Z

Working for me:

However, separate topic, did the "job runner" cylc-ui end go in?

hjoliver · 2020-12-21T03:42:37Z

However, separate topic, did the "job runner" cylc-ui end go in?

No but Bruno approved it already (on holiday!) and it's a simple change, so I'll merge it now. (That's a PR from me, today, BTW).

hjoliver · 2020-12-21T03:43:57Z

(cylc-ui change merged)

hjoliver · 2020-12-21T04:28:22Z

Update on this problem for others: google.protobuf.message.DecodeError: Error parsing message:

double check that you've done pip install . from this branch of cylc-flow into cylc-uiserver.

cylc/flow/data_store_mgr.py

hjoliver · 2020-12-21T23:38:04Z

(branch conflicted now)

hjoliver · 2020-12-23T00:09:29Z

The delta calls have been spread through most the event/task/job managers. ... Thoughts?

My feeling is leave it as-is for now. This works, and we could consider moving them into the task proxy object later. However, it may not be worth bothering because we want to move to a proper event-loop driven system before too long: objects raise events (with associated data) and a central event loop processes all events, dispatching them to systems (like the datastore) that have registered interest in them. (Something like that).

hjoliver

LGTM 👍, no problems found, so approving this before I bail out for the xmas break. Thanks @dwsutherland 💐

oliver-sanders

Looks good, runs fine, happy to have Tui showing succeeded tasks again 👏.

cylc/flow/data_store_mgr.py

oliver-sanders · 2020-12-29T15:14:54Z

cylc/flow/data_store_mgr.py

+            tp_id, PbTaskProxy(id=tp_id))
+        tp_delta.stamp = f'{tp_id}@{update_time}'
+        ext_trigger = tp_delta.external_triggers[trig]
+        ext_trigger.message = message


As with outputs the message field is updated unnecessarily.

This one's a bit different, as I don't set the message in on task/node creation.. Reason being; I think it's user defined on the fly (via command).

cylc/flow/scheduler.py

oliver-sanders · 2020-12-29T15:34:38Z

cylc/flow/task_events_mgr.py

-        itask.state.reset(TASK_STATUS_WAITING)
+        if itask.state.reset(TASK_STATUS_WAITING):
+            self.data_store_mgr.delta_task_state(itask)


There are a lot of these delta_task_state so a high risk of missing one, would it make sense to put the delta_task_state bit inside TaskProxy.reset to ensure consistency?

I wrestled with this too..

On the one hand it would be simpler to pass the data_store_mgr object into ever TaskProxy, and then the proxies' Outputs, TaskState, and then to Prerequisites object and so on... You can see it being passed around a lot, like public bicycle but worse..

But on the other hand, the data_store_mgr doesn't need to know about all the task proxies (i.e. runahead and other not ready), and/or all their events...

At least this way we can be intentional about it... but I'm open..

Ok, might open a follow-on issue as this isn't entirely a data-store issue. Recently hit a bug where the DB wasn't being updated for a particular state change because the full procedure wasn't being followed, so I'm a bit worried about potential bugs now we have the task proxy, data store, outputs and data base to keep in sync.

(Long-term I would really like task events to be pushed to a centralised "event queue" with multiple consumers able to subscribe to the event stream.)

For now I guess we could write an callback method in the task_pool and pass that to each task_proxy to avoid having to pass a handful of managers down to every object?

TaskPool already has access to suite_db_mgr, task_events_mgr & data_store_mgr.

Bumped - #4037

cylc/flow/scripts/show.py

oliver-sanders · 2020-12-30T15:02:45Z

Prerequisites only seem to be listed for active tasks, presumably a SoD consequence, will need to find a way to generate the output for N>0 tasks as at the moment cylc show incorrectly shows a waiting task as having no prerequisites, should bump to another issue for another day.

oliver-sanders · 2021-01-06T15:07:46Z

Does this close #3921?

The information was already there, but in a different form (updated on event) .. So let's say yes.

kinow

UI working fine with the changes, great work!

@dwsutherland, I noticed while playing with the new code in GraphiQL that I was getting empty deltas, and empty task proxies too. Should them have been stripped/removed from the response instead? Easy to filter in the UI, but looks unnecessary to send {}. And you can see that the number of {}'s corresponds to the number of task proxies (that's five, the same bar/foo task proxies it's always displaying)

No other issues found 👍 I haven't tested it with cylc/cylc-ui#530, as it's addressing that issue partially. @oliver-sanders did a great review. I had a cursory look at the code, and found no other issues +1 pending only @dwsutherland to address @oliver-sanders ' review.

hjoliver · 2021-01-11T00:18:00Z

Prerequisites only seem to be listed for active tasks, presumably a SoD consequence, will need to find a way to generate the output for N>0 tasks as at the moment cylc show incorrectly shows a waiting task as having no prerequisites, should bump to another issue for another day.

Correct, although I'd point out that the old system only had prerequisites for waiting tasks that happened to have corresponding task proxies spawned ahead already.

oliver-sanders · 2021-01-19T09:57:05Z

Chucked the prerequisites thing into another issue - #4036

dwsutherland added efficiency For notable efficiency improvements sod-follow-up labels Nov 12, 2020

dwsutherland added this to the cylc-8.0a3 milestone Nov 12, 2020

dwsutherland self-assigned this Nov 12, 2020

dwsutherland force-pushed the event-driven-deltas branch from 1536fa8 to f0bdaa5 Compare November 12, 2020 02:38

dwsutherland force-pushed the event-driven-deltas branch 4 times, most recently from eb372b6 to 254193b Compare December 18, 2020 02:31

dwsutherland marked this pull request as ready for review December 18, 2020 02:31

dwsutherland force-pushed the event-driven-deltas branch from 254193b to 5e836bf Compare December 18, 2020 02:48

dwsutherland requested review from hjoliver, oliver-sanders and kinow December 18, 2020 02:49

dwsutherland force-pushed the event-driven-deltas branch from 5e836bf to f21ed78 Compare December 18, 2020 05:08

hjoliver reviewed Dec 21, 2020

View reviewed changes

cylc/flow/network/schema.py Show resolved Hide resolved

dwsutherland requested a review from hjoliver December 21, 2020 19:09

hjoliver reviewed Dec 21, 2020

View reviewed changes

cylc/flow/data_store_mgr.py Outdated Show resolved Hide resolved

dwsutherland force-pushed the event-driven-deltas branch from f21ed78 to 8f9942e Compare December 22, 2020 00:20

dwsutherland force-pushed the event-driven-deltas branch from 8f9942e to 92fd21d Compare December 22, 2020 01:10

hjoliver approved these changes Dec 23, 2020

View reviewed changes

hjoliver mentioned this pull request Dec 23, 2020

task progress bar update cylc/cylc-ui#563

Closed

dwsutherland force-pushed the event-driven-deltas branch 4 times, most recently from aebe2e7 to 0f5d7ab Compare December 24, 2020 02:18

oliver-sanders reviewed Dec 30, 2020

View reviewed changes

kinow approved these changes Jan 10, 2021

View reviewed changes

hjoliver mentioned this pull request Jan 11, 2021

showing prerequites of waiting tasks #4030

Closed

dwsutherland added 4 commits January 15, 2021 10:48

event driven task state, output, prereq, and trigger deltas

a582f22

job pool merge

6977fb4

workflow field deltas

892c395

tests & fixes

0dc97ea

dwsutherland force-pushed the event-driven-deltas branch 2 times, most recently from 1c8a612 to 19cee8f Compare January 15, 2021 04:09

Oliver review & prereqs fixes

8b18fe7

dwsutherland force-pushed the event-driven-deltas branch from 19cee8f to 8b18fe7 Compare January 15, 2021 04:15

oliver-sanders mentioned this pull request Jan 19, 2021

compute prerequisites for tasks outside n=0 #4036

Closed

oliver-sanders approved these changes Jan 19, 2021

View reviewed changes

oliver-sanders mentioned this pull request Jan 19, 2021

centralise task state changes #4037

Open

oliver-sanders merged commit af770d4 into cylc:master Jan 19, 2021

kinow mentioned this pull request Feb 15, 2021

Display message triggers cylc/cylc-ui#530

Merged

12 tasks

hjoliver modified the milestones: cylc-8.0a3, cylc-8.0b0 Feb 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Event driven data-store deltas #3938

Event driven data-store deltas #3938

dwsutherland commented Nov 12, 2020 •

edited

Loading

dwsutherland commented Dec 18, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020

dwsutherland commented Dec 21, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020

dwsutherland commented Dec 21, 2020

hjoliver commented Dec 21, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020

hjoliver commented Dec 21, 2020

hjoliver commented Dec 21, 2020

hjoliver commented Dec 23, 2020

hjoliver left a comment

oliver-sanders left a comment •

edited

Loading

oliver-sanders Dec 29, 2020

dwsutherland Jan 14, 2021 •

edited

Loading

oliver-sanders Dec 29, 2020

dwsutherland Jan 14, 2021 •

edited

Loading

oliver-sanders Jan 15, 2021 •

edited

Loading

oliver-sanders Jan 15, 2021 •

edited

Loading

oliver-sanders Jan 19, 2021

oliver-sanders commented Dec 30, 2020

oliver-sanders commented Jan 6, 2021 •

edited by dwsutherland

Loading

kinow left a comment •

edited

Loading

hjoliver commented Jan 11, 2021

oliver-sanders commented Jan 19, 2021

Event driven data-store deltas #3938

Event driven data-store deltas #3938

Conversation

dwsutherland commented Nov 12, 2020 • edited Loading

dwsutherland commented Dec 18, 2020 • edited Loading

hjoliver commented Dec 21, 2020 • edited Loading

hjoliver commented Dec 21, 2020

dwsutherland commented Dec 21, 2020 • edited Loading

hjoliver commented Dec 21, 2020 • edited Loading

hjoliver commented Dec 21, 2020

dwsutherland commented Dec 21, 2020

hjoliver commented Dec 21, 2020 • edited Loading

hjoliver commented Dec 21, 2020

hjoliver commented Dec 21, 2020

hjoliver commented Dec 21, 2020

hjoliver commented Dec 23, 2020

hjoliver left a comment

Choose a reason for hiding this comment

oliver-sanders left a comment • edited Loading

Choose a reason for hiding this comment

oliver-sanders Dec 29, 2020

Choose a reason for hiding this comment

dwsutherland Jan 14, 2021 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Dec 29, 2020

Choose a reason for hiding this comment

dwsutherland Jan 14, 2021 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Jan 15, 2021 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Jan 15, 2021 • edited Loading

Choose a reason for hiding this comment

oliver-sanders Jan 19, 2021

Choose a reason for hiding this comment

oliver-sanders commented Dec 30, 2020

oliver-sanders commented Jan 6, 2021 • edited by dwsutherland Loading

kinow left a comment • edited Loading

Choose a reason for hiding this comment

hjoliver commented Jan 11, 2021

oliver-sanders commented Jan 19, 2021

dwsutherland commented Nov 12, 2020 •

edited

Loading

dwsutherland commented Dec 18, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020 •

edited

Loading

dwsutherland commented Dec 21, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020 •

edited

Loading

hjoliver commented Dec 21, 2020 •

edited

Loading

oliver-sanders left a comment •

edited

Loading

dwsutherland Jan 14, 2021 •

edited

Loading

dwsutherland Jan 14, 2021 •

edited

Loading

oliver-sanders Jan 15, 2021 •

edited

Loading

oliver-sanders Jan 15, 2021 •

edited

Loading

oliver-sanders commented Jan 6, 2021 •

edited by dwsutherland

Loading

kinow left a comment •

edited

Loading