support circular dependencies

[alexcjohnson]

Circular deps keep coming up. Most of the use cases I've seen are fairly simple, like 2 alternate ways to accomplish the same task, that you want to keep in sync. For example today on the community board someone asked the perfectly reasonable question of how to make the selected tab reflected in the url, and vice versa so you could load your app with the desired tab open. https://community.plot.ly/t/combining-dcc-tabs-and-dcc-location/27831

The code they tried seems like the natural way to do it, but fails as it requires a circular dependency between Location.pathname and Tabs.value. I don't see a simple alternative. We could make one using patterns we've used elsewhere (like adding a *_timestamp prop to one of these components, or having Location.pathname update href and vice versa, which it should arguably do) but this feels hacky and fragile - it's making an explicit circular ref into an implicit one that may make infinite loops that are hard to debug and even harder to fix.

---

My proposal: create a new dash.dependencies class, tentatively CircularOutput, by which users could specify that this particular output may be in a circular dependency chain. Then in the renderer we would:

• Prevent unchanged values from dispatching callbacks. I can't think of a reason ever to allow this, anyone want to argue with that? See [BUG] Callback with dcc.Location 'search' input is fired 2 times  #883 where I first proposed this - also would help with things like [BUG] Loading + relayoutData => infinite events dash-core-components#608, and once we have this handled in the renderer we should be able to remove the corresponding checks from component code, making it simpler and easier to write robust components.
• Allow circular deps if somewhere in the chain there's a CircularOutput
• Keep track of the chain of props that led from a user action (or, more generally, a browser event, like for Interval components) to the output. This would be useful anyway to show with the stack trace when debugging errors.
• When a CircularOutput is changed, look back at this chain; if the same CircularOutput is already in the chain (meaning this would be its second change in response to the same stimulus) consider this a possible infinite loop and throw an error.

[rpkyle]

This would be useful anyway to show with the stack trace when debugging errors.

This is an interesting idea; how would this look in practice? Would each callback’s props and their values optionally appear in the stack trace — just before the sequence of calls which follows them?

[alexcjohnson]

Would each callback’s props and their values optionally appear in the stack trace — just before the sequence of calls which follows them?

Something like that - probably in an expandable section. I think we should be able to manage all this purely on the front end, without needing to send anything new (or make any code changes) to the back ends.

[chriddyp]

👍 . There is also a case for using circular dependencies to sync up dcc.Store with controls so that you can access the the control values (via dcc.Store) in separate tabs or pages. It'd be circular so that when you'd render the initial page, dcc.Store would sync the value back to the original component.

There's another use case around crossfiltering - with the right property design, syncing or providing the union of selected points could be a lot simpler.

I wrote about some of these use cases in #844.

[alexcjohnson]

Ah thanks @chriddyp - I forgot about #844, really nice collection of use cases. We could merge the two issues if you like, though TBH it might be easier to leave that one as a smörgåsbord to pick and choose from and have the discussion over here.

I feel like Synced covers a number of common use cases, but a will hit a wall of real-world complexity it'll be hard to get past; whereas if we just extend the existing callback formalism we naturally have access to its full power. Some of these you mentioned in #884:

• Non-identity linkages like unit conversion, type conversion: Some of these will probably need to stay in Python for full flexibility, but I bet a lot of them will be amenable to magic-method-based automatic transpiling to JS (we've talked about this before though I can't find it now)
• Self-consistency of callback cycles: when we've explicitly defined a CircularOutput, it's natural to take that one as the lowest-priority item in the loop, and start evaluation from each prop that's not an Output of anything, even if it's a CircularOutput.

But also:

• Loops bigger than two nodes. I don't have a simple use case here but I bet this will come up for example in iterative model tuning.

• Multiple inputs: let's say we have four inputs:

  • height
  • width
  • lock-aspect-ratio (checkbox)
  • aspect ratio (number)

  height and width would make a loop, but their callbacks would depend on the other two as inputs (plus either PreventUpdate or itself as State for the unlocked case)

[alexcjohnson]

Prevent unchanged values from dispatching callbacks. I can't think of a reason ever to allow this, anyone want to argue with that? See #883

Just came across a use case for this - extendData for graphs. https://github.com/plotly/dash-core-components/pull/621/files#r318682728 - this is a funny one, it's a prop that really just interfaces to an imperative command for adding data to a graph incrementally. So if we were to do this we would need to find a way to exclude extendData, and see if there are any other props in this mold.

Actually, perhaps there's an easy solution at the component level: clear extendData (setProps({extendData: null})?) once it has been incorporated in the figure. Hopefully nobody is listening to extendData as an input, that would be weird... but perhaps it would even still work, they'd just get the new data and then another call with null.

Seems like that approach would work for other imperative props, as long as there's a suitable noop value to assign it (could be null but not necessarily; you could have a multiplyBy prop whose noop value is 1 for example)

[kylemcmearty]

@chriddyp "There is also a case for using circular dependencies to sync up dcc.Store with controls so that you can access the the control values (via dcc.Store) in separate tabs or pages. It'd be circular so that when you'd render the initial page, dcc.Store would sync the value back to the original component."

Has there been any updates on circular dependencies and dcc.store? I have an input value / slider (A) that I use to calculate a dataframe and store that data into a dcc.store. The problem is that I want to use the dataframe from dcc.store in a callback that uses input (A) as an Output, thus creating a circular dependency. Is this something at all related to what you were mentioning about syncing up dcc.store with controls?

[sdementen]

In order to allow cycles in the graph yet avoid infinite loops in the callbacks, could the following "marking" solution work ?

• keep in the frontend a triggered_threads=defaultdict(list) that will associate triggered_thread_id -> [dependency (component_id.component_property), ...]
• whenever the frontend triggers a callback (UI or Interval) via a dependency:
  1. we create a triggered_thread_id as a uuid and add to its associated list the dependency that triggered the callback
  2. we call the callbacks (adding if needed the triggered_thread_id to the callback_context)
  3. for all outputs of the callbacks that have been updated (ie not no_update)
     • if the dependency was already in the list triggered_threads[triggered_thread_id], we do not trigger the potential new callbacks
     • otherwise, we add the dependency to the list triggered_threads[triggered_thread_id] and continue handling the dependencies
  4. once, there is no more callbacks triggered, we clean the dict from its triggered_thread_id key

I haven't tested the logic but I could do it (in python with synthetic dependency graphs) if needed.
Did you already try something in this vein ?

[emilhe]

I would also be very interested in a generic solution :). Until then, i have made a custom Monitor component, which makes it possible to lift state into the monitor, thereby avoiding the circular dependency error. Here is an example of keeping a Slider and an Input component in sync using dash-extensions==0.0.33,

import dash_core_components as dcc
from dash import Dash
from dash.dependencies import Input, Output
from dash_extensions import Monitor

app = Dash()
app.layout = Monitor([
    dcc.Input(id="input", autoComplete="off", type="number", min=0, max=100, value=50),
    dcc.Slider(id="slider", min=0, max=100, value=50)],
    probes=dict(probe=[dict(id="input", prop="value"), dict(id="slider", prop="value")]), id="monitor")


@app.callback([Output("input", "value"), Output("slider", "value")], [Input("monitor", "data")])
def sync(data):
    probe = data["probe"]
    return probe["value"], probe["value"]


if __name__ == '__main__':
    app.run_server()

The syntax is more involved than a native solution though, and only components in the Monitor component tree can be monitored. As all of there children are visited at each render, performance might also become an issue at some point.

[rusiano]

+1. I am also interested in this.

[chadaeschliman]

What about a much more limited approach:

• Circularity is only allowed in the special case of the same value(s) appearing as both an Input and Output of the same callback
• Output values which match an Input value would not trigger a call to the same callback again

For something like a linked slider and input, the paradigm would be to pass both values as Inputs and Outputs and use callback_context to determine which value to propagate to the outputs. More complicated examples like length/width/aspect_ratio would involve more Inputs and Outputs but would follow the same paradigm.

The advantages to this approach are:

• It pushes any complexity or ambiguity on how to resolve the circularity into the callback rather than dash attempting to resolve it
• It supports arbitrarily complex circularity by simply passing all required inputs and outputs to a single function. Probably not good practice, but the capability would be there if needed.
• It still prevents multi-callback circularity which could be unintentional and is more difficult to reason about and debug
• There would be no change in functionality for existing working code
• There is probably no need to introduce a new class since having the same value as both an Input and Output to the same callback is already pretty explicit.

[romanodev]

I would be very interested in it as well. My use case is the following:

I run python code in a container and, as a result, I get an iterator that gives standard output as it comes. I need to visualize this stream into the app. I am trying to do this via two callbacks that call each other with this iterator (not hashable) being a global variable. Maybe there is a better way to do this.

UPDATE: I found the answer here: https://community.plotly.com/t/how-to-turn-off-interval-event/5565/7