Add long_callback decorator

[jonmmease]

This PR adds the @long_callback decorator the was developed in Dash Labs.

Draft documentation below

TODO:

• Integration tests
• long_callback docstring
• Merge Add support for adding "extra components" to an app's layout #1700 and resolve conflicts

Overview

This decorator is designed to make it easier to create callback functions that take a long time to run, without locking up the Dash app or timing out.

@long_callback is designed to support multiple backend executors. Two backends are currently implemented:

• A diskcache backend that runs callback logic in a separate process and stores the results to disk using the diskcache library. This is the easiest backend to use for local development.
• A Celery backend that runs callback logic in a celery worker and returns results to the Dash app through a Celery broker like RabbitMQ or Redis.

The @long_callback decorator requires a long callback manager instance. This manager instance may be provided to the dash.Dash app constructor as the long_callback_manager keyword argument. Or, it may be provided as the manager argument to the @app.long_callback decorator itself.

The @app.long_callback decorator supports the same arguments as the normal @app.callback decorator, but also includes support for several additional optional arguments that will be discussed below: manager, running, cancel, progress, and progress_default.

Dependencies

The examples below use the discache manager, which requires the diskcache, multiprocess, and psutil libraries

$ pip install diskcache multiprocess psutil

Example 1: Simple background callback

Here is a simple example of using the @long_callback decorator to register a callback function that updates an html.P element with the number of times that a button has been clicked. The callback uses time.sleep to simulate a long-running operation.

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    manager=long_callback_manager,
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 2: Disable button while callback is running

In the previous example, there is no visual indication that the long callback was running. It is also possible to click the "Run Job!" button multiple times before the original job has the chance to complete. This example addresses these shortcomings by disabling the button while the callback is running, and re-enabling it when the callback completes.

This is accomplished using the running argument to @long_callback. This argument accepts a list of 3-element tuples. The first element of each tuple should be an Output dependency object referencing a property of a component in the app layout. The second elements is the value that the property should be set to while the callback is running, and the third element is the value the property should be set to when the callback completes.

This example uses running to set the disabled property of the button to True while the callback is running, and False when it completes. In this example, the long callback manager is provided to the dash.Dash app constructor instead of the @app.long_callback decorator.

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache

cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
    ],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 3: Cancelable callback

This example builds on the previous example, adding support for canceling a long-running callback using the cancel argument to the @long_callback decorator. The cancel argument should be set to a list of Input dependency objects that reference a property of a component in the app's layout. When the value of this property changes while a callback is running, the callback is canceled. Note that the value of the property is not significant, any change in value will result in the cancellation of the running job (if any).

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache

cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 4: Progress bar

This example uses the progress argument to the @long_callback decorator to update a progress bar while the callback is running. The progress argument should be set to an Output dependency grouping that references properties of components in the app's layout.

When a dependency grouping is assigned to the progress argument of @long_callback, the decorated function will be called with a new special argument as the first argument to the function. This special argument, named set_progress in the example below, is a function handle that the decorated function should call in order to provide updates to the app on its current progress. The set_progress function accepts a single argument, which correspond to the grouping of properties specified in the Output dependency grouping passed to the progress argument of @long_callback.

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                html.Progress(id="progress_bar"),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
        (
            Output("paragraph_id", "style"),
            {"visibility": "hidden"},
            {"visibility": "visible"},
        ),
        (
            Output("progress_bar", "style"),
            {"visibility": "visible"},
            {"visibility": "hidden"},
        ),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
    progress=[Output("progress_bar", "value"), Output("progress_bar", "max")],
)
def callback(set_progress, n_clicks):
    total = 10
    for i in range(total):
        time.sleep(0.5)
        set_progress((str(i + 1), str(total)))
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 5: Progress bar chart graph

The progress argument to the @long_callback decorator can be used to update arbitrary component properties. This example creates and updates a plotly bar graph to display the current calculation status. This example also uses the progress_default argument to long_callback to specify a grouping of values that should be assigned to the components specified by the progress argument when the callback is not in progress. If progress_default is not provided, all the dependency properties specified in progress will be set to None when the callback is not running. In this case, progress_default is set to a figure with a zero width bar.

import time
import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output
import plotly.graph_objects as go

## Diskcache
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

def make_progress_graph(progress, total):
    progress_graph = (
        go.Figure(data=[go.Bar(x=[progress])])
        .update_xaxes(range=[0, total])
        .update_yaxes(
            showticklabels=False,
        )
        .update_layout(height=100, margin=dict(t=20, b=40))
    )
    return progress_graph


app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                dcc.Graph(id="progress_bar_graph", figure=make_progress_graph(0, 10)),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
        (
            Output("paragraph_id", "style"),
            {"visibility": "hidden"},
            {"visibility": "visible"},
        ),
        (
            Output("progress_bar_graph", "style"),
            {"visibility": "visible"},
            {"visibility": "hidden"},
        ),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
    progress=Output("progress_bar_graph", "figure"),
    progress_default=make_progress_graph(0, 10),
    interval=1000,
)
def callback(set_progress, n_clicks):
    total = 10
    for i in range(total):
        time.sleep(0.5)
        set_progress(make_progress_graph(i, 10))

    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Caching results with long_callback

The long_callback decorator can optionally memoize callback function results through caching, and it provides a flexible API for configuring when cached results may be reused.

Note: The current caching configuration API is fairly low-level, and in the future it might be useful to provide preconfigured caching profiles.

How it works

Here is a high-level description of how caching works in long_callback. Conceptually, you can imagine a dictionary is associated with each decorated callback function. Each time the decorated function is called, the input arguments to the function (and potentially other information about the environment) are hashed to generate a key. The long_callback decorator then checks the dictionary to see if there is already a value stored using this key. If so, the decorated function is not called, and the cached result is returned. If not, the function is called and the result is stored in the dictionary using the associated key.

The built-in functools.lru_cache decorator uses a Python dict just like this. The situation is slightly more complicated with Dash for two reasons:

1. We might want the cache to persist across server restarts.
2. When an app is served using multiple processes (e.g. multiple gunicorn workers on a single server, or multiple servers behind a load balancer), we might want to shared cached values across all of these processes.

For these reasons, a simple Python dict is not a suitable storage container for caching Dash callbacks. Instead, long_callback uses the current diskcache or Celery callback manager to store cached results.

Caching flexibility requirements

To support caching in a variety of development and production use cases, long_callback may be configured by one or more zero-argument functions, where the return values of these functions are combined with the function input arguments when generating the cache key. Several common use-cases will be described below.

Enabling caching

Caching is enabled by providing one or more zero-argument functions to the cache_by argument of long_callback. These functions are called each time the status of a long_callback function is checked, and their return values are hashed as part of the cache key.

Here is an example using the diskcache callback manager. In this example, the cache_by argument is set to a lambda function that returns a fixed UUID that is randomly generated during app initialization. The implication of this cache_by function is that the cache is shared across all invocations of the callback across all user sessions that are handled by a single server instance. Each time a server process is restarted, the cache is cleared an a new UUID is generated.

import time
from uuid import uuid4
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache
launch_uid = uuid4()
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(
    cache, cache_by=[lambda: launch_uid], expire=60,
)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)
app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)


@app.long_callback(
    output=(Output("paragraph_id", "children"), Output("button_id", "n_clicks")),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"], (n_clicks or 0) % 4


if __name__ == "__main__":
    app.run_server(debug=True)

Here you can see that it takes a few seconds to run the callback function, but the cached results are used after n_clicks cycles back around to 0. By interacting with the app in a separate tab, you can see that the cache results are shared across user sessions.

cache_by function workflows

Various cache_by functions can be used to accomplish a variety of caching policies. Here are a few examples:

• A cache_by function could return the file modification time of a dataset to automatically invalidate the cache when an input dataset changes.
• In a Heroku or Dash Enterprise deployment setting, a cache_by function could return the git hash of the app, making it possible to persist the cache across redeploys, but invalidate it when the app's source changes.
• In a Dash Enterprise setting, the cache_by function could return user meta-data to prevent cached values from being shared across authenticated users.

Celery configuration

Here is an example of configuring the CeleryLongCallbackManager for use in @long_callback. This callback manager uses Celery as the execution backend rather than a background process with diskcache.

from dash.long_callback import CeleryLongCallbackManager
from celery import Celery

celery_app = Celery(
    __name__, broker="redis://localhost:6379/0", backend="redis://localhost:6379/1"
)
long_callback_manager = CeleryLongCallbackManager(celery_app)

See the Celery documentation for more information on configuring a Celery app instance.

[LiamConnors]

@jonmmease encountering some issues here getting this up and running on my machine. Tested with examples 1 and 5 (using diskcache) and get this:

traceback (most recent call last):
  File "/Users/liamconnors/Documents/GitHub/dash/dash/dash.py", line 1238, in callback
    callback_manager.call_and_register_background_fn(
  File "/Users/liamconnors/Documents/GitHub/dash/dash/long_callback/managers/diskcache_manager.py", line 74, in call_and_register_background_fn
    future.start()
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/liamconnors/opt/anaconda3/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'make_update_cache.<locals>._callback'

Currently trying the Celery option.

[jonmmease]

@LiamConnors Can you try change this to if True: and install the multiprocess library and see if that makes a difference?

[LiamConnors]

Thanks @jonmmease. Yep, that worked.

[jonmmease]

Great! I went ahead and made this change, so all platforms will use this logic path.

[jonmmease]

Note: n_clicks needs to be set back to 0 in the output so that it's value is always zero when used as part of the cache key. Otherwise, every incremented value of n_clicks invalidates the cache.

[alexcjohnson]

Not too cumbersome but does feel a bit hacky, as discussed this morning you're going to try and find a way to exclude certain args from the cache key?

[jonmmease]

@alexcjohnson @chriddyp

This is ready for review, with a few issues still to consider.

1. I've added integration tests that use the diskcache long callback manager, but I don't know what to do about testing the Celery manager since celery requires an external celery process as well as an external executor. I'd appreciate suggestions here.

2. We haven't really discussed the API. The API I have here is to pass the long callback manager object as the first argument to @app.long_callback. After this first argument, the remaining arguments are compatible with @app.callback.

3. I noticed a kind of workflow quirk with caching that I wanted to point out. Right now, the callback function source code and callback function arguments are always used as part of the cache key. Then the return value of all of the cache_by functions is included in the hash of the key as well. The quirk is that a common work flow would be to trigger a long-running callback by clicking a button. But if the callback just has the n_clicks property of a button as an Input, then this value is part of the cache key, and so the cache key will change as the n_clicks property increments. To work around this in the test noted above, I have the callback output to the n_clicks property to set it back to zero. This works, but I wanted to point it out in case other folks have thoughts on the situation.

[nicolaskruchten]

random API idea... another decorator that stacks onto @app.callback that just contains the long stuff?

@app.long(...)
@app.callback(...)
def thing():
    return thing

[alexcjohnson]

One limitation I think this has is you can't use a long callback with MATCH wildcards - because the wrapper adds non-wildcard outputs to the real callbacks. Seems fine to me, that's a weird case that we could find a way around (likely involving a helper to create more of the extra components with appropriate pieces of their ID provided by the user when they create the corresponding inputs & outputs) but I wonder if we need to catch this in the wrapper and throw an error the user will understand?

Relatedly: what happens if the inputs and outputs you're attaching this to don't exist on the page in the initial layout? Then you have some partially-defined inputs & outputs, which will be an error, right? This seems like a common requirement, do we need a way to pull the extra components out and add them to the layout manually?

[jonmmease]

Thanks @alexcjohnson, those are both good questions. I'm working on getting celery tests up and running, but then I'll think about these more carefully.

[LiamConnors]

@jonmmease I understand Dash 2.0 is going to have a simplified import statement (https://github.com/plotly/ddk-dash-docs/issues/41)

Are Long Callbacks part of that? Does that mean, example 1 import statement above should look like this in the docs for diskcache examples?

import time
from dash import Dash, html, DiskcacheLongCallbackManager, Input, Output
import diskcache

[jonmmease]

We haven't talked about whether to elevate DiskcacheLongCallbackManager to the top-level. I'd personally be inclined to leave it under dash.long_callback since it won't be as common of an import.

[jonmmease]

Summary of updates since Friday

• The long callback manager may now be provided either as the long_callback_manager argument to dash.Dash, or as the manager keyword argument to @app.long_callback.
• @app.long_callback Has a new optional cache_args_to_ignore keyword argument. docstring:

    Arguments to ignore when caching is enabled. If callback is configured
    with keyword arguments (Input/State provided in a dict),
    this should be a list of argument names as strings. Otherwise,
    this should be a list of argument indices as integers.

• Tests were refactored to run all of the long_callback tests with both the diskcache and celery long callback managers.
• Added validation to raise an informative error when pattern matching ids are provided to @app.long_callback.
• I did a bunch of QA and experiments which led to some internal refactoring to make things coordinate properly across processes (on gunicorn). I also ran through the documentation examples on Windows to make sure things are still working there.

Loose ends:

• Still need to resolve conflicts after Add support for adding "extra components" to an app's layout #1700 is merged into dev.

Relatedly: what happens if the inputs and outputs you're attaching this to don't exist on the page in the initial layout? Then you have some partially-defined inputs & outputs, which will be an error, right? This seems like a common requirement, do we need a way to pull the extra components out and add them to the layout manually?

@alexcjohnson I'm not sure I understand this one. The extra components that long callback adds to the layout will always be there from the start. If some of the other input/output components aren't on the page yet, wouldn't the situation be the same as for a regular callback, where you'd need to suppress the callback errors?

[alexcjohnson]

If some of the other input/output components aren't on the page yet, wouldn't the situation be the same as for a regular callback, where you'd need to suppress the callback errors?

try it out, but I think suppressing only works when all the inputs and outputs are missing. When only some are present it’s still an error, right?

[jonmmease]

Yup, you're right. I didn't realize that suppress_callback_exceptions only works when all of the dependencies of a callback are missing. Would it make sense for it to also handle this case?

[jonmmease]

Ok, I got one approach working.

• The extra components are now added to the validation_layout as well as layout.
• When prevent_initial_call=True, the associated interval is initialized as disabled.

So now a long_callback can reference components not in the initial layout if:

• The components are part of validation_layout
• prevent_initial_call=True

How does that sound?

---

I also added a test that does this, and makes sure there are no errors logged (test_lcbc007_validation_layout)

[jonmmease]

cc @chriddyp for opinion on whether the approach above is a reasonable enough workflow for using @app.long_callback with components that aren't in the initial layout.

[alexcjohnson]

Even though the first sentence should be clear, the second sentence makes it sound like you can just give the class, like long_callback_manager=DiskcacheLongCallbackManager

[jonmmease]

Good point. updated in 3892ecb (apologies for the bogus commit message)

[alexcjohnson]

So now a long_callback can reference components not in the initial layout if:

• The components are part of validation_layout
• prevent_initial_call=True

How does that sound?

Works for me. Longer term I think we'll want to find a nice way to relax some of these constraints, but that's going to take some thought to do right.

💃