Skip to content

Commit

Permalink
feat: First-class caching (#1104)
Browse files Browse the repository at this point in the history
Caching is now a core Hamilton feature. You can add caching
by using `Builder().with_cache()` and the cache is accessible
via the `Driver.cache` attribute.

The feature is implemented using lifecycle hooks, but this could
change in Hamilton 2.0.

This also introduces the concept of `MetadataStore` and `ResultStore`
which could be further reused with the Hamilton UI or in 
specific deployment scenarios.

This also includes a bug fix for the `@dataloader` decorator which
had accidentally flipped two tags compared to `@load_from` and `from_.`.
The fix includes changes to `@dataloader`, a related test, and to the
visualization function in `graph.create_graphviz_graph`.

The reference documentation page `caching-logic.rst` includes many
TODOs, quirks, limitations, and roadmap items. Eventually, we will need
to deprecate the other caching mechanisms.

---------

Co-authored-by: zilto <tjean@DESKTOP-V6JDCS2>
  • Loading branch information
zilto and zilto authored Oct 3, 2024
1 parent c400290 commit a6b3898
Show file tree
Hide file tree
Showing 48 changed files with 15,943 additions and 22 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -152,3 +152,6 @@ examples/**/hamilton-env
**.pkl
**.lance
**.txn

# hamilton default caching directory
**/.hamilton_cache/
167 changes: 167 additions & 0 deletions docs/concepts/_caching/view_run_example.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions docs/concepts/builder.rst
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,24 @@ Adds `DataSaver` and `DataLoader` nodes to your dataflow. This allows to visuali
:align: center


with_cache()
--------------

This enables Hamilton's caching feature, which allows to automatically store intermediary results and reuse them in subsequent executions to skip computations. Learn more in the :doc:`/concepts/caching` section.

.. code-block:: python
from hamilton import driver
import my_dataflow
dr = (
driver.Builder()
.with_modules(my_dataflow)
.with_cache()
.build()
)
with_adapters()
---------------

Expand Down
Loading

0 comments on commit a6b3898

Please sign in to comment.