Aggressive Memory Pruning

[elijahbenizzy]

Is your feature request related to a problem? Please describe.
Currently the Hamilton Executor holds onto everything until the end. This is problematic if one is processing immutable, large objects.

Describe the solution you'd like
Hamilton should prune everything that's not needed. Two levels of improvements:

1. At every step of the DAG, do a GC. That is, determine everything whose depended on nodes are (a) in the critical path and (b) have been computed, deleting them from the results dictionary. We should be able to do this when we compute it -- loop through its dependencies and compute:

   hamilton/hamilton/execution/graph_functions.py

   Line 146 in 28c955e

   value = adapter.execute_node(node_, kwargs)

   . Note we have a PR to do this for parallelism -- its quite a bit simpler, as its just a matter of pruning the nodes that are not needed by the tasks: Garbage collection/memory optimization #374.
2. We could actually compute any chains as one -- E.G. in a loop. A "closed" chain could be fused, so as to not hold onto any memory. This is effectively the same as handling it in the dict, although the dict makes garbage collection a little gnarlier...

So, let's start with (1) and see how that helps us.

[elijahbenizzy]

See #374