Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes formatting in lineage README #181

Merged
merged 1 commit into from
May 25, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions examples/lineage/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ straightforward way to store evolutions of your code and configuration, and ther
By versioning code, you are therefore versioning lineage information. This means you can go back in time and ask
questions about the past. For example, you can ask what the lineage information was for a model that was trained at
a specific point in time. This is as simple as checking out the git SHA of the code that produced the model, and
asking Hamilton to visualize the DAG and ask questions of it.
asking Hamilton to visualize (e.g. see [`visualize_execution()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.visualize_execution)), the DAG and ask questions of it.

### Auditing and Compliance
The [`@tag`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/decorators.html#tag) and (`@tag_outputs`) feature allows
Expand All @@ -102,9 +102,10 @@ nodes with certain tags, e.g. get me all my "sources", or "what is PII, and what
The [Hamilton Driver](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver)
has a lot of functions that allow you to ask questions of your DAGs to make (1) easier. The driver code can be run in
a CI/CD system, or as part of a reporting/auditing pipeline. For example, you can ask:
* What are all the functions and their tags via [`list_available_variables()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.list_available_variables)
* What are the possible places that consume the output of this function via [`what_is_downstream_of()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.what_is_downstream_of)
* What are the possible sources that feed into this function via [`what_is_upstream_of()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.what_is_upstream_of)

* What are all the functions and their tags via [`list_available_variables()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.list_available_variables)
* What are the possible places that consume the output of this function via [`what_is_downstream_of()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.what_is_downstream_of)
* What are the possible sources that feed into this function via [`what_is_upstream_of()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.what_is_upstream_of)

With these three functions you can the find functions with specific tags and then ask questions in relation to them.

Expand All @@ -114,6 +115,13 @@ troubleshooting and debugging.

Debugging is methodical and procedural with Hamilton. The way functions are written
and executed mean that one can easily walk through just the part of the DAG of interest to debug an issue.
To help with this, Hamilton has various methods to visualize lineage so you can more easily see what you're walking through connects to:

* [`display_all_functions()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.display_all_functions)
* [`display_downstream_of()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.display_downstream_of)
* [`display_upstream_of()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.display_upstream_of)
* [`visualize_execution()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.visualize_execution)
* [`visualize_path_between()`](https://hamilton.readthedocs.io/en/latest/reference/api-reference/drivers.html#hamilton.driver.Driver.visualize_path_between)

### Collaboration
When any organization scales, or has personnel changes, it's important to have a system that helps people get up to
Expand Down