feat(docs): tutorial for writing a custom transformer

docs-website/sidebars.js

@@ -22,6 +22,8 @@ function list_ids_in_directory(directory) {
   return ids;
 }
 
+// note: to handle errors where you don't want a markdown file in the sidebar, add it as a comment.
+// this will fix errors like `Error: File not accounted for in sidebar: ...`
 module.exports = {
   // users
   // architects
@@ -73,6 +75,7 @@ module.exports = {
       "docs/docker/development",
       "metadata-ingestion/adding-source",
       "metadata-ingestion/s3-ingestion",
+      //"metadata-ingestion/examples/transforms/README"
       "metadata-ingestion/transformers",
       //"docs/what/graph",
       //"docs/what/search-index",

metadata-ingestion/README.md

@@ -950,7 +950,7 @@ sink:
 
 If you'd like to modify data before it reaches the ingestion sinks – for instance, adding additional owners or tags – you can use a transformer to write your own module and integrate it with DataHub.
 
-Check out the [transformers guide](./transformers.md) for more info!.
+Check out the [transformers guide](./transformers.md) for more info!
 
 ## Using as a library
 
@@ -1018,4 +1018,4 @@ In order to use this example, you must first configure the Datahub hook. Like in
 
 ## Developing
 
-See the [developing guide](./developing.md), [adding a source guide](./adding-source.md) and the [using transformers](./transformers.md) guides.
+See the guides on [developing](./developing.md), [adding a source](./adding-source.md) and the [using transformers](./transformers.md).

metadata-ingestion/examples/transforms/README.md

@@ -0,0 +1,5 @@
+# Custom transformer script
+
+This script sets up a transformer that reads in a list of owner URNs from a JSON file specified via `owners_json` and appends these owners to every MCE.
+
+See the transformers tutorial (https://datahubproject.io/docs/metadata-ingestion/transformers) for how this module is built and run.

metadata-ingestion/transformers.md

@@ -25,11 +25,14 @@ transformers:
         - "urn:li:tag:Legacy"
 ```
 
-:::tip
-
 If you'd like to add more complex logic for assigning tags, you can use the more generic add_dataset_tags transformer, which calls a user-provided function to determine the tags for each dataset.
 
-:::
+```yaml
+transformers:
+  - type: "add_dataset_tags"
+    config:
+      get_tags_to_add: "<your_module>.<your_function>"
+```
 
 ### Setting ownership
 
@@ -47,6 +50,15 @@ transformers:
         - "urn:li:corpGroup:groupname"
 ```
 
+If you'd like to add more complex logic for assigning ownership, you can use the more generic `add_dataset_ownership` transformer, which calls a user-provided function to determine the ownership of each dataset.
+
+```yaml
+transformers:
+  - type: "add_dataset_ownership"
+    config:
+      get_owners_to_add: "<your_module>.<your_function>"
+```
+
 ## Writing a custom transformer from scratch
 
 In the above couple of examples, we use classes that have already been implemented in the ingestion framework. However, it’s common for more advanced cases to pop up where custom code is required, for instance if you'd like to utilize conditional logic or rewrite properties. In such cases, we can add our own modules and define the arguments it takes as a custom transformer.
@@ -182,8 +194,10 @@ def transform_one(self, mce: MetadataChangeEventClass) -> MetadataChangeEventCla
 
 ### Installing the package
 
-Now that we've defined the transformer, we need to make it visible to DataHub. This can be done by making sure the Python file is available as a local import.
+Now that we've defined the transformer, we need to make it visible to DataHub. The easiest way to do this is to just place it in the same directory as your recipe, in which case the module name is the same as the file – in this case, `custom_transform_example`.
 
+<details>
+  <summary>Advanced: installing as a package</summary>
 Alternatively, create a `setup.py` in the same directory as our transform script to make it visible globally. After installing this package (e.g. with `python setup.py` or `pip install -e .`), our module will be installed and importable as `custom_transform_example`.
 
 ```python
@@ -198,6 +212,8 @@ setup(
 )
 ```
 
+</details>
+
 ### Running the transform
 
 ```yaml