Include data_use and data_category metadata in upload of access results

[adamsachs]

Closes #3266

Description Of Changes

We want to include data_use and data_category metadata as part of the output package for access requests. In order to do this, we need to make that metadata available to the upload() functions that produce and serialize the output packages. This PR updates the request execution internals to pass those pieces of metadata to the appropriate upload() function.

The data_category metadata was already available as the data_category_field_mapping attribute on the DatasetGraph that's used during privacy request execution -- the update here is simply to pass that along as another argument to the upload() function.

The data_use metadata is not something we'd been keeping track of at all in privacy request execution. A new data structure has been created to maintain that metadata - a dict that maps each Collection in the graph traversal to a set of data_uses associated with the Collection, where the association is determined by the Collection --> ConnectionConfig --> System --> DataUse relationship chain. That dict is populated and stored in the redis cache with a key unique to the particular privacy request as request execution beings; the cache entry is then retrieved right before upload() is called at the end of request execution.

Code Changes

• pass along the DatasetGraph.data_category_field_mapping as an argument to upload()
• populate a "data use map" and store it in the redis cache, before request execution, with a key that's unique to the privacy request
• retrieve the "data use map" that's stored in the redis cache and pass it along as an argument to upload()

Steps to Confirm

• hard to confirm this functionality manually until we actually make use of the new metadata in the output package
• in the meantime, we should just confirm manually that request execution does not regress with these additions!

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated

[cypress]

Passing run #2951 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge f4e8b98 into 0337c66...
Project: fides	Commit: 8257cead6e ℹ️
Status: Passed	Duration: 00:47 💡
Started: Jun 28, 2023 6:17 PM	Ended: Jun 28, 2023 6:18 PM

_{This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.}

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.01 🎉

Comparison is base (0337c66) 87.09% compared to head (f4e8b98) 87.10%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3674      +/-   ##
==========================================
+ Coverage   87.09%   87.10%   +0.01%     
==========================================
  Files         310      310              
  Lines       19033    19048      +15     
  Branches     2437     2438       +1     
==========================================
+ Hits        16576    16592      +16     
  Misses       2028     2028              
+ Partials      429      428       -1     

Impacted Files	Coverage Δ
.../service/privacy_request/request_runner_service.py	77.17% <ø> (ø)
src/fides/api/graph/graph.py	94.44% <100.00%> (+0.05%)	⬆️
src/fides/api/models/privacy_request.py	96.30% <100.00%> (+0.06%)	⬆️
...es/api/service/storage/storage_uploader_service.py	95.00% <100.00%> (+0.12%)	⬆️
src/fides/api/task/graph_task.py	93.15% <100.00%> (+0.40%)	⬆️
src/fides/api/tasks/storage.py	77.50% <100.00%> (+0.28%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[adamsachs]

@galvana let me know if this looks good, based on what we discussed. (side note: do we have follow up issues that track the work of actually putting this data in the output package? if not, i'm happy to create those.)

@pattisdr figured i'd tag you in as well, if you have a moment, to make sure you don't notice any red flags with the enhancement to privacy request execution and a new cache entry!

[pattisdr]

starting review!

[pattisdr]

good work, all comments are minor. I like the incremental nature of this PR, first collecting this, before we actually start using the new metadata in the output package.

[galvana]

What a quick turnaround! This looks good and is consistent with what we discussed. The only part we might have to change in the future is the structure of the data use and data category maps but we won't know the ideal structure until we finalize the design for DSR package (collection address as key instead of the data category for the data category map).

[adamsachs]

What a quick turnaround! This looks good and is consistent with what we discussed. The only part we might have to change in the future is the structure of the data use and data category maps but we won't know the ideal structure until we finalize the design for DSR package (collection address as key instead of the data category for the data category map).

thanks @galvana, that makes sense! should be relatively easy to switch that around if/when needed.