feat(core): improve entitlements performance

[suchak1]

Context:

While developing COP, we found that GetEntitlements could take around 5.5 seconds to return a response for the federal dataset. This forced us to increase the server timeout, increase the grpc message size, and implement caches.

After investigating the latency, we determined the two main causes: excess logs (33%) and excess database queries (66%).

Proposed Solution:

Primary

1. In the case of logging subject mappings, we now log their count instead of their content.
2. In the case of database calls, we now list attributes, list subject mappings, and match them based on their values. The database calls were O(n) time complexity because they were dependent on the number of attribute values. Now the database calls are constant time. We still loop through all the values to match their subject mappings; however we were already doing that in the prepareValues values function, so the new approach is strictly better (especially due to our ubiquitous use of maps).

Rego Query Optimization:

Yet if we simply match subject mappings and attribute values, the rego query becomes massive (65 mb). It takes 3 seconds to build (20%) and evaluate (80%). To optimize for not only time but also space, we remove unrelated values for each fqn/attribute pair in the rego input (unless the attribute rule is hierarchical).

After all the optimizations, fetching entitlements using the federal dataset now takes about 125 ms. This is a latency reduction of 98%.

resolves: #1259

[github-actions]

Warning

This pull request does not reference any issues. Please add a reference to an issue in the body of the pull request description.

[jrschumacher]

@suchak1 this looks good. Could we add some comments to make sure others understand what's happening?

[jrschumacher]

This looks good to me.

[strantalis]

@suchak1 Would the MatchSubjectMappings rpc have helped here?

[suchak1]

No, since the MatchSubjectMappings rpc just matches SubjectConditionSets with SubjectMappings and returns those SubjectMappings based on the provided SubjectConditionSets conditions.