Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Policy: improve attribute FQN re-indexing #1603

Open
jakedoublev opened this issue Oct 2, 2024 · 0 comments · May be fixed by #1679
Open

Policy: improve attribute FQN re-indexing #1603

jakedoublev opened this issue Oct 2, 2024 · 0 comments · May be fixed by #1679
Labels
comp:policy Policy Configuration ( attributes, subject mappings, resource mappings, kas registry)

Comments

@jakedoublev
Copy link
Contributor

Background

FQNs are utilized within TDFs and are the identifiers of attributes outside of the policy administration/database. There is a concept in the platform of attribute FQN reindexing. FQNs in the lookup table can get out of sync if:

  1. an update was made to an attribute object (namespace name, definition name, value value) via RPC but there was a connection issue and the upsert to replace the FQNs containing the change failed
  2. a manual database change was made altering any of the same policy objects

A functionality was added to the platform binary to reindex FQNs and there is an open issue to add it to otdfctl.

However, some potential issues have become clear over time.

  1. It should be possible to run a re-index on an interval

Because reindexing is actually crucial for accessing TDFs containing the intended updated state of the FQN, consistency could be guaranteed at said interval.

  1. The upsert queries have room for optimization and efficiency

The upsert db query for a namespace only updates the https://<namespace> FQN, the definition upsert query only updates the rows matching https://<namespace>/attr/<attribute definition name>, and so on.

It is not a problem because the CLI reindex command runs every query for every item, but optimization will be more important as at-scale deployments of the platform are realized and the volume of items in the database goes up.

An improvement was recently unblocked by the SQLC migration see comments.

Acceptance Criteria

  1. FQN reindexing happens on a configurable interval (jobs/workflows approach?)
  2. FQN namespace upsert query overwrites the new namespace name in every FQN built from the namespace ID/name being upserted
  3. FQN definition upsert query overwrites the new namespace name in every FQN built from the definition ID/name being upserted
  4. FQN value upsert query overwrites the new namespace name in every FQN built from the value ID/name being upserted
@jakedoublev jakedoublev added the comp:policy Policy Configuration ( attributes, subject mappings, resource mappings, kas registry) label Oct 2, 2024
@ryanulit ryanulit linked a pull request Oct 23, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:policy Policy Configuration ( attributes, subject mappings, resource mappings, kas registry)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant