Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require UUIDv7 for serviceowner-provided ids #969

Closed
4 tasks
oskogstad opened this issue Aug 6, 2024 · 1 comment
Closed
4 tasks

Require UUIDv7 for serviceowner-provided ids #969

oskogstad opened this issue Aug 6, 2024 · 1 comment
Assignees

Comments

@oskogstad
Copy link
Collaborator

oskogstad commented Aug 6, 2024

Introduction

This change introduces a UUIDv7 requirement for dialogIds, transmissionIds and activityIds

Description

To avoid fragmentation issues caused by the inherit randomness of UUIDv4 values in b-tree indexes, we will require all self-supplied IDs to be UUIDv7, which are stored sequentially and lets eg. joins be perfomed better.

Implementation

A valication routine will have to at a minimum check that the correct version bits are set. We need to consider if the timestamp part (the first 48 bits) should be validated somehow. Only accepting values within a small window relative to server time will not work, as items might be been generated on the server provider side much earlier. A more lax validation (eg. must be after 2010 and not in future) might be more suitable.

Tasks

Preview Give feedback

Threat modelling

Preview Give feedback
@elsand
Copy link
Member

elsand commented Aug 12, 2024

Refinement: There are serious concerns about index fragmentation and performance issues stemming from spamming the b-tree index with UUIDv4 values from Altinn 3/2 instances. This will probably negatively affect the index performance, even if we at a later stage introduce v7 requirements.

We work to have Altinn 3 apps start using UUIDv7 as their PKs. In order to support historic instances, they will need to introduce as "dialogId" column containing the UUIDv7 value. This should be pre-generated and stored before transferring in order to ensure idempotence and avoid duplications.

This change is to be considered breaking, and will have to be communicated with service providers.

@elsand elsand changed the title Discussion: Require UUIDv7 Require UUIDv7 for serviceowner-provided ids Aug 20, 2024
@elsand elsand removed the needs consideration Requires additional consideration label Aug 20, 2024
@oskogstad oskogstad self-assigned this Aug 20, 2024
oskogstad added a commit that referenced this issue Aug 30, 2024
<!--- Provide a general summary of your changes in the Title above -->

## Description

This adds a requirement for user-supplied IDs to be UUIDv7, and that the
timestamp on these are in the past.
While implementing this I stumbled over a bug where the old way of
creating IDs meant they were sorted in the wrong order because GUIDs in
.NET are little endian.
(Sort by ID ascending would have the newest at the top, and oldest at
the bottom)

The idea with UUIDv7 is that the first 6 bytes is the timestamp of
creation, and this helps us avoid a fragmented index
There will only be appends on the index b-tree when creating new
entities, and no inserts further up the index.
If we were to allow old UUIDv4 for the legacy archive imports, the index
would forever be fragmented and have a performance hit.

We now require that all incoming IDs are V7 and big endian (this is the
standard in most other languages)

Sorting by ID ascending on f.ex. DialogID should now list out oldest at
the top, and newest at the bottom.

## Related Issue(s)
- #969 

## Verification

- [ ] **Your** code builds clean without any errors or warnings
- [ ] Manual testing done (required)
- [ ] Relevant automated test added (if you find this hard, leave it and
we'll help out)

## Documentation

- [ ] Documentation is updated (either in `docs`-directory, Altinnpedia
or a separate linked PR in
[altinn-studio-docs.](https://github.com/Altinn/altinn-studio-docs), if
applicable)
@elsand elsand closed this as completed Sep 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants