suggestion: save state from temporal temporal #7253
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Branching off this PR and thinking about a different approach: #7219
Main things I'm trying to focus on:
DefaultJobPersistence
. I think it's important to keep these databases separate in the code.Option 1: Separate Abstraction for "complex" queries in the configs db.
In this PR, I added a new persistence abstraction for the config db, called
ConfigPersistence2
(just ignore the bad naming for now, if we go in this direction we can fix it). The new abstraction is akin to what we do for theJobsPersistence
, it takes in the database and then all the queries we write against that db can go in there. If we go with this approach we will have 3 abstractions.*
ConfigPersistence
- document-database-style interface over the config database.*
ConfigRepository
- syntactic sugar overConfigPersistence
*
ConfigPersistence2
- all non document-database-style queries.* as we normalize the config database schema, ConfigPersistence2 will get bigger and
ConfigPersistence
will get deemphasized.Option 2: treat sync state like we do other objects in the configs db
we just add the sync state "table" the
airbyte_configs
table and have it look the same as all of the other configs. Then we worry about normalizing it later when we do it for the rest of the configs.If we think Option2 is the right thing for now, that's fine with me too. Option 1 leaves us in a halfsey state and if we are not going to normalize the schemas soon, then that's probably a bad thing.