Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results #9155

Merged
merged 10 commits into from
Dec 6, 2023
Merged

Fix ensuring we produce valid jsonschema artifacts for manifest, catalog, sources, and run-results #9155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f17f982
Drop `all_refs=True` from jsonschema-ization build process
QMalcolm Nov 28, 2023
c9d44a1
Explicity produce jsonschemas using DRAFT_2020_12 dialect
QMalcolm Nov 28, 2023
94361bc
Bump manifest version to v12
QMalcolm Nov 28, 2023
2f8203f
Begin including schema dialect specification in produced jsonschema
QMalcolm Nov 28, 2023
b80e4d4
Create manifest v12 jsonschema specification
QMalcolm Nov 28, 2023
e9acf41
Add change documentation for jsonschema schema production fix
QMalcolm Nov 28, 2023
21ecdb9
Bump run-results version to v6
QMalcolm Nov 30, 2023
747995f
Generate new v6 run-results jsonschema
QMalcolm Nov 30, 2023
b82d069
Regenerate catalog v1 and sources v3 with fixed jsonschema production
QMalcolm Nov 30, 2023
9933638
Update tests to handle bumped versions of manifest and run-results
QMalcolm Dec 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/unreleased/Fixes-20231127-165244.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Fixes
body: Ensure we produce valid jsonschema schemas for manifest, catalog, run-results,
and sources
time: 2023-11-27T16:52:44.590313-08:00
custom:
Author: QMalcolm
Issue: "8991"
3 changes: 2 additions & 1 deletion core/dbt/contracts/graph/manifest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1550,7 +1550,7 @@ def __init__(self, macros) -> None:


@dataclass
@schema_version("manifest", 11)
@schema_version("manifest", 12)
gshank marked this conversation as resolved.
Show resolved Hide resolved
class WritableManifest(ArtifactMixin):
nodes: Mapping[UniqueID, ManifestNode] = field(
metadata=dict(description=("The nodes defined in the dbt project and its dependencies"))
Expand Down Expand Up @@ -1618,6 +1618,7 @@ def compatible_previous_versions(cls) -> Iterable[Tuple[str, int]]:
("manifest", 8),
("manifest", 9),
("manifest", 10),
("manifest", 11),
]

@classmethod
Expand Down
3 changes: 2 additions & 1 deletion core/dbt/contracts/results.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ def write(self, path: str):


@dataclass
@schema_version("run-results", 5)
@schema_version("run-results", 6)
class RunResultsArtifact(ExecutionResult, ArtifactMixin):
results: Sequence[RunResultOutput]
args: Dict[str, Any] = field(default_factory=dict)
Expand All @@ -275,6 +275,7 @@ def from_execution_results(
def compatible_previous_versions(cls) -> Iterable[Tuple[str, int]]:
return [
("run-results", 4),
("run-results", 5),
]

@classmethod
Expand Down
3 changes: 2 additions & 1 deletion core/dbt/contracts/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
ValidationError,
)
from mashumaro.jsonschema import build_json_schema
from mashumaro.jsonschema.dialects import DRAFT_2020_12
import functools


Expand Down Expand Up @@ -197,7 +198,7 @@
@classmethod
@functools.lru_cache
def json_schema(cls) -> Dict[str, Any]:
json_schema_obj = build_json_schema(cls, all_refs=True)
json_schema_obj = build_json_schema(cls, dialect=DRAFT_2020_12, with_dialect_uri=True)

Check warning on line 201 in core/dbt/contracts/util.py

View check run for this annotation

Codecov / codecov/patch

core/dbt/contracts/util.py#L201 

Added line #L201 was not covered by tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that this isn't a default in mashumaro.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is actually! 🙂

I put the details of why I did this in the commit message.

Previously were were implicitly using the DRAFT_2020_12 dialect through
mashumaro. It felt wise to begin explicitly specifying this. First, it
is closest in available mashumaro provided dialects to what we produced
pre 1.7. Secondly, if mashumaro changes its default for whatever reason
(say a new dialect is added, and mashumaro moves to that), we don't want
to automatically inherit that.

json_schema = json_schema_obj.to_dict()
json_schema["$id"] = str(cls.dbt_schema_version)
return json_schema
Expand Down
Loading
Loading