perf: defer snapshot writes until end of session #606
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
noahnu
commented
May 28, 2022
Comment on lines
74
to
85
| def write_file(cls, snapshot_fossil: "SnapshotFossil", merge: bool = False) -> None: | ||
| """ | ||
| Writes the snapshot data into the snapshot file that be read later. | ||
| Writes the snapshot data into the snapshot file that can be read later. | ||
| """ | ||
| filepath = snapshot_fossil.location | ||
| if merge: | ||
| base_snapshot = cls._read_file(filepath) | ||
| base_snapshot.merge(snapshot_fossil) | ||
| snapshot_fossil = base_snapshot | ||
|
|
||
| with open(filepath, "w", encoding=TEXT_ENCODING, newline=None) as f: | ||
| for snapshot in sorted(snapshot_fossil, key=lambda s: s.name): | ||
| snapshot_data = str(snapshot.data) |
There was a problem hiding this comment.
Maybe we grab a lock for the file by filepath here.
noahnu
commented
May 28, 2022
| @@ -86,13 +91,7 @@ def write_file(cls, snapshot_fossil: "SnapshotFossil") -> None: | |||
| f.write(f"\n{cls._marker_divider}\n") | |||
|
|
|||
| @classmethod | |||
| @functools.lru_cache() | |||
There was a problem hiding this comment.
We need to ensure the data we read is fresh during the batch reads.
noahnu
commented
May 28, 2022
src/syrupy/session.py
Outdated
| @@ -79,6 +104,7 @@ def finish(self) -> int: | |||
| assertions=self._assertions, | |||
| options=self.pytest_session.config.option, | |||
| ) | |||
| self.flush_snapshot_write_queue() | |||
| if self.report.num_unused: | |||
| if self.update_snapshots: | |||
| self.remove_unused_snapshots( | |||
There was a problem hiding this comment.
Unused snapshot removal will also need to be updated.
Codecov Report
@@ Coverage Diff @@
## next #606 +/- ##
==========================================
- Coverage 99.69% 99.62% -0.07%
==========================================
Files 20 20
Lines 1292 1318 +26
==========================================
+ Hits 1288 1313 +25
- Misses 4 5 +1 |
noahnu
changed the title
noahnu
added a commit
that referenced
this pull request
Aug 11, 2022
noahnu
added a commit
that referenced
this pull request
Aug 11, 2022
noahnu
added a commit
that referenced
this pull request
Aug 11, 2022
noahnu
added a commit
that referenced
this pull request
Aug 11, 2022
noahnu
added a commit
that referenced
this pull request
Sep 20, 2022
noahnu
added a commit
that referenced
this pull request
Oct 4, 2022
noahnu
added a commit
that referenced
this pull request
Nov 3, 2022
noahnu
added a commit
that referenced
this pull request
Nov 3, 2022
noahnu
added a commit
that referenced
this pull request
Nov 3, 2022
noahnu
added a commit
that referenced
this pull request
Nov 8, 2022
noahnu
added a commit
that referenced
this pull request
Nov 8, 2022
noahnu
added a commit
that referenced
this pull request
Nov 24, 2022
noahnu
added a commit
that referenced
this pull request
Nov 30, 2022
noahnu
added a commit
that referenced
this pull request
Dec 30, 2022
noahnu
added a commit
that referenced
this pull request
Dec 30, 2022
noahnu
added a commit
that referenced
this pull request
Dec 30, 2022
noahnu
added a commit
that referenced
this pull request
Dec 30, 2022
tophat-opensource-bot
pushed a commit
that referenced
this pull request
Feb 2, 2023
# [4.0.0](v3.0.6...v4.0.0) (2023-02-02) ### Bug Fixes * defer snapshot writes until end of session ([#606](#606)) ([68f1d5f](68f1d5f)) * ensure all pytest options are serializable ([#667](#667)) ([e8ed9f2](e8ed9f2)) * improve pytest-xdist compatibility ([9b9090f](9b9090f)) * lru_cache on snapshot reads ([#629](#629)) ([c1a675f](c1a675f)) * remove legacy path usage to support no:legacypath, closes [#677](#677) ([#684](#684)) ([6385979](6385979)) ### Code Refactoring * simplify data serializer for ambr ([#676](#676)) ([3d296e1](3d296e1)) * write performance improvements, api clarity ([#645](#645)) ([2c31c39](2c31c39)) ### Features * **json:** serialize None as null, close [#622](#622) ([c330680](c330680)) * numerically sort snapshots if possible, close [#657](#657) ([4ca0716](4ca0716)) * **serializer:** preserve key ordering of OrderedDict ([0a2289a](0a2289a)) * support overriding the amber serializer class ([#683](#683)) ([662c93f](662c93f)) * update python version, pytest version ([#658](#658)) ([c360b95](c360b95)) ### BREAKING CHANGES * Serializers may now throw a TaintedSnapshotError which will tell the user to regenerate the snapshot even if the underlying data has not changed. This is to support rolling out more subtle changes to the serializers, such as the introduction of serializer metadata. * Renamed DataSerializer to AmberDataSerializer. * **serializer:** Key order is now preserved if using OrderedDict in both the Amber serializer and JSON serializer. * **json:** The JSONSnapshotExtension now serializes Python's None as "null" rather than "None". * Raise minimum python version to 3.8.1 and min. pytest version to v7. * PyTestLocation.filename has been renamed to .basename * refactor: add test_location kwarg to get_snapshot_name * refactor: get_snapshot_name is now static as a classmethod * refactor: remove pre and post read/write hooks * Pre and post read/write hooks have been removed without replacement to make internal refactor simpler. Please open a GitHub issue if you have a use case for these hooks. * refactor: rename Fossil to Collection * The term 'fossil' has been replaced by the clearer term 'collection'. * refactor: pass test_location to read_snapshot * refactor: remove singular write_snapshot method * refactor: dirname property to method * refactor: pass test_location to discover_snapshots * refactor: remove usage of self.test_location * refactor: make write_snapshot a classmethod * refactor: do not instantiate extension with test_location * Numerous instance methods have been refactored as classmethods.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Defers snapshot writes until the end of the session. Previously we wrote the snapshots while evaluating the assertion in each individual test case.
Related Issues