feat!: Compress binary ECCs using zlib #498

aborgna-q · 2024-07-19T16:00:44Z

Breaking change for the compiled .rwr ECC sets. The format is now compressed using zstd.

Comparative figures for the nam_6_3.rwr included in the python package:

File size reduced from 69MB to 4.4MB.
Load time on my machine reduced from ~560ms to ~420ms.
(offline) ECC compilation time increased from 2.4s to 3s.
This is using the highest compression rate. Faster encodings produce files of around 5MB.

Closes #488

BREAKING CHANGE: .rwr ECC files generated with older versions are no longer supported. Please recompile them, or compress the file with zstd.

aborgna-q · 2024-07-19T16:06:39Z

I tried both zstd and flate2 (gzip encoding) here. The differences were minimal;
flate2 was slower when using high compression levels and produced slightly bigger files.

It seems the main bottleneck when loading ECCs is the rmp deserialisation, so the choice of compression library does not change much.

codecov · 2024-07-19T16:06:41Z

Codecov Report

Attention: Patch coverage is 55.88235% with 15 lines in your changes missing coverage. Please review.

Project coverage is 82.92%. Comparing base (8c7c338) to head (18d4b89).

Files	Patch %	Lines
tket2/src/rewrite/ecc_rewriter.rs	76.00%	0 Missing and 6 partials ⚠️
badger-optimiser/src/main.rs	0.00%	5 Missing ⚠️
compile-rewriter/src/main.rs	0.00%	4 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #498   +/-   ##
=======================================
  Coverage   82.92%   82.92%           
=======================================
  Files          64       64           
  Lines        6588     6612   +24     
  Branches     6067     6091   +24     
=======================================
+ Hits         5463     5483   +20     
  Misses        848      848           
- Partials      277      281    +4

Flag	Coverage Δ
python	`96.73% <ø> (ø)`
rust	`81.74% <55.88%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

doug-q · 2024-07-19T16:47:28Z

tket2/src/rewrite/ecc_rewriter.rs

    pub fn save_binary_io<W: io::Write>(
        &self,
        writer: &mut W,
    ) -> Result<(), RewriterSerialisationError> {
-        rmp_serde::encode::write(writer, &self)?;
+        let mut encoder = zstd::Encoder::new(writer, 9)?;


no coverage here, can we have a test that saves and loads an ECCRewriter and checks it roundtripped?

aborgna-q · 2024-07-22T11:22:50Z

I added a test for the methods that write and read from a buffer.
The file-reading ones are just wrappers over that, and I'd prefer to avoid fiddling with temp files if possible.

doug-q

Thank you!

🤖 I have created a release *beep* *boop* --- <details><summary>tket2-py: 0.2.0</summary> ## [0.2.0](tket2-py-v0.1.0...tket2-py-v0.2.0) (2024-08-01) ### ⚠ BREAKING CHANGES * increased minimum required version of hugr to 0.9.0 * `.rwr` ECC files generated with older versions are no longer supported. Please recompile them, or compress the file with `zstd`. ### Features * Add timeout options and stats to Badger ([#496](#496)) ([32a9885](32a9885)) * Compress binary ECCs using zlib ([#498](#498)) ([d9a713c](d9a713c)) * Expose advanced Badger timeout options to tket2-py ([#506](#506)) ([fe7d40e](fe7d40e)) * Move the compiled eccs to a separate package ([#517](#517)) ([7247cc6](7247cc6)) ### Bug Fixes * Recompile ecc sets after [#441](#441) ([#484](#484)) ([1122fa4](1122fa4)) ### Miscellaneous Chores * bump hugr version to 0.10.0 ([#508](#508)) ([eca258b](eca258b)) </details> <details><summary>tket2-eccs: 0.1.0</summary> ## 0.1.0 (2024-08-01) ### Features * Move the compiled eccs to a separate package ([#517](#517)) ([7247cc6](7247cc6)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: Agustín Borgna <121866228+aborgna-q@users.noreply.github.com>

## 🤖 New release * `tket2`: 0.1.0-alpha.2 -> 0.1.0 * `tket2-hseries`: 0.1.0 <details><summary>Changelog</summary> ## `tket2` <blockquote> ## [0.1.0](tket2-v0.1.0-alpha.2...tket2-v0.1.0) - 2024-08-01 ### Bug Fixes - Single source of truth for circuit names, and better circuit errors ([#390](#390)) - Support non-DFG circuits ([#391](#391)) - Portmatching not matching const edges ([#444](#444)) - Pattern matcher discriminating on opaqueOp description ([#441](#441)) - `extract_dfg` inserting the output node with an invalid child order ([#442](#442)) - Recompile ecc sets after [#441](#441) ([#484](#484)) ### Documentation - Update tket2-py readme ([#431](#431)) - Better error reporting in portmatching ([#437](#437)) - Improved multi-threading docs for Badger ([#495](#495)) ### New Features - `Circuit::operations` ([#395](#395)) - tuple unpack rewrite ([#406](#406)) - guppy → pytket conversion ([#407](#407)) - Drop linear bits, improve pytket encoding/decoding ([#420](#420)) - *(py)* Allow using `Tk2Op`s in the builder ([#436](#436)) - Initial support for `TailLoop` as circuit parent ([#417](#417)) - Support tuple unpacking with multiple unpacks ([#470](#470)) - Partial tuple unpack ([#475](#475)) - [**breaking**] Compress binary ECCs using zlib ([#498](#498)) - Add timeout options and stats to Badger ([#496](#496)) - Expose advanced Badger timeout options to tket2-py ([#506](#506)) ### Refactor - [**breaking**] Simplify tket1 conversion errors ([#408](#408)) - Cleanup tket1 serialized op structures ([#419](#419)) ### Testing - Add coverage for Badger split circuit multi-threading ([#505](#505)) </blockquote> ## `tket2-hseries` <blockquote> ## [0.1.0](https://github.com/CQCL/tket2/releases/tag/tket2-hseries-v0.1.0) - 2024-08-01 ### New Features - [**breaking**] init tket2-hseries ([#368](#368)) - *(tket2-hseries)* Add `tket2.futures` Hugr extension ([#471](#471)) - Add lazify-measure pass ([#482](#482)) - add results extensions ([#494](#494)) - *(tket2-hseries)* [**breaking**] Add `HSeriesPass` ([#487](#487)) </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/MarcoIeni/release-plz/). --------- Co-authored-by: Douglas Wilson <douglas.wilson@quantinuum.com>

## 🤖 New release * `tket2`: 0.1.0 -> 0.1.1 * `tket2-hseries`: 0.1.0 -> 0.1.1 <details><summary>Changelog</summary> ## `tket2` <blockquote> ## [0.1.0](tket2-v0.1.0-alpha.2...tket2-v0.1.0) - 2024-08-01 ### Bug Fixes - Single source of truth for circuit names, and better circuit errors ([#390](#390)) - Support non-DFG circuits ([#391](#391)) - Portmatching not matching const edges ([#444](#444)) - Pattern matcher discriminating on opaqueOp description ([#441](#441)) - `extract_dfg` inserting the output node with an invalid child order ([#442](#442)) - Recompile ecc sets after [#441](#441) ([#484](#484)) ### Documentation - Update tket2-py readme ([#431](#431)) - Better error reporting in portmatching ([#437](#437)) - Improved multi-threading docs for Badger ([#495](#495)) ### New Features - `Circuit::operations` ([#395](#395)) - tuple unpack rewrite ([#406](#406)) - guppy → pytket conversion ([#407](#407)) - Drop linear bits, improve pytket encoding/decoding ([#420](#420)) - *(py)* Allow using `Tk2Op`s in the builder ([#436](#436)) - Initial support for `TailLoop` as circuit parent ([#417](#417)) - Support tuple unpacking with multiple unpacks ([#470](#470)) - Partial tuple unpack ([#475](#475)) - [**breaking**] Compress binary ECCs using zlib ([#498](#498)) - Add timeout options and stats to Badger ([#496](#496)) - Expose advanced Badger timeout options to tket2-py ([#506](#506)) ### Refactor - [**breaking**] Simplify tket1 conversion errors ([#408](#408)) - Cleanup tket1 serialized op structures ([#419](#419)) ### Testing - Add coverage for Badger split circuit multi-threading ([#505](#505)) </blockquote> ## `tket2-hseries` <blockquote> ## [0.1.1](tket2-hseries-v0.1.0...tket2-hseries-v0.1.1) - 2024-08-15 ### New Features - *(tket2-hseries)* make result operation internals public ([#542](#542)) </blockquote> </details> --- This PR was generated with [release-plz](https://github.com/MarcoIeni/release-plz/). --------- Co-authored-by: Seyon Sivarajah <seyon.sivarajah@cambridgequantum.com>

aborgna-q requested a review from a team as a code owner July 19, 2024 16:00

aborgna-q requested a review from doug-q July 19, 2024 16:00

doug-q reviewed Jul 19, 2024

View reviewed changes

aborgna-q added 4 commits July 22, 2024 11:58

Print ECCs load time in badger runner and ecc compiler

903278f

feat!: Compress binary ECCs using zlib

70eacd1

Test the ecc io encoding roundtrip

e89aa9c

drive-by: Fix typos

18d4b89

aborgna-q force-pushed the ab/compressed-eccs branch from 1534a8e to 18d4b89 Compare July 22, 2024 11:18

aborgna-q requested a review from doug-q July 22, 2024 11:22

doug-q approved these changes Jul 22, 2024

View reviewed changes

aborgna-q added this pull request to the merge queue Jul 22, 2024

Merged via the queue into main with commit d9a713c Jul 22, 2024
15 of 16 checks passed

aborgna-q deleted the ab/compressed-eccs branch July 22, 2024 12:36

This was referenced Jul 22, 2024

chore(main): release tket2-py 0.2.0 #502

Closed

chore: release #409

Closed

This was referenced Aug 1, 2024

chore: release main #518

Merged

chore: release #519

Merged

hugrbot mentioned this pull request Aug 1, 2024

chore: release #521

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Compress binary ECCs using zlib #498

feat!: Compress binary ECCs using zlib #498

aborgna-q commented Jul 19, 2024 •

edited

Loading

aborgna-q commented Jul 19, 2024

codecov bot commented Jul 19, 2024 •

edited

Loading

doug-q Jul 19, 2024

aborgna-q commented Jul 22, 2024 •

edited

Loading

doug-q left a comment

feat!: Compress binary ECCs using zlib #498

feat!: Compress binary ECCs using zlib #498

Conversation

aborgna-q commented Jul 19, 2024 • edited Loading

aborgna-q commented Jul 19, 2024

codecov bot commented Jul 19, 2024 • edited Loading

Codecov Report

doug-q Jul 19, 2024

Choose a reason for hiding this comment

aborgna-q commented Jul 22, 2024 • edited Loading

doug-q left a comment

Choose a reason for hiding this comment

aborgna-q commented Jul 19, 2024 •

edited

Loading

codecov bot commented Jul 19, 2024 •

edited

Loading

aborgna-q commented Jul 22, 2024 •

edited

Loading