Workaround symengine serialization payload incompatibility (backport #13251) #13255
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
* Workaround symengine serialization payload incompatibility In QPY we rely on symengine's internal serialization to represent the internal symbolic expression stored inside a ParameterExpression object. However, this format is nominally symengine version specific and will raise an error if there is a mismatch between the version used to generate the payload and what is trying to read it. This became an issue in the recent symengine 0.13 release which started to raise an error when people installed it and tried to load QPY payloads across the versions. This makes the symengine serialization unsuitable for use in QPY because it's supposed to be independent of these kind of concerns, especially when QPY is used in a server-client model where you don't necessarily control the installed environment of symengine. To correctly address this issue we'll need a new version of the QPY format that owns the serialization format of ParameterExpressions directly instead of relying on symengine which doesn't offer a compatibility guarantee on the format. However this won't be quick solution and users are encountering issues since the release of 0.13. This commit introduces a workaround for this specific instance of the mismatch. It turns out the payload format between 0.11 and 0.13 is completely unchanged except for the version number. So before passing the parameter expression payload to symengine for deserialization this commit checks the versions numbers are the same, if they're not it checks that we're dealing with 0.11 or 0.13, and if so it changes the version number in the payloads header appropriately. If the version number is outside those bounds it raises an exception because while this hack is known to be safe for translating between symengine 0.11 and 0.13, it's not possible to know for a future version whether the payload format changed or not. Longer term we will need a proper fix in qpy version 13 that introduces a qiskit native serialization format for parameter expression instead of relying on symengine or sympy to do it for us. * Handle schedules too This commit updates the schedule serialization path too, as it was also directly loading symengine expressions. The code handling the workaround is extracted to a standalone function which is used in both spots now instead of calling symengine directly. * Remove unused imports * Gracefully handle failure to parse historical symengine files During the discovery on the fix in this PR we discovered that setting the ``use_symengine`` flag from Qiskit 0.45.x and 0.46.x would result in newer versions of Qiskit being unable to parse the QPY file for the same issue as being addressed here. This commit expands the logic to account for this and raise a useful warning. It also updates the release notes and documentation to document these limitations. * Cap upper version of symengine in requirements list Out of an abundance of caution this commit places a cap on the allowed version of symengine users can install to be compatible with Qiskit. Due to the symengine version dependence discovered in QPY around serializing ParameterExpressions, we'll likely have a similar issue when symengine 0.14.0 releases. Pre-emptively capping this means we aren't going to be in this situation until we can confirm compatibility with QPY serialization. The real solution for this will come in #13252, although as this behavior is embedded in QPY formats 10, 11, and 12 at this point we'll have to handle this edge case moving forward regardless of whether we introduce a better solution in 1.3.0 or not. Although realistically in that case we will likely need to just document this as a limitation when exporting QPY payloads with Qiskit 0.45.0 through 1.2.3 (and with the ``version`` flag set to >= 10 and < 13) and have explicit error checking around the symengine version (which this PR adds) when in that code path. * Fix release note upgrade section label * Rewrite support documentation * Fix mistakes in release note Co-authored-by: Matthew Treinish <mtreinish@kortar.org> --------- Co-authored-by: Jake Lishman <jake.lishman@ibm.com> Co-authored-by: Jake Lishman <jake@binhbar.com> (cherry picked from commit fee9f77)
|
Thank you for opening a new pull request. Before your PR can be merged it will first need to pass continuous integration tests and be reviewed. Sometimes the review process can be slow, so please be patient. While you're waiting, please feel free to review other open PRs. While only a subset of people are authorized to approve pull requests for merging, everyone is encouraged to review open pull requests. Doing reviews helps reduce the burden on the core team and helps make the project's code better for everyone. One or more of the following people are relevant to this code:
|
github-actions
bot
added
priority: high
Changelog: Bugfix
jakelishman
previously approved these changes
Oct 2, 2024
Merged
jakelishman
approved these changes
Oct 2, 2024
Pull Request Test Coverage Report for Build 11145045340Details
💛 - Coveralls |
Eric-Arellano
approved these changes
Oct 2, 2024
abbycross
reviewed
Oct 2, 2024
| a :class:`.ParameterExpression`, if the versions of ``symengine`` installed | ||
| in the generating and loading environments were not the same. For example, | ||
| if a QPY file containing :class:`.ParameterExpression`\ s was generated | ||
| using Qiskit 1.2.2 with ``symengine==0.11.0`` installed, Qiskit 1.2.2 with |
There was a problem hiding this comment.
Suggested change
| using Qiskit 1.2.2 with ``symengine==0.11.0`` installed, Qiskit 1.2.2 with | |
| using Qiskit v1.2.2 with ``symengine==0.11.0`` installed, Qiskit v1.2.2 with |
using v before version number was prescribed by the editorial team - not sure how urgent it is to apply it in API docs
There was a problem hiding this comment.
If this is a chance the editorial team want, we might want to do it all in one go in a follow up - if I do this here, the rest of the Qiskit 1.2 release notes page (so 1.2.2, 1.2.1 and 1.2.0) will still be using the old form, so it might look disjointed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
In QPY we rely on symengine's internal serialization to represent the internal symbolic expression stored inside a ParameterExpression object. However, this format is nominally symengine version specific and will raise an error if there is a mismatch between the version used to generate the payload and what is trying to read it. This became an issue in the recent symengine 0.13 release which started to raise an error when people installed it and tried to load QPY payloads across the versions. This makes the symengine serialization unsuitable for use in QPY because it's supposed to be independent of these kind of concerns, especially when QPY is used in a server-client model where you don't necessarily control the installed environment of symengine.
To correctly address this issue we'll need a new version of the QPY format that owns the serialization format of ParameterExpressions directly instead of relying on symengine which doesn't offer a compatibility guarantee on the format. However this won't be quick solution and users are encountering issues since the release of 0.13. This commit introduces a workaround for this specific instance of the mismatch. It turns out the payload format between 0.11 and 0.13 is completely unchanged except for the version number. So before passing the parameter expression payload to symengine for deserialization this commit checks the versions numbers are the same, if they're not it checks that we're dealing with 0.11 or 0.13, and if so it changes the version number in the payloads header appropriately. If the version number is outside those bounds it raises an exception because while this hack is known to be safe for translating between symengine 0.11 and 0.13, it's not possible to know for a future version whether the payload format changed or not.
Longer term we will need a proper fix in qpy version 13 that introduces a qiskit native serialization format for parameter expression instead of relying on symengine or sympy to do it for us.
Details and comments
This is an automatic backport of pull request #13251 done by [Mergify](https://mergify.com).