Encapsulate Python sequence-like indexers #12669

jakelishman · 2024-06-26T13:53:54Z

Summary

This encapsulates a lot of the common logic around Python sequence-like indexers (SliceOrInt) into iterators that handle adapting negative indices and slices in usize for containers of a given size.

These indexers now all implement ExactSizeIterator and DoubleEndedIterator, so they can be used with all Iterator methods, and can be used (with Iterator::map and friends) as inputs to PyList::new_bound, which makes code simpler at all points of use.

The special-cased uses of this kind of thing from CircuitData are replaced with the new forms. This had no measurable impact on performance on my machine, and removes a lot noise from error-handling and highly specialised functions.

Details and comments

This PR was not meant to be this big - I only started writing something up because I needed similar logic to things we already had in 3+ different places for SparseObservable, and then I let it get a bit more out of hand.

This introduces thiserror as a crate for generating the boilerplate for implementing Error for return values. I barely used it in this PR, but I split this off from a follow-up defining SparseObservable, which uses it more heavily.

qiskit-bot · 2024-06-26T13:53:59Z

One or more of the following people are relevant to this code:

@Qiskit/terra-core
@kevinhartman
@levbishop
@mtreinish

coveralls · 2024-06-26T15:11:17Z

Pull Request Test Coverage Report for Build 9681825290

Details

194 of 240 (80.83%) changed or added relevant lines in 4 files are covered.
16 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.02%) to 89.768%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/accelerate/src/euler_one_qubit_decomposer.rs	3	9	33.33%
crates/accelerate/src/two_qubit_decompose.rs	3	9	33.33%
crates/circuit/src/circuit_data.rs	82	93	88.17%
crates/circuit/src/slice.rs	106	129	82.17%

Files with Coverage Reduction	New Missed Lines	%
crates/qasm2/src/lex.rs	4	91.86%
crates/qasm2/src/parse.rs	12	96.23%

Totals
Change from base Build 9680727079:	0.02%
Covered Lines:	63806
Relevant Lines:	71079

💛 - Coveralls

coveralls · 2024-06-26T21:38:07Z

Pull Request Test Coverage Report for Build 9686831288

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

192 of 236 (81.36%) changed or added relevant lines in 4 files are covered.
93 unchanged lines in 4 files lost coverage.
Overall coverage increased (+0.06%) to 89.806%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/accelerate/src/euler_one_qubit_decomposer.rs	3	9	33.33%
crates/accelerate/src/two_qubit_decompose.rs	3	9	33.33%
crates/circuit/src/circuit_data.rs	82	93	88.17%
crates/circuit/src/slice.rs	104	125	83.2%

Files with Coverage Reduction	New Missed Lines	%
crates/qasm2/src/expr.rs	1	94.02%
crates/qasm2/src/lex.rs	4	92.88%
qiskit/circuit/quantumcircuit.py	16	95.69%
crates/circuit/src/operations.rs	72	72.35%

Totals
Change from base Build 9680727079:	0.06%
Covered Lines:	63843
Relevant Lines:	71090

💛 - Coveralls

Cryoris

Nice to see so much code repetition to go away! I mainly have a question about negative Python slices (like vec[-5:-1]) 🙂

Cryoris · 2024-06-27T06:35:15Z

crates/circuit/src/slice.rs

+                    let index = *index as usize;
+                    if index >= len {


Could this also be written in a single line as follows? 🙂

Suggested change

let index = *index as usize;

if index >= len {

if *index as usize >= len {

I don't remember exactly why I split it off like this, but if you want to change it, line 54 would also need changing to the same thing. If you want to remake the suggestion with both I'll commit it.

crates/circuit/src/slice.rs

Cryoris · 2024-06-27T06:54:34Z

crates/circuit/src/slice.rs

+                gap / *step + (gap % *step != 0) as usize
+            }
+            Self::NegRange { start, stop, step } => 'arm: {
+                let Some(start) = start else { break 'arm 0 };


Now I'm relieved we don't share an office with you breaking limbs 😉 but on a more serious note, I'm a bit confused about this, couldn't I have a Python slice with negative start and stop and then this would end up with two None values?

Haha yeah, I meant the "match arm" but I didn't even think about that!

Yes, you can have two None, and in fact I think that's the only time indices.start can be None. Either if it weren't, the length would still always be zero because the start point isn't a valid index.

This encapsulates a lot of the common logic around Python sequence-like indexers (`SliceOrInt`) into iterators that handle adapting negative indices and slices in `usize` for containers of a given size. These indexers now all implement `ExactSizeIterator` and `DoubleEndedIterator`, so they can be used with all `Iterator` methods, and can be used (with `Iterator::map` and friends) as inputs to `PyList::new_bound`, which makes code simpler at all points of use. The special-cased uses of this kind of thing from `CircuitData` are replaced with the new forms. This had no measurable impact on performance on my machine, and removes a lot noise from error-handling and highly specialised functions.

coveralls · 2024-06-28T16:55:58Z

Pull Request Test Coverage Report for Build 9715953948

Details

192 of 236 (81.36%) changed or added relevant lines in 4 files are covered.
15 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.04%) to 89.783%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/accelerate/src/euler_one_qubit_decomposer.rs	3	9	33.33%
crates/accelerate/src/two_qubit_decompose.rs	3	9	33.33%
crates/circuit/src/circuit_data.rs	82	93	88.17%
crates/circuit/src/slice.rs	104	125	83.2%

Files with Coverage Reduction	New Missed Lines	%
crates/qasm2/src/lex.rs	3	92.62%
crates/qasm2/src/parse.rs	12	97.15%

Totals
Change from base Build 9714383578:	0.04%
Covered Lines:	63840
Relevant Lines:	71105

💛 - Coveralls

mtreinish

This lgtm, this is a great abstraction to add and makes working with slices much less error prone. All the logic seems super through and accounted for edge cases I didn't even realize were there. As a standalone PR I think this is good to merge.

But I'm wondering about rustworkx and probably any other pyo3 users out there. I'm thinking we should contribute this to PyO3 or have it somewhere more publicly accessible because I think it'll be useful for any custom sequence types people are creating with PyO3. It's providing a nice API to work with slices which are super error prone with the current pyo3 interface because they're super low level and the Python C API doesn't document things super well.

mtreinish · 2024-07-01T12:24:25Z

crates/circuit/src/circuit_data.rs

+        let py = value.py();
+        match index.with_len(self.data.len())? {
+            SequenceIndex::Int(index) => set_single(self, index, value),
+            indices @ SequenceIndex::PosRange {


I think this is the first time I've seen the match binding syntax used in practice.

I think I used it in the OQ2 lexer as well - I don't use it often, but it's occasionally useful.

mtreinish · 2024-07-01T12:27:39Z

crates/circuit/src/circuit_data.rs

+    fn pack(&mut self, inst: PyRef<CircuitInstruction>) -> PyResult<PackedInstruction> {
+        let py = inst.py();


This feels a bit unnecessary but it's not a big deal I guess.

Yeah, I don't think I intended to make a change here - it might be a remnant of something else I was up to.

mtreinish · 2024-07-01T12:32:28Z

crates/circuit/src/slice.rs

+        // `slice` can't be subclassed in Python, so it's safe (and faster) to check for it exactly.
+        // The `downcast_exact` check is just a pointer comparison, so while `slice` is the less
+        // common input, doing that first has little-to-no impact on the speed of the `isize` path,
+        // while the reverse makes `slice` inputs significantly slower.
+        if let Ok(slice) = ob.downcast_exact::<PySlice>() {
+            return Ok(Self::Slice(slice.clone()));
+        }
+        Ok(Self::Int(ob.extract()?))


Heh, this makes me think of: Qiskit/rustworkx#1096 where you opted for just flipping the order. This is starting to make me think maybe we want to contribute this to PyO3 so there is a common impl of this for everyone.

You're just watching me learning more of the low-level internals of everything live haha.

In Rustworkx I didn't know enough other than to leave the FromPyObject derivation in place, in which case flipping the order just makes things faster. But for the sake of completeness here, since I'd already wasted enough time that a little longer made no difference, I actually checked whether slice is subclassable (turns out it isn't), in which case the derived FromPyObject is needlessly slow; you can't subclass it, so the type is slice check is sufficient and is just a pointer comparison.

jakelishman · 2024-07-01T13:08:23Z

Yeah, I considered talking to PyO3 about contributing it upstream, though I'm thinking maybe we play with it a shade longer within Qiskit first to see if the abstraction actually bears weight.

I'm ok with the interface I've come up with here, but I think it could still be better. If nothing else, there's tons of specialisations of Range that could be passed through the various iterator methods. I've mostly just added the bits we need for Qiskit immediately (though tbh DoubleEndedIterator for Descending was probably not 100% required) to avoid further bloating a PR that was already more involved than I was expecting.

This encapsulates a lot of the common logic around Python sequence-like indexers (`SliceOrInt`) into iterators that handle adapting negative indices and slices in `usize` for containers of a given size. These indexers now all implement `ExactSizeIterator` and `DoubleEndedIterator`, so they can be used with all `Iterator` methods, and can be used (with `Iterator::map` and friends) as inputs to `PyList::new_bound`, which makes code simpler at all points of use. The special-cased uses of this kind of thing from `CircuitData` are replaced with the new forms. This had no measurable impact on performance on my machine, and removes a lot noise from error-handling and highly specialised functions.

jakelishman added type: qa Issues and PRs that relate to testing and code quality Changelog: None Do not include in changelog Rust This PR or issue is related to Rust code in the repository labels Jun 26, 2024

jakelishman requested a review from a team as a code owner June 26, 2024 13:53

jakelishman mentioned this pull request Jun 26, 2024

WIP: Add base representation of SparseObservable #12671

Draft

Cryoris reviewed Jun 27, 2024

View reviewed changes

jakelishman force-pushed the sequence-index branch from 6083896 to dc20c20 Compare June 28, 2024 16:29

mtreinish approved these changes Jul 1, 2024

View reviewed changes

mtreinish added this pull request to the merge queue Jul 1, 2024

Merged via the queue into Qiskit:main with commit 373e8a6 Jul 1, 2024
15 checks passed

jakelishman deleted the sequence-index branch July 1, 2024 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encapsulate Python sequence-like indexers #12669

Encapsulate Python sequence-like indexers #12669

jakelishman commented Jun 26, 2024

qiskit-bot commented Jun 26, 2024

coveralls commented Jun 26, 2024 •

edited

Loading

coveralls commented Jun 26, 2024 •

edited

Loading

Cryoris left a comment

Cryoris Jun 27, 2024

jakelishman Jun 27, 2024

Cryoris Jun 27, 2024

jakelishman Jun 27, 2024

coveralls commented Jun 28, 2024 •

edited

Loading

mtreinish left a comment

mtreinish Jul 1, 2024

jakelishman Jul 1, 2024

mtreinish Jul 1, 2024

jakelishman Jul 1, 2024

mtreinish Jul 1, 2024

jakelishman Jul 1, 2024

jakelishman commented Jul 1, 2024

	let index = *index as usize;
	if index >= len {
	if *index as usize >= len {

		fn pack(&mut self, inst: PyRef<CircuitInstruction>) -> PyResult<PackedInstruction> {
		let py = inst.py();

Encapsulate Python sequence-like indexers #12669

Encapsulate Python sequence-like indexers #12669

Conversation

jakelishman commented Jun 26, 2024

Summary

Details and comments

qiskit-bot commented Jun 26, 2024

coveralls commented Jun 26, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9681825290

Details

💛 - Coveralls

coveralls commented Jun 26, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9686831288

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Cryoris left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jun 28, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9715953948

Details

💛 - Coveralls

mtreinish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakelishman commented Jul 1, 2024

coveralls commented Jun 26, 2024 •

edited

Loading

coveralls commented Jun 26, 2024 •

edited

Loading

coveralls commented Jun 28, 2024 •

edited

Loading