-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove psutil as a dependency #11254
Comments
Currently Aer is using Qiskit's local_hardware_info() function which to determine the total amount of system memory which is used to compute the largest statevector the system can build. However, this function wasn't really intended to be used outside of Qiskit and also Qiskit is looking to remove the memory reporting (see: Qiskit/qiskit#11254). This commit just pivots to using psutil directly which is what qiskit is doing internally.
Currently Aer is using Qiskit's local_hardware_info() function which to determine the total amount of system memory which is used to compute the largest statevector the system can build. However, this function wasn't really intended to be used outside of Qiskit and also Qiskit is looking to remove the memory reporting (see: Qiskit/qiskit#11254). This commit just pivots to using psutil directly which is what qiskit is doing internally.
Currently Aer is using Qiskit's local_hardware_info() function which to determine the total amount of system memory which is used to compute the largest statevector the system can build. However, this function wasn't really intended to be used outside of Qiskit and also Qiskit is looking to remove the memory reporting (see: Qiskit/qiskit#11254). This commit just pivots to using psutil directly which is what qiskit is doing internally.
* add skip Python 3.12 for GPU build (#1965) * Fix basis gates of Aer backends (#1976) * move reset and switch_case ops to custom istr * Fix reset in AerStatevector * add test case * format * fix installing built Aer in some test cases * Applying global phase multiplication to initialize operation (#1980) * Applying global phase to initialize operation * fix format * remove recursive, add omp * Release 0.13.1 * Revert too many deprecations in Estimator (#1990) * Revert too much deprecation * fix typo * fix tests * Change priority of method selection of noise simulation (#1989) * Avoid selecting stabilizer method when noise model contains rotational gates * remove checking noise opsets, change priority selecting density_matrix * format * modify test cases use auto method result may change by this PR * modify one more test case * Remove use of opflow in Estimator (#1996) * Update misspelling apply_gate method doc (#1998) Co-authored-by: Jun Doi <doichan@jp.ibm.com> * Add optimization_level=0 to transpiler for compiling dynamic circuits (#2000) ``id`` gate was removed by transpiler called from aer_compiler without optimization_level for dynamic circuits. This commits adds ``optimization_level=0`` to avoid removing id gates * fix ry gate for stabilizer (#2001) Co-authored-by: Hiroshi Horii <hhorii@users.noreply.github.com> * Directly use psutil to get total system memory (#2002) Currently Aer is using Qiskit's local_hardware_info() function which to determine the total amount of system memory which is used to compute the largest statevector the system can build. However, this function wasn't really intended to be used outside of Qiskit and also Qiskit is looking to remove the memory reporting (see: Qiskit/qiskit#11254). This commit just pivots to using psutil directly which is what qiskit is doing internally. * test build fix (#2004) * Reverse ordering to read out error in sampling measure (#2003) * reverse ordering of read out error in sampling measure * fix batch check --------- Co-authored-by: Hiroshi Horii <hhorii@users.noreply.github.com> * Fix extended stabilizer thread safety in apply_ops_parallel (#1993) * Extended stabilizer simulator no longer shares RngEngine amongst states when ops are applied in parallel * Added release note * Fixed ugly cast --------- Co-authored-by: Jun Doi <doichan@jp.ibm.com> * add note (#1992) Co-authored-by: Jun Doi <doichan@jp.ibm.com> * Fix AerBackend issues caused by upgrading BackendV2 (#1995) * add description if no description is provided, build coupling map if it is provided * move import line * fix target for simulator backend * format * remove unused import * use translation plugin to rebuild gate sets for simulator * rename plugin * rebuild of gate sets is eanbled only for opt level 0 and 1 * fix custom pass manager * fix pass_manager function * added ccx in NAME_MAPPING * added missed gates in NAME_MAPPING * added release note * add check if opnodes is None * add check config * decrease return * check opt level * fix searching ops in control flow blocks * Update qiskit_aer/backends/plugin/aer_backend_plugin.py Co-authored-by: Matthew Treinish <mtreinish@kortar.org> * Update qiskit_aer/backends/plugin/aer_backend_plugin.py Co-authored-by: Matthew Treinish <mtreinish@kortar.org> * refer review comments * remove unused import --------- Co-authored-by: Matthew Treinish <mtreinish@kortar.org> * add prelude --------- Co-authored-by: Ikko Hamamura <ikkoham@users.noreply.github.com> Co-authored-by: Elena Peña Tapia <57907331+ElePT@users.noreply.github.com> Co-authored-by: jon <70080228+notcruz@users.noreply.github.com> Co-authored-by: Hiroshi Horii <hhorii@users.noreply.github.com> Co-authored-by: Matthew Treinish <mtreinish@kortar.org> Co-authored-by: eliotheinrich <38039898+eliotheinrich@users.noreply.github.com>
I can take this issue! |
Just to make sure I understand what we're wanting to do here, in multiprocessing.py we're wanting to set I'm a bit confused on how we're looking to move away from psutil to calculate memory usage. |
@rmhowe425 There is actually already an open PR implementing this issue here: #11336 if you'd like to review it and leave any feedback you might have on it that'd also be very helpful. To answer your question, that is basically the suggestion, although for memory I was proposing we just drop that field. The only thing that was using it in practice was the |
Ahhh I did not see the existing PR! Thanks for the heads up and for the info! |
What should we add?
Right now we have
psutil
as a dependency which is used to get the number of physical CPUs and also the available amount of memory at the local system level. The physical CPU count is used to determine the max default number of parallel processes inparallel_map
and the memory information is used to determine statevector size limits forBasicAer
and also is printed in the jupyter version table magic. However, relying on an external lib for just these small uses seems a bit excessive primarily because psutil doesn't have precompiled wheels available for all our supported platforms which requires users on those platforms to have the appropriate compiler and library headers to build it from source to use qiskit. But additionally the checking we're doing via psutil in these cases isn't actually sound. These checks only look at what's reported from the OS for the entire system, but don't actually reflect any process limits that might be in place (this is documented in the psutil docs) so we shouldn't be making task scheduling or job limit decisions based on this information.As discussed in #10868 we're using the physical core count in
parallel_map()
to try and maximize performance because in the case of systems with SMT, scheduling jobs on each logical core can have lower performance. I think a reasonable approach is to defaultCPU_COUNT
to beint(len(os.sched_getaffinity(0)) / 2)
, as most (but obviously not all) modern systems have 2 way SMT. This obviously isn't as robust a detection solution to avoid using logical cores, but I think as a default it's reasonable especially as for any users that want different parallel dispatch behavior there is theQISKIT_NUM_PROCS
env variable or thenum_processes
setting in the qiskit config file (https://qiskit.org/documentation/configuration.html).The memory usage we can just remove the limits from basic aer and set the qubit count to 24 as that's ~270MB and realistically is always the limit for users. If there isn't sufficient memory available for a 24 qubit statevector for some reason, then numpy will error on allocation anyway. The only use case not covered is the jupyter magic, but I think we can just drop the memory reported from that as it's not super critical functionality.
The text was updated successfully, but these errors were encountered: