Invalid NUMBA_NUM_THREADS while running with Numpy

[alecandido]

While debugging a very simple tests for qibotn (on the qibotn:pypkg branch) I encountered the error reported below, coming from Numba backend, even though the Numpy one was selected.

As @stavros11 already noticed correctly, the cupy behavior is perfectly expected and adequate, so the actual issues are:

• why is Numba complaining about the number of threads?
• why is Numba involved at all, if Numpy was selected?

Most likely the problem is machine dependent, since my machine has 6 cores (x2 with hyperthreading), so the total number (i.e. 12) exceeds the maximum of 6 requested by Numba.
However, I don't get why Numba is imposing a maximum at all, nor why it is happening with qibojit, but never happened with other Numba projects (for which the answer might be actually more complicated, and I don't expect to be found here...)

Pytest run with traceback

❯ pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.10.7, pytest-7.2.1, pluggy-1.0.0
rootdir: /media/alessandro/moneybin/Projects/Qibo/qibotn, configfile: pyproject.toml, testpaths: tests/
plugins: env-0.8.1, cov-4.0.0
collected 4 items

tests/test_qasm_quimb_backend.py FFFF                                                                                                                   [100%]

========================================================================== FAILURES ===========================================================================
________________________________________________________________________ test_eval[1] _________________________________________________________________________

[...]

________________________________________________________________________ test_eval[10] ________________________________________________________________________

backend = 'qibojit', platform = 'numpy', runcard = None

    def construct_backend(backend, platform=None, runcard=None):
        if backend == "qibojit":
            from qibojit.backends import CupyBackend, CuQuantumBackend, NumbaBackend

            if platform == "cupy":  # pragma: no cover
                return CupyBackend()
            elif platform == "cuquantum":  # pragma: no cover
                return CuQuantumBackend()
            elif platform == "numba":
                return NumbaBackend()
            else:  # pragma: no cover
                try:
>                   return CupyBackend()

env/lib/python3.10/site-packages/qibo/backends/__init__.py:22:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = numpy

    def __init__(self):
        NumpyBackend.__init__(self)

>       import cupy as cp  # pylint: disable=import-error
E       ModuleNotFoundError: No module named 'cupy'

env/lib/python3.10/site-packages/qibojit/backends/gpu.py:18: ModuleNotFoundError

During handling of the above exception, another exception occurred:

nqubits = 10

    @pytest.mark.parametrize("nqubits", [1, 2, 5, 10])
    def test_eval(nqubits: int):
        print(f"Testing for {nqubits} nqubits")
>       result = qasm_quimb.eval_QI_qft(nqubits)

tests/test_qasm_quimb_backend.py:9:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
env/lib/python3.10/site-packages/qibotn/qasm_quimb.py:187: in eval_QI_qft
    qibo.set_backend(backend=qibo_backend, platform="numpy")
env/lib/python3.10/site-packages/qibo/backends/__init__.py:108: in set_backend
    GlobalBackend.set_backend(backend, platform, runcard)
env/lib/python3.10/site-packages/qibo/backends/__init__.py:84: in set_backend
    cls._instance = construct_backend(backend, platform, runcard)
env/lib/python3.10/site-packages/qibo/backends/__init__.py:24: in construct_backend
    return NumbaBackend()
env/lib/python3.10/site-packages/qibojit/backends/cpu.py:68: in __init__
    self.set_threads(len(psutil.Process().cpu_affinity()))
env/lib/python3.10/site-packages/qibojit/backends/cpu.py:79: in set_threads
    numba.set_num_threads(nthreads)
env/lib/python3.10/site-packages/numba/np/ufunc/parallel.py:607: in set_num_threads
    snt_check(n)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

n = 12

    def snt_check(n):
        if n > NUMBA_NUM_THREADS or n < 1:
>           raise ValueError(msg)
E           ValueError: The number of threads must be between 1 and 6

env/lib/python3.10/site-packages/numba/np/ufunc/parallel.py:569: ValueError
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
Testing for 10 nqubits
====================================================================== warnings summary =======================================================================
env/lib/python3.10/site-packages/quimb/linalg/approx_spectral.py:11
  /media/alessandro/moneybin/Projects/Qibo/qibotn/env/lib/python3.10/site-packages/quimb/linalg/approx_spectral.py:11: DeprecationWarning: Please use `uniform_filter1d` from the `scipy.ndimage` namespace, the `scipy.ndimage.filters` namespace is deprecated.
    from scipy.ndimage.filters import uniform_filter1d

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.10.7-final-0 -----------
Coverage XML written to file coverage.xml

=================================================================== short test summary info ===================================================================
FAILED tests/test_qasm_quimb_backend.py::test_eval[1] - ValueError: The number of threads must be between 1 and 6
FAILED tests/test_qasm_quimb_backend.py::test_eval[2] - ValueError: The number of threads must be between 1 and 6
FAILED tests/test_qasm_quimb_backend.py::test_eval[5] - ValueError: The number of threads must be between 1 and 6
FAILED tests/test_qasm_quimb_backend.py::test_eval[10] - ValueError: The number of threads must be between 1 and 6
================================================================ 4 failed, 1 warning in 2.64s =================================================================

CPU details

❯ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    CPU family:          6
    Model:               158
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            10
    CPU(s) scaling MHz:  82%
    CPU max MHz:         4600,0000
    CPU min MHz:         800,0000
    BogoMIPS:            6399.96
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdp
                         e1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monito
                         r ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rd
                         rand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fs
                         gsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida a
                         rat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
Virtualization features:
  Virtualization:        VT-x
Caches (sum of all):
  L1d:                   192 KiB (6 instances)
  L1i:                   192 KiB (6 instances)
  L2:                    1,5 MiB (6 instances)
  L3:                    12 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-11
Vulnerabilities:
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                   Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:              Mitigation; PTI
  Mmio stale data:       Mitigation; Clear CPU buffers; SMT vulnerable
  Retbleed:              Mitigation; IBRS
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Mitigation; Microcode
  Tsx async abort:       Mitigation; TSX disabled

[alecandido]

https://github.com/numba/numba/blob/28573bb985cf4816e1c6f54ce2f4bd1e44617bcd/numba/np/ufunc/parallel.py#L563-L573

def gen_snt_check():
    from numba.core.config import NUMBA_NUM_THREADS
    msg = "The number of threads must be between 1 and %s" % NUMBA_NUM_THREADS

    def snt_check(n):
        if n > NUMBA_NUM_THREADS or n < 1:
            raise ValueError(msg)
    return snt_check

snt_check = gen_snt_check()

So someone is setting NUMBA_NUM_THREADS. I checked it is not available in my environment as an env var. Nor it is specified anywhere in qibotn.

[alecandido]

And also, when inside IPython in the same environment, I get:

[ins] In [1]: import numba

[ins] In [2]: numba.config
Out[2]: <module 'numba.core.config' from '/media/alessandro/moneybin/Projects/Qibo/qibotn/env/lib/python3.10/site-packages/numba/core/config.py'>

[ins] In [3]: numba.config.NUMBA_NUM_THREADS
Out[3]: 12

while clearly Numba is resolving it to 6 during pytest run...

[scarrazza]

Did you check quimb import call?

[alecandido]

The quimb import in qibotn is trivial:

import quimb as qu
import quimb.tensor as qtn

I'm going to check immediately inside quimb package itself.

[alecandido]

You were right, culprit found:
https://github.com/jcmgray/quimb/blob/8ecf9d6ef7d2055ebbae5f7dbb5aeb0e65c0d422/quimb/core.py#L37-L51

    import psutil

    _NUM_THREAD_WORKERS = psutil.cpu_count(logical=False)

if "NUMBA_NUM_THREADS" in os.environ:
    if int(os.environ["NUMBA_NUM_THREADS"]) != _NUM_THREAD_WORKERS:
        import warnings

        warnings.warn(
            "'NUMBA_NUM_THREADS' has been set elsewhere and doesn't match the "
            "value 'quimb' has tried to set - "
            f"{os.environ['NUMBA_NUM_THREADS']} vs {_NUM_THREAD_WORKERS}."
        )
else:
    os.environ["NUMBA_NUM_THREADS"] = str(_NUM_THREAD_WORKERS)

[alecandido]

Indeed, this is working:

❯ QUIMB_NUM_PROCS=12 pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.10.7, pytest-7.2.1, pluggy-1.0.0
rootdir: /media/alessandro/moneybin/Projects/Qibo/qibotn, configfile: pyproject.toml, testpaths: tests/
plugins: env-0.8.1, cov-4.0.0
collected 4 items

tests/test_qasm_quimb_backend.py ....                                                                                                                   [100%]

The explanation is a few lines above those cited in the previous comment.

[alecandido]

@scarrazza then I guess I shoul move this issue to qibotn, and set the correct variable there.

If you agree, I will "transfer" the issue there (i.e. open a new one, since being a private repo I can't actually transfer).

[scarrazza]

Sure, please go ahead.

[alecandido]

This is at least outdated.

The issue content will still be available, if needed, so there is no need to keep it opened.

[renatomello]

This is causing an issue with tests in qibo because of the added qibotn test that set numba threads globally. The following tests that need multithreading are failing locally. The issue is actually explained here.