Docplex is not fully compatible with cplex 20.1

[t-imamichi]

Information

• Qiskit Optimization version: NA
• Python version: 3.7.9, 3.8.7, 3.9.1
• Operating system: macOS 11.2

What is the current behavior?

There are two issues related to docplex as follows.

(1)
Docplex 2.14.186 (May 2020) or older does not work with cplex 20.1 (Jan 2021)

(2)
Docplex 2.16.196 (Nov 2020) or newer does not work without installing cplex. But, cplex is not included as the dependency of docplex. It seems a bug. Because no cplex wheel available for Python 3.9, docplex 2.16.196 or newer does not work with Python 3.9. cplex version also matters.

This change is mentioned as follows.

Changed in 2.16.196 (2020.11):
In docplex.mp:
For solving, docplex.mp now uses the cplex module if it has been installed. If not, docplex.mp checks for the latest installed version of CPLEX Optimization Studio (COS) (using the CPLEX_STUDIO_DIRXXX environment variables) and use the cplex if a COS is found, unless the configuration of the engine states otherwise.
https://pypi.org/project/docplex/#description

Summary of compatibility:

docplex →	2.14.186 or older	2.15.194	2.16.196 - 2.20.204	2.21.207
w/ cplex 12.10 (Dec 2019 - Jul 2020)	OK	OK	OK	OK
w/ cplex 20.1 (Jan 2021)	NG	OK	OK	OK
w/o cplex	OK	OK	NG (macOS), OK (Ubuntu, Windows)	OK

• cplex 12.10 supports Python 3.6 and 3.7 https://pypi.org/project/cplex/12.10.0.3/#files
• cplex 20.1 supports Python 3.7 and 3.8 https://pypi.org/project/cplex/20.1.0.1/#files

Steps to reproduce the problem

from docplex.mp.model import Model
a = Model()
x = a.binary_var()
print(a.add_constraint(x <= 1))

Python 3.7, docplex 2.15.194 without cplex

x1 <= 1

Python 3.7, docplex 2.16.196 with cplex 20.1

x1 <= 1

Python 3.7, docplex 2.16.196 without cplex

Traceback (most recent call last):
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 290, in get_cplex_module
    import cplex  #@UnresolvedImport
ModuleNotFoundError: No module named 'cplex'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c.py", line 2, in <module>
    a = Model()
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/model.py", line 397, in __init__
    self._environment = self._make_environment()
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/model.py", line 174, in _make_environment
    env = Environment.get_default_env()
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 427, in get_default_env
    Environment._default_env = Environment.make_new_configured_env()
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 422, in make_new_configured_env
    return Environment(start_auto_configure=True)
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 84, in __init__
    self.auto_configure(logger=logger)
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 220, in auto_configure
    self.check_cplex(logger=logger)
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 328, in check_cplex
    cplex = self.get_cplex_module(logger=logger)
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 314, in get_cplex_module
    cplex = load_cplex_from_cos_root(loc) if loc else None
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 286, in load_cplex_from_cos_root
    return load_cplex(full_path, version=version)
  File "/Users/ima/envs/test/lib/python3.7/site-packages/docplex/mp/environment.py", line 261, in load_cplex
    raise FileNotFoundError("Could not load module from %s" % module_location)
FileNotFoundError: Could not load module from /Applications/CPLEX_Studio201/cplex/python/3.7/x86-64osx/cplex/__init__.py

Python 3.8, docplex 2.15.194 with and without cplex

x1 <= 1

Python 3.8, docplex 2.16.196 with cplex 20.1

x1 <= 1

Python 3.8, docplex 2.16.196 without cplex

Traceback (most recent call last):
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 290, in get_cplex_module
    import cplex  #@UnresolvedImport
ModuleNotFoundError: No module named 'cplex'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c.py", line 2, in <module>
    a = Model()
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/model.py", line 417, in __init__
    self._environment = self._make_environment()
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/model.py", line 181, in _make_environment
    env = Environment.get_default_env()
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 427, in get_default_env
    Environment._default_env = Environment.make_new_configured_env()
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 422, in make_new_configured_env
    return Environment(start_auto_configure=True)
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 84, in __init__
    self.auto_configure(logger=logger)
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 220, in auto_configure
    self.check_cplex(logger=logger)
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 328, in check_cplex
    cplex = self.get_cplex_module(logger=logger)
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 314, in get_cplex_module
    cplex = load_cplex_from_cos_root(loc) if loc else None
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 286, in load_cplex_from_cos_root
    return load_cplex(full_path, version=version)
  File "/Users/ima/envs/test/lib/python3.8/site-packages/docplex/mp/environment.py", line 261, in load_cplex
    raise FileNotFoundError("Could not load module from %s" % module_location)
FileNotFoundError: Could not load module from /Applications/CPLEX_Studio201/cplex/python/3.8/x86-64osx/cplex/__init__.py

Python 3.8, docplex 2.14.186 with cplex 20.1

Traceback (most recent call last):
  File "c.py", line 4, in <module>
    print(a.add_constraint(x <= 1))
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/model.py", line 3518, in add_constraint
    ct = self._add_constraint_internal(ct, ctname)
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/model.py", line 2995, in _add_constraint_internal
    self._post_constraint(ct)
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/model.py", line 3003, in _post_constraint
    ct_engine_index = self._create_engine_constraint(ct)
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/model.py", line 2881, in _create_engine_constraint
    return eng.create_linear_constraint(ct)
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/cplex_engine.py", line 937, in create_linear_constraint
    return self._make_cplex_linear_ct(cpx_lin_expr=cpx_linexp,
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/cplex_engine.py", line 925, in _make_cplex_linear_ct
    ret_add = fast_add_linear(self._cplex, cpx_lin_expr, cpx_sense, cpx_rhs, cpxnames)
  File "/Users/ima/envs/dev38/lib/python3.8/site-packages/docplex/mp/cplex_engine.py", line 215, in fast_add_linear1290
    cpxenv._apienc) as (rmat, nnz):
AttributeError: 'Environment' object has no attribute '_apienc'

Python 3.9, docplex 2.15.194 without cplex

x1 <= 1

Python 3.9, docplex 2.16.196 without cplex

Traceback (most recent call last):
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 290, in get_cplex_module
    import cplex  #@UnresolvedImport
ModuleNotFoundError: No module named 'cplex'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ima/tasks/1_2021/qiskit/aqua/tmp/c.py", line 2, in <module>
    a = Model()
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/model.py", line 417, in __init__
    self._environment = self._make_environment()
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/model.py", line 181, in _make_environment
    env = Environment.get_default_env()
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 427, in get_default_env
    Environment._default_env = Environment.make_new_configured_env()
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 422, in make_new_configured_env
    return Environment(start_auto_configure=True)
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 84, in __init__
    self.auto_configure(logger=logger)
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 220, in auto_configure
    self.check_cplex(logger=logger)
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 328, in check_cplex
    cplex = self.get_cplex_module(logger=logger)
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 314, in get_cplex_module
    cplex = load_cplex_from_cos_root(loc) if loc else None
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 286, in load_cplex_from_cos_root
    return load_cplex(full_path, version=version)
  File "/Users/ima/envs/dev39/lib/python3.9/site-packages/docplex/mp/environment.py", line 261, in load_cplex
    raise FileNotFoundError("Could not load module from %s" % module_location)
FileNotFoundError: Could not load module from /Applications/CPLEX_Studio201/cplex/python/3.9/x86-64osx/cplex/__init__.py

What is the expected behavior?

QuadraticProgram works fine.

Suggested solutions

I recommend to pin 2.15.194 (Jul 2020) for the time being because it is the only docplex that works with any Python 3.6-3.9 even without cplex wheel.

[t-imamichi]

I can fix this as part of #23

[t-imamichi]

We need to include a fix for Aqua 0.8.2 bugfix release. I suggest to pin docplex 2.15.194 for the release. What do you think?

[manoelmarques]

The PR #30 runs all unit tests under Ubuntu for all python versions, with and without cplex and cvxpy with no restrictions.

[woodsp-ibm]

As Manoel noted above, the unit tests are running with the latest docplex and all seem to be working fine without cplex being installed. Can we review the analysis that led to us pinning an older version in the recent stable release as to how to move forwards here? Are we missing some unit test that would show the failure that the review above hit - maybe its there on Mac but does not manifest in the platform used for the build?

[t-imamichi]

@manoelmarques @woodsp-ibm Thank you for the unit tests without cplex. I notice from the result that the issue of docplex without cplex occurs only on macOS. Docplex 2.20 works fine without cplex on Ubuntu and Windows. I updated the table. So, we need version pin of docplex is necessary for only macOS.

[woodsp-ibm]

@t-imamichi I see there is a cplex 20.1.0.1 that was released and says its from Feb 8th. Is it worth double checking the status of the docplex/cplex ahead of release to see if the version pinning we have in place still makes sense or whether it can be relaxed at all.

[t-imamichi]

Thank you for the notice. I tried them and confirmed that the two NGs in the table remain as of Mar. 26, 2021.
#24 (comment)

[t-imamichi]

This issue is resolved with docplex 2.21.207. This issue will be close by #155.