-
-
Notifications
You must be signed in to change notification settings - Fork 348
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove has no python #483
Remove has no python #483
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a couple of places where HAS_NO_PYTHON
is still referenced in the test suite, which should be cleaned up.
The mess involved in iterating over Python 2 and 3 certainly makes me look forward to dropping support for Python 2...
@@ -109,6 +109,10 @@ if 'clean' in COMMAND_LINE_TARGETS: | |||
else: | |||
Alias('clean', []) | |||
|
|||
if 'test-clean' in COMMAND_LINE_TARGETS: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This command needs to be documented in the docstring for SConstruct
(and possibly elsewhere)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I need to update the docs for the new python2_*
options I added too. And probably also describe the small changes in behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK I updated the docs for most things, but this command was already documented in the docstring for SConstruct
. The bit added here just cleans up some more files that weren't being cleaned up before.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, sorry, my mistake. I forgot that this was already a command. 😬
SConstruct
Outdated
if env['python2_package'] == 'default': | ||
env['python2_package'] = 'n' | ||
if env['python3_package'] == 'default': | ||
env['python3_package'] = 'n' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The end effect is the same, but if none
is the canonical name, we should use that here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
sys.exit(1) | ||
elif want_python: | ||
print('WARNING: ' + message) | ||
env['python_package'] = 'minimal' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does setting python_package
at this point have the desired effect, or does the code above which sets python2_package
and python3_package
based on python_package
need to come after this point?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this has the desired effect (at least, I'm pretty sure its doing what I expect it to do; whether that's desired or not is a different question 😄).
If any of python_package
, python2_package
, or python3_package
are full
(after the code above, y
and new
become full
), require_python
is True
, and if Cython can't be imported with the python_cmd
interpreter, we want to error out, because the build can't be completed.
If all of python_package
, python2_package
, and python3_package
are 'default'
(or any combination are 'none'
(but not all of them 'none'
) with any one of them as 'default'
), then require_python
will be False
and want_python
will be True
. In that case, if we can't find Cython, python_package
will be set to 'minimal'
and the flow will continue to check what version of Python is specified in python_cmd
. Then, the Python in python_cmd
will be used to build the minimal interface for that version of Python (provided that the python*_package
that matches the version of Python in python_cmd
is not specified).
That's what I wanted this line to do, and I'm pretty sure that's what its doing.
SConstruct
Outdated
print("ERROR: The python_package points to Python {v} and python{v}_package has been " | ||
"specified. Please change python_package to point to a different version of " | ||
"Python or disable it.".format(v=major)) | ||
sys.exit(1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find this error message confusing. Is the idea that specifying both python_package
and whichever of python3_package
or python2_package
corresponds to python_cmd
is not allowed?
Should we allow this if the settings are not incompatible?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's the intention. How would we determine if the settings are incompatible? What if python_cmd
points to Python 3.5 and python3_cmd
points to Python 3.6? Which should we build? I think it simpler if we just error out when both are specified, although the error message could certainly be improved.
SConstruct
Outdated
sys.exit(1) | ||
elif env[py_pkg] == 'default': | ||
py_vars = {'python{}_array_home': '', 'python{}_cmd': 'python{}'.format(major), | ||
'python{}_prefix': '$prefix'} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't these values correspond to the values of env['python_cmd']
etc.?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are the default values for these variables. I'm checking whether the value present in the env
is equal to the default. This could be clearer, I'll clean it up.
SConstruct
Outdated
" import numpy", | ||
" print(numpy.__version__)", | ||
"except ImportError as err:", | ||
# " print(err, file=sys.stderr)", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not print the error?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The error is reported later, so the printing here is duplicate (and not very clear, because it doesn't say which interface is reporting the error).
SConstruct
Outdated
@@ -1787,6 +1821,9 @@ if 'msi' in COMMAND_LINE_TARGETS: | |||
|
|||
### Tests ### | |||
if any(target.startswith('test') for target in COMMAND_LINE_TARGETS): | |||
if env['python2_package'] == 'none' and env['python3_package'] == 'none': | |||
env['python{}_package'.format(env['python_version'][0])] = 'minimal' | |||
SConscript('interfaces/python_minimal/SConscript') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder whether this is insufficient to do what you presumably want, and doesn't trigger the 'build' actions in the SConscript file when you only run scons test
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is indeed insufficient, and I need to fix this still.
interfaces/cython/SConscript
Outdated
@@ -172,7 +170,7 @@ if localenv['python_package'] == 'full': | |||
py2env['py_extension'] = ext[0].name | |||
py2env.SubstFile('setup2.py', 'setup.py.in') | |||
build_cmd = ('cd interfaces/cython &&' | |||
' $python_cmd_esc setup2.py build --build-lib=../../build/python2') | |||
' $python2_cmd setup2.py build --build-lib=../../build/python2') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we will continue to need to use a quoted/escaped Python command for the same reason it was originally introduced (see 4f3dd06)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But the Python 3 interface does not use the quoted command. Should that be changed to use a quoted command?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, presumably. I guess the quoting can be done more locally than in the current version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we then use the escaped command everywhere? Like, all the places we check for NumPy, in the tests, etc?
interfaces/python_minimal/SConscript
Outdated
localenv.Depends(make_setup, s) | ||
|
||
build_cmd = ('cd interfaces/python_minimal &&' | ||
' $python{v}_cmd setup{v}.py build --build-lib=../../build/python{v}'.format(v=v)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
likewise here
I was thinking this morning that it would probably make sense to abstract all the Python checking code into a function so that the loop is neater, so I'll do that. It will indeed be nice when we can drop Python 2, but then they'll release Python 4 which won't be compatible with Python 3 😃 |
Hopefully the decade of growing pains associated with the Python 2 to 3 transition will convince them never to make a Python 4. |
@speth What is the Also, what is the purpose of this logic in setup_cantera: if [ "@python_module_loc_sc@" != "" ]; then
if [ -z $PYTHONPATH ]; then
PYTHONPATH=@python_module_loc@
else
PYTHONPATH=@python_module_loc@:$PYTHONPATH
fi
fi Why do we check the value of Also, in those same two SConscript files, why do we depend on |
interfaces/cython/SConscript
Outdated
""") | ||
b = build(py2env.Command([], '#build/python2/cantera/examples', convert_script)) | ||
py2env.Depends(b, a) | ||
py2env.Depends(mod, b) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@speth I'm really not sure what's going on here. I'm trying to run the converter before the module is built, and after the example files are copied to the right place. Instead, the setup script is copying the example files and isn't converting them. I don't see how what I'm doing here is different from how it used to be in terms of the dependency, but clearly something is different. Any advice would be appreciated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it because the Depends
is on a folder, rather than a set of files?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the issue is that in your new way, b
is an empty list (because you specify an empty list as the target
for the Command
), so there's no state for SCons to base the dependency on. I think what you actually want here is env.AddPostAction
, which will run after a
, but only if the a
action has to run in the first place.
OK so the tests all pass now (but installation is still broken, pending clarification of my other comment above), but 3to2 can't be found because I changed how the executable was found. So rather than relying on the executable, who's installation location is platform- and installation-method-dependent, I thought it would be better to run the converter function directly from the library. However, as I mentioned in my comment, its not actually running that code, so I don't know what's going wrong. |
The
What's the alternative, though, if the user is installing to a directory that's not on
Beats me. I have no idea why both Python 2 and Python 3 look at the same variable for this, it makes working with both simultaneously a nightmare. I guess at this point I would default to the Python 3 module location if both are built? |
OK, I suspected that was the case, but glad to have confirmation.
How often do users install to non-standard paths? I guess this is something we have no way of knowing, and you're right that A potentially better way of doing this is to install a FWIW, this would work like bryan@solus ~$ echo "/home/bryan/canteratest" > ~/.local/lib/python3.6/site-packages/cantera.pth
bryan@solus ~$ python
Python 3.6.3 |Anaconda, Inc.| (default, Oct 6 2017, 17:14:46)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/bryan/miniconda3/lib/python36.zip', '/home/bryan/miniconda3/lib/python3.6', '/home/bryan/miniconda3/lib/python3.6/lib-dynload',
'/home/bryan/.local/lib/python3.6/site-packages', '/home/bryan/canteratest', '/home/bryan/miniconda3/lib/python3.6/site-packages'] Note that the
Will do. What should I do about this line: cantera/platform/posix/SConscript Line 26 in 684262e
|
6db0057
to
5e0685b
Compare
I don't really understand the point of
Yes, if we have are building the Python 3 module, then I think this should be for the Python 3 install action, and we should be using the |
Presumably, the user doesn't want to install to that directory for some reason. However, it should be writable in general, and if we were to remove the
Yes, I agree with that use case, that would be the disadvantage of the
👍
Ah, that makes much more sense, thanks! |
In f982156 I reformatted the Provided that the tests pass and there are no further requested changes, I think this is good to go. |
SConstruct
Outdated
try: | ||
print(site.getusersitepackages()) | ||
except AttributeError: | ||
print(site.USER_SITE)""") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the prefered indentation following textwrap.dedent
? I see this style used in the documentation for the textwrap
module, but honestly, I think it looks horrible -- it's very hard to see where the triple-quoted string ends. Wouldn't it be better to indent this one level?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean like
script = textwrap.dedent("""\
from __future__ import print_function
import sys
print('{v.major}.{v.minor}'.format(v=sys.version_info))
try:
import numpy
print(numpy.__version__)
except ImportError:
print('0.0.0')
import site
try:
print(site.getusersitepackages())
except AttributeError:
print(site.USER_SITE)
""")
SConstruct
Outdated
if env['python2_package'] == 'full': | ||
env['python2_example_loc'] = pjoin(env['python2_module_loc'], 'cantera', 'examples') | ||
install_message += textwrap.indent(textwrap.dedent(""" | ||
Python 2 package (cantera) {python2_module_loc!s} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
textwrap.indent
is new as of Python 3.3, so we can't use it.
I guess this also means our automated tests don't run the install
action, so this isn't getting tested, which probably ought to be fixed as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I guess I'll write my own indent function for this specific case. Can't wait until Python 2 is done...
Yes, I realized that as well, but didn't want to make this PR any bigger... it already blew up a little bit 😄
interfaces/cython/SConscript
Outdated
else: | ||
# Install Python module in the default location | ||
extra = '' | ||
|
||
env['python%s_module_loc' % ver] = '<unspecified>' | ||
env['python%s_module_loc' % major] = '<unspecified>' | ||
localenv['python{}_cmd_esc'.format(major)] = quoted(env['python{}_cmd'.format(major)]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't this redundant with the setting of pythonX_module_loc
in the block for each Python version below before the install_module
function is called?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As best I can tell, pythonX_module_loc
is only set in the install_module
function. This is set to the empty string in SConstruct, but only if that Python interface isn't being built.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I got confused and read the wrong line while writing that -- I meant pythonX_cmd_esc
.
In 1c46173 I fixed a problem with parallel building of the Matlab interface on macOS. Prior to that change, building in parallel with the Matlab interface was giving g++ -o interfaces/matlab/toolbox/ctmethods.mexmaci64 -Wl,-exported_symbol,_mexFunction -dynamiclib build/src/matlab/ctfunctions.os build/src/matlab/ctmethods.os build/src/matlab/flowdevicemethods.os build/src/matlab/funcmethods.os build/src/matlab/kineticsmethods.os build/src/matlab/mixturemethods.os build/src/matlab/onedimmethods.os build/src/matlab/phasemethods.os build/src/matlab/reactormethods.os build/src/matlab/reactornetmethods.os build/src/matlab/reactorsurfacemethods.os build/src/matlab/surfmethods.os build/src/matlab/thermomethods.os build/src/matlab/transportmethods.os build/src/matlab/wallmethods.os build/src/matlab/xmlmethods.os -Lbuild/lib -L/Applications/MATLAB_R2016b.app/bin/maci64 -lcantera -lmx -lmex -lmat -framework Accelerate
ld: archive has no table of contents file 'build/lib/libcantera.a' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [interfaces/matlab/toolbox/ctmethods.mexmaci64] Error 1
ranlib build/lib/libcantera.a
scons: building terminated because of errors. which I think was because the Matlab library was building against the static Cantera library which wasn't finished yet. However, a few lines below my change, we set the Matlab library to depend on the Cantera shared library, which is why I changed to put the shared libs in the link line. Is that the right fix? |
I don't recall offhand the reason why we link to the static library on macOS, but I'd rather leave that alone and change the |
1c46173
to
052a04f
Compare
OK I made the change to the |
OK, I think this is fine as far as the end state goes. Could you squash a few of the commits that are fixups internal to this PR, to reduce the amount of churn, and then we can merge this? |
This variable is not needed since in the two places its used we do a runtime check of the presence of the Python interface anyways
Use relative imports in the package, compatible with Python 2 and 3
If no full or minimal Python interface is being built, copy the minimal interface into the build directory and use the sys.executable to run it, so the tests that require CTML or CTI conversion can run.
052a04f
to
17f8dd6
Compare
Done and done! Thanks for all the comments, very helpful to understand how SCons works a bit better 😄 |
Sorry, just found a couple more problems when trying this PR on the Buildbot builders (http://gir.mit.edu:8010/waterfall) -- there appear to be a couple of uses of |
Update and make more consistent the specification of Python package building. Since SCons can be run by Python 3 now, we cannot assume that the Python running SCons is Python 2. This changes a bunch of assumptions in SConstruct about where things should be built or installed. This commit addresses those assumptions by making the options for Python 2 and Python 3 symmetric.
Use Python 3 variables by default, and Python 2 only if Python 3 isn't being built
17f8dd6
to
37392a0
Compare
Switch to importing the lib3to2 as a check, which is platform agnostic and doesn't depend on how 3to2 was installed. Also, take advantage of the fact that the 3to2 converter recurses by default to avoid spawning a bunch of subprocesses. Finally, don't depend on the location of the 3to2 script and just use the library directly to do the conversion.
The triple quoted strings made 3to2 think that was the module docstring, so it was putting __future__ imports in the wrong places
Building in parallel would error out because the macOS Matlab library was relying on the static Cantera library before that was finished.
37392a0
to
4447830
Compare
@speth Good catch, thanks! Fixed now. This is why I deleted those variables 😄 although clearly I didn't test the packaging machinery |
Changes proposed in this pull request:
HAS_NO_PYTHON
configuration variable to allow building just the library, then adding the Python interfaces without rebuilding everything. See https://groups.google.com/forum/#!topic/cantera-users/a3IK3A4XEfII'd appreciate some feedback on this, since the changes are pretty big to SConstruct. AFAICT, everything should keep working with old versions of SCons. This is still WIP because the tests aren't working yet, and I need to update the documentation (if this gets approved). Would still appreciate some feedback while I try to fix the tests.