Upgrade cvxopt in sage from 0.9 to 1.1.3

[williamstein]

We are shipping an ancient version of cvxopt in sage. It's worse than Debian shipping sage-3.0.5!

Note that upgrading cvxopt should be fairly easy, since basically nothing in Sage depends on cvxopt.

New spkg: http://sage.math.washington.edu/home/dima/packages/cvxopt-1.1.3.spkg
and the patch to use the interface to GLPK : https://github.com/sagemath/sage/files/ticket6456/6456-cvxopt-glpk-interface.patch.gz

Upstream: Completely fixed; Fix reported upstream

Component: packages: standard

Author: Harald Schilly, Dmitrii Pasechnik

Reviewer: Mike Hansen, Karl-Dieter Crisman

Merged: sage-4.6.1.alpha2

Issue created by migration from https://trac.sagemath.org/ticket/6456

[wdjoyner]

comment:1

Fly in the ointment: The recent version has this statement on its license page http://abel.ee.ucla.edu/cvxopt/copyright.html:

CVXOPT is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

Older versions apparently did not have that.

[haraldschilly]

comment:2

I've recently upgraded it for me to 1.1.2. The problem is, that the sources from them did not compile on my ubuntu 9.10 machine. So I went to the ubuntu packges db for lucid and grabbed their version. I don't know what they did in their patch, but I guess it's non trivial...

My spkg is here.

The only remaining modification I had to made to run the examples from the cvxopt website in %python mode was to replace

from random import ...

to

from sage.misc.prandom import ...

in ./src/src/python/__init__.py at several places.

Q:

1. what's the usual/best mechanism to avoid using Sage's random and switch back to python's random?!
2. i have no idea what the solaris patches did in the older version, neither do i know how to get it building on another system :(

[haraldschilly]

comment:3

I've created an updated p1 spkg.

Using the 1.1.2 sources directly, I get this error site-packages/cvxopt/base.so: undefined symbol: _g95_stop_blank ... I also fiddled around with the setup.py file.

cvxopt 1.1.2 p1 spkg is here

[sagetrac-mvngu]

comment:4

Ticket #1620 is a duplicate of this one.

[dimpase]

comment:5

Replying to @haraldschilly:

I've created an updated p1 spkg.

Using the 1.1.2 sources directly, I get this error site-packages/cvxopt/base.so: undefined symbol: _g95_stop_blank ... I also fiddled around with the setup.py file.

cvxopt 1.1.2 p1 spkg is here

sage -t "devel/sage-work/sage/numerical/optimize.py"

bumps out essentially due to:

sage: from cvxopt import cholmod
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/home/dima/sage/devel/sage-work/sage/<ipython console> in <module>()

ImportError: /home/dima/sage/local/lib/python2.6/site-packages/cvxopt/cholmod.so: undefined symbol: _g95_filename
sage: 

Can you reproduce this on a stand-alone build of cvxopt?

[dimpase]

Attachment: trac-6456.patch.gz

patch for cvxopt-1.1.2.p1/src/src/setup.py

[dimpase]

comment:6

Replying to @haraldschilly:

I've created an updated p1 spkg.

Using the 1.1.2 sources directly, I get this error site-packages/cvxopt/base.so: undefined symbol: _g95_stop_blank ... I also fiddled around with the setup.py file.

cvxopt 1.1.2 p1 spkg is here

please check the patch I just uploaded. It fixes this problem;
you just had to link against more dynamic libs...

[haraldschilly]

comment:7

Replying to @dimpase:

please check the patch I just uploaded. It fixes this problem; you just had to link against more dynamic libs...

Thanks, I knew that it is something with that! Works now!!! ;)

I've uploaded 1.1.1.p2 here

Next I'll try if using the debian/ubuntu version of it was really necessary.

For everyone who wants to try this, don't forget that you have to disable the preparser in Sage via preparser(False) ... otherwise there are unknown types when you try to create a matrix with cvxopt's matrix command.

[dimpase]

comment:8

Replying to @haraldschilly:

I've uploaded 1.1.1.p2 here

Next I'll try if using the debian/ubuntu version of it was really necessary.

please take out f77blas all over in setup.py, for this is apparently obsolete and not needed - and also nukes the installation on Mac OS X (ppc) -- (otherwise it works on the latter platform)

[dimpase]

comment:9

Replying to @haraldschilly:

I think that really what remains to be done is to remove dependencies on
obsolete fortrans (f77), see my other comment on this.
Let's get it done!

Replying to @dimpase:

please check the patch I just uploaded. It fixes this problem; you just had to link against more dynamic libs...

Thanks, I knew that it is something with that! Works now!!! ;)

I've uploaded 1.1.1.p2 here

Next I'll try if using the debian/ubuntu version of it was really necessary.

I don't see a necessity to try non-debian/ubunty version.
If you look at the debian patches, you see that all they changed is the source
were slight tweaks in setup.py

For everyone who wants to try this, don't forget that you have to disable the preparser in Sage via preparser(False) ... otherwise there are unknown types when you try to create a matrix with cvxopt's matrix command.

Does this mean that we should think of importing cvxopt's matrix into Sage under some other name? I don't know what the usual Sage's way to deal with such things, i.e. name clashes between packages, is.

[sagetrac-drkirkby]

comment:10

• Have the license issue been resolved?
  • Has this been tested on Solaris?

[dimpase]

comment:11

Replying to @sagetrac-drkirkby:

• Have the license issue been resolved?

it is GPL v3 or later. Does it matter?
I suppose I can ask the authors to tweak it, if it is really necessary.

• Has this been tested on Solaris?

no, but I can try on one of Skynet's machines (perhaps you can tell me which
one is most likely to work :)), or you can try it yourself.
(I don't have a ready Solaris install anywhere).

[sagetrac-drkirkby]

comment:12

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

sage: license()

says all code except jsmath is GPL2, and apparently jsmath is ok, as Sage does not link to it.

The code will not build on skynet, as there are no SPARC machines there. It should build it on 't2' easily though.

I suggest you download sage 4.3.0.1 from one of the mirrors

http://www.sagemath.org/download-solaris.html

Use the following settings.

kirkby@t2:[~] $ echo $PATH
/usr/local/gcc-4.4.1-sun-linker/bin:/usr/local/bin2:/usr/bin:/usr/ccs/bin:/usr/local/bin:/usr/sfw/bin:/bin:/usr/sbin
kirkby@t2:[~] $ echo $LD_LIBRARY_PATH
/usr/local/gcc-4.4.1-sun-linker/lib:=/usr/local/gcc-4.4.1-sun-linker/lib/sparcv9:/usr/local/lib

type make, and build Sage, then try your package.

There is also a binary of Sage on the mirrors. You could download that. I'm not precisely sure what you then need to do to build just your package using the binary as a starting point.

The latest Sage source will not build on Solaris, but 4.3.0.1 will.

Dave

[dimpase]

comment:13

Replying to @sagetrac-drkirkby:

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

Well, cvxopt is an optional package, so it must be in the same boat as jmath,
or some gap packages, that are also not GPL 2.

sage: license()

says all code except jsmath is GPL2, and apparently jsmath is ok, as Sage does not link to it.

The code will not build on skynet, as there are no SPARC machines there. It should build it on 't2' easily though.

hmm, isn't this a sparc/solaris?

SunOS mark2 5.10 Generic_127111-01 sun4u sparc SUNW,Sun-Blade-2500

Dima

[sagetrac-drkirkby]

comment:14

Replying to @dimpase:

Replying to @sagetrac-drkirkby:

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

Well, cvxopt is an optional package, so it must be in the same boat as jmath,
or some gap packages, that are also not GPL 2.

Fair enough.

sage: license()

says all code except jsmath is GPL2, and apparently jsmath is ok, as Sage does not link to it.

The code will not build on skynet, as there are no SPARC machines there. It should build it on 't2' easily though.

hmm, isn't this a sparc/solaris?

SunOS mark2 5.10 Generic_127111-01 sun4u sparc SUNW,Sun-Blade-2500

Dima

Yes, it is. Sorry, I was not aware of the existance of that machine.

However, I do not know how the compilers and paths are configured on that machine. You need to have GNU make & GNU tar in your path before the Sun ones, and you need to have the Sun linker (/usr/ccs/bin/ld) in your path before any GNU ones. There are some general instructions on building Sage on Solaris at http://wiki.sagemath.org/solaris which you would need to follow.

I've written some somewhat simpler instructions at http://wiki.sagemath.org/devel/Building-Sage-on-the-T5240-t2 on how to build Sage on 't2'. They are simpler, as I have already put the right tools in the right locations.

The Sun Blade 2500 (mark2) should be quicker than the T5240 (t2) at building Sage. However it would require some setting up of the build environment to build Sage. If you just want an easy solution, 't2' will just work, albeit not as quickly as the Sun Blade 2500.

Dave

[dimpase]

comment:15

Replying to @sagetrac-drkirkby:

hmm, isn't this a sparc/solaris?

SunOS mark2 5.10 Generic_127111-01 sun4u sparc SUNW,Sun-Blade-2500

Dima

Yes, it is. Sorry, I was not aware of the existance of that machine.

However, I do not know how the compilers and paths are configured on that machine. You need to have GNU make & GNU tar in your path before the Sun ones, and you need to have the Sun linker (/usr/ccs/bin/ld) in your path before any GNU ones. There are some general instructions on building Sage on Solaris at http://wiki.sagemath.org/solaris which you would need to follow.

well, on Skynet there is /usr/local/skynet_bash_profile
that you can source upon login (from .bashrc, or just manually),
and this gives you the ready setup to build Sage.

I don't have an account on t2, it seems to me.
By the way, absent-mindedly I started building 4.3.3.alpha on there,
and it went till gnutls, where it stopped... I noticed that gnutls is over 2 years old, version 2.2.2, whereas the current one is 2.8.5. Shouldn't one upgrade to this one, before even trying to fix this?

Dima

[dimpase]

comment:16

Replying to @sagetrac-drkirkby:

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

I emailed William about this, and he said it's OK in this case, it can be v3.

[sagetrac-drkirkby]

comment:17

Replying to @dimpase:

Replying to @sagetrac-drkirkby:

However, I do not know how the compilers and paths are configured on that machine. You need to have GNU make & GNU tar in your path before the Sun ones, and you need to have the Sun linker (/usr/ccs/bin/ld) in your path before any GNU ones. There are some general instructions on building Sage on Solaris at http://wiki.sagemath.org/solaris which you would need to follow.

well, on Skynet there is /usr/local/skynet_bash_profile
that you can source upon login (from .bashrc, or just manually),
and this gives you the ready setup to build Sage.

I'm not however aware of anyone building Sage on there recently, so I don't know if the environment is set up suitably. Quite a few Solaris-specific changes have been made in the last year, and some of them might not be compatible with the build system on there. I don't know. Specifically, if

gcc -v 

shows gcc was configured with the GNU linker, then the GNU linker must be in your path before the Sun linker. (Basically, whatever linker gcc uses, must be in your path first, as some code makes the assumption the first linker in your path is the one gcc uses, which might not be true.) I'm not aware of a foolproof test of this.

There should be something like

--with-ld=/usr/ccs/bin/ld

if the Sun linker was used, or

--with-ld=/path/to/gnu/ld

if the GNU linker was used.

Certainly, I've never had a problem with gnutls failing on 't2'.

I don't have an account on t2, it seems to me.

I've emailed William to ask if you can have an account on t2, as that might be the simplest solution, though that Blade 2500 would be significantly faster than 't2'.

By the way, absent-mindedly I started building 4.3.3.alpha on there,
and it went till gnutls, where it stopped... I noticed that gnutls is over 2 years old, version 2.2.2, whereas the current one is 2.8.5. Shouldn't one upgrade to this one, before even trying to fix this?

Dima

I can see your point about upgrading, though I'm not aware of any particular issues with the version in Sage. That version will build on Solaris. You could try appending /usr/sfw/lib to your LD_LIBRARY_PATH. I have known of issues with gnutls on OpenSolaris, but not on Solaris 10.

I see your post about the GPL 3. That bit is sorted out then.

Dave

[dimpase]

comment:18

Replying to @dimpase:

Replying to @haraldschilly:

Here is an updated version, that also works on Solaris (this needed copying sun_complex.h from old cvxopt-0.9
and patching cvxopt.h). I also turned on building GSL-extension (by turning on the appropriate option in setup.py,
and supplying right include and lib-paths)

http://boxen.math.washington.edu/home/dima/packages/cvxopt-1.1.2.p3.spkg

[dimpase]

changes for the doctested examples in the numerical_sage section for cvxopt

[dimpase]

comment:154

Attachment: 6456-numerical_sage_cvxopt-with-updated-commitmessage.patch.gz

Replying to @jdemeyer:

Replying to @dimpase:

Also, you should change the commit message of the patches (use hg qrefresh -e for that) such that the first line contains the ticket number and a short description of the patch (you may use following lines for a longer description).

done.

Except that the ticket number in attachment: 6456-numerical_sage_cvxopt-with-updated-commitmessage.patch is wrong...

AUB

[jdemeyer]

Merged: sage-4.6.1.alpha2

[qed777]

comment:156

There's a failure with 4.6.1.alpha2 on Skynet's iras (ia64-Linux-suse):

$ ./sage -t -long  -force_lib devel/sage/doc/en/numerical_sage/cvxopt.rst 
sage -t -long -force_lib "devel/sage/doc/en/numerical_sage/cvxopt.rst"
**********************************************************************
File "/tmp/sage_iras/sage-4.6.1.alpha2/devel/sage/doc/en/numerical_sage/cvxopt.rst", line 129:
    sage: print sol['x']      # ... below since can get -00 or +00 depending on architecture
Expected:
    [ 1.00e+00]
    [ 1.00e+00]
Got:
    [ 1.00e-00]
    [ 1.00e+00]
    <BLANKLINE>

[jdemeyer]

comment:157

Replying to @qed777:

There's a failure with 4.6.1.alpha2 on Skynet's iras (ia64-Linux-suse):

Patch at #10309

[jdemeyer]

Attachment: 10309_cvxopt_doctest.patch.gz

[kcrisman]

comment:158

Is there anything else that needs to happen for this to get positive review? I'm unclear on how spkgs merged before positive review finally do get positive review.

[jdemeyer]

comment:159

Replying to @kcrisman:

Is there anything else that needs to happen for this to get positive review?

Somebody should look at the spkg, check that spkg-install, SPKG.txt make sense, check any patches (I believe there are none in this spkg),...

[dimpase]

comment:160

Replying to @jdemeyer:

Replying to @kcrisman:

Is there anything else that needs to happen for this to get positive review?

Somebody should look at the spkg, check that spkg-install, SPKG.txt make sense, check any patches (I believe there are none in this spkg),...

Being an author, I am biased, but I think it has been reviewed over and over to near-death already, and found sound enough :-)

[kcrisman]

comment:161

Replying to @dimpase:

Replying to @jdemeyer:

Replying to @kcrisman:

Is there anything else that needs to happen for this to get positive review?

Somebody should look at the spkg, check that spkg-install, SPKG.txt make sense, check any patches (I believe there are none in this spkg),...

Being an author, I am biased, but I think it has been reviewed over and over to near-death already, and found sound enough :-)

Of course!

I have a brief question about the patches. The Solaris one and the init one seem fine, and most of the setup.py makes sense for finding Sage-specific things.

I do have a question about BUILD_GLPK being set to 1, since there is a comment that one can only do this if glpk is installed, "optional at this moment". So will this cause problems? Apparently not... There is a hanging parenthesis there as well. Anyway, I'd like to know what that is about.

Also, in SPKG.txt there is a spelling "enchance" instead of "enhance", and "recongnision" instead of "recognition".

Everything else seems to make sense, though I should note I am not a shell script or install expert :) Still, with no problems reported on 4.6.1.alpha2, perhaps it's time for positive review once those things get resolved.

[dimpase]

comment:162

Replying to @kcrisman:

I do have a question about BUILD_GLPK being set to 1, since there is a comment that one can only do this if glpk is installed, "optional at this moment". So will this cause problems? Apparently not... There is a hanging parenthesis there as well. Anyway, I'd like to know what that is about.

Fixed. A leftover from times when GLPK was still optional. Now GLPK is standard.

Also, in SPKG.txt there is a spelling "enchance" instead of "enhance", and "recongnision" instead of "recognition".

fixed.

Everything else seems to make sense, though I should note I am not a shell script or install expert :) Still, with no problems reported on 4.6.1.alpha2, perhaps it's time for positive review once those things get resolved.

OK so the fixes are in the spkg file in the same location.

Thanks for your time,

Dima

[kcrisman]

comment:163

Okay, these were just cosmetic changes, and I verified they are in the updated spkg, and it's been tested at a basic level without destroying Sage for a while now, so this should go in.

[jdemeyer]

comment:164

Since the package is not pure upstream and contains pacthes, I believe it should be renamed to cvxopt-1.1.3.p0.spkg.

[sagetrac-drkirkby]

comment:165

Replying to @jdemeyer:

Since the package is not pure upstream and contains pacthes, I believe it should be renamed to cvxopt-1.1.3.p0.spkg.

I thought this had been debated on sage-devel before, and William's opinion is that it should not be .p0. There's noting in the Sage Developers Guide to say it should be either. As such, I don't think that this is a reasonable reason for putting the ticket to needs work.

Dave

[dimpase]

comment:166

Replying to @jdemeyer:

Since the package is not pure upstream and contains pacthes, I believe it should be renamed to cvxopt-1.1.3.p0.spkg.

I believe the opposite, and was specifically indoctorinated into my belief when I was updating GAP and its GAP-packages spkgs.
Hereby I blatantly change the status to Positive Review :-)