list_plot3d plots extraneous points at z=0 and doesn't take color or rgbcolor as keywords

[ppurka]

1. list_plot3d plots extra points at the horizontal x-y plane. This seems to stem from the default value of 0.0 that is assigned in the code. See this post.

2. This function also does not take in color or rgbcolor as keywords. Rest of the plot commands including plot, list_plot, plot3d do take in these keywords.

First attached patch fixes both these problems, second one fixes more doc and clarifies output.

---

Apply attachment: trac_12798-fix_extraneous_points_and_kwds_in_list_plot3d.patch, attachment: trac_12798-more-doc.3.patch and attachment: trac_12798-dont_pass_nan_to_ParametricSurface.patch to devel/sage.

CC: @kcrisman

Component: graphics

Keywords: list_plot3d, sd40.5

Author: Punarbasu Purkayastha, Karl-Dieter Crisman

Reviewer: Karl-Dieter Crisman, Punarbasu Purkayastha, Jeroen Demeyer

Issue created by migration from https://trac.sagemath.org/ticket/12798

[ppurka]

Author: Punarbasu Purkayastha

[kcrisman]

Changed keywords from list_plot3d to list_plot3d, sd40.5

[kcrisman]

comment:3

I like this idea, but have a couple questions.

• The way you have it set up currently has color etc. overriding texture. Should we be explicit about that? Is it even desirable? (Maybe it is, I just want to ask, as I am not sure.)
• Do you have any idea what the original default 0.0 was supposed to have done in the past (say, for an empty plot)? I'm not suggesting you know, and I did read your very sensible post, but I'm just curious for any ideas you may have.

[ppurka]

comment:4

Replying to @kcrisman:

I like this idea, but have a couple questions.

• The way you have it set up currently has color etc. overriding texture. Should we be explicit about that? Is it even desirable? (Maybe it is, I just want to ask, as I am not sure.)

Actually, color and the other keywords will never override texture. color will be used only when texture is not present, and color is present. As it is setup, if texture is provided, then the others should be ignored. Actually, maybe we should do this explicitly and remove the other keywords if they are present. Something like

if texture == 'automatic':
    if 'color' in kwds:
        texture = kwds.pop('color')
        if 'rgbcolor' in kwds:
            del kwds['rgbcolor']
     ...
texture = rgbcolor(texture)
if 'color' in kwds:
    del kwds['color']
...

Does that sound reasonable?

• Do you have any idea what the original default 0.0 was supposed to have done in the past (say, for an empty plot)? I'm not suggesting you know, and I did read your very sensible post, but I'm just curious for any ideas you may have.

I have no idea why this was done. Was it some bug in matplotlib earlier? Was it just an oversight? For an empty plot, I suppose you would not want there to be plot points at 0.0!

[kcrisman]

Reviewer: Karl-Dieter Crisman

[kcrisman]

comment:5

I like this idea, but have a couple questions.

• The way you have it set up currently has color etc. overriding texture. Should we be explicit about that? Is it even desirable? (Maybe it is, I just want to ask, as I am not sure.)

Actually, color and the other keywords will never override texture. color will be used only when texture is not present, and color is present.

Sorry for the inaccuracy, of course that is what I meant.

As it is setup, if texture is provided, then the others should be ignored. Actually, maybe we should do this explicitly and remove the other keywords if they are present. Something like

if texture == 'automatic':
    if 'color' in kwds:
        texture = kwds.pop('color')
        if 'rgbcolor' in kwds:
            del kwds['rgbcolor']
     ...
texture = rgbcolor(texture)
if 'color' in kwds:
    del kwds['color']
...

Does that sound reasonable?

Yes, as long as we document it. We do similar things lots of other places.

• Do you have any idea what the original default 0.0 was supposed to have done in the past (say, for an empty plot)? I'm not suggesting you know, and I did read your very sensible post, but I'm just curious for any ideas you may have.

I have no idea why this was done. Was it some bug in matplotlib earlier? Was it just an oversight? For an empty plot, I suppose you would not want there to be plot points at 0.0!

Hmm, looking at your plots from the post, I think I know. The idea was probably to have the list_plot cause a list to be plotted and then have the plot be zero elsewhere. Then there is the weird interpolation thing going on. Anyway, this makes things interpretable, though I'm not sure the original was really that useful, due to the output. I wonder who has used this in the past...

[ppurka]

comment:6

Updated the patch.

[kcrisman]

comment:7

Okay, I think this idea is okay. William says to defer to what Mathematica appears to do, since this is where the function was inspired, many years ago.

There will be holes in the surface corresponding to array elements that do not represent explicit height values.

And see here, the second example - clearly not continued beyond the three points given.

---

But I can't get it to work.

sage:  list_plot3d([(0,0,1), (2,3,4)], color='black')

sage: list_plot3d([(0,0,1), (2,3,4)], color='black', rgbcolor='#0f0')

sage: list_plot3d([(0,0,1), (2,3,4)], rgbcolor='#0f0')

sage: list_plot3d([(0,0,1), (2,3,4)], rgbcolor='#fff')

sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)], rgbcolor='#fff')

sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)], rgbcolor='#00f')

sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)], color='black')

None of these really seemed to look like what I wanted, or even sometimes to have anything plot. Am I using this incorrectly?

[ppurka]

comment:8

sage:  list_plot3d([(0,0,1), (2,3,4)], color='black')

sage: list_plot3d([(0,0,1), (2,3,4)], color='black', rgbcolor='#0f0')

sage: list_plot3d([(0,0,1), (2,3,4)], rgbcolor='#0f0')

sage: list_plot3d([(0,0,1), (2,3,4)], rgbcolor='#fff')

There seems to be a problem with the colors here. This is a bug that is present even without my patch. It seems like the texture is not passed onto the final function which plots the line.

sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)], rgbcolor='#fff', num_points=100)

sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)], rgbcolor='#00f', num_points=100)

sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)], color='black', num_points=100)

Can you check these with the additional arguments num_points=100 or higher numbers? It seems there are not enough interpolation points and so the plot is either missing or very jagged with a small value of num_points. With the default value of f.default_value = 0.0 (as it was earlier), you do get a plot but the plot is completely incorrect and useless also.

[ppurka]

Apply to devel/sage

[ppurka]

comment:9

Attachment: trac_12798-fix_extraneous_points_and_kwds_in_list_plot3d.patch.gz

Ok. I updated the patch to fix the color in 3D lines.

The other problem with interpolation and num_points is something that I don't know how to fix. Giving a high value of num_points gives a very good and accurate plot, but is also very slow. I have added some notes regarding this in the documentation of list_plot3d.

[kcrisman]

comment:10

Great, color works now - nice catch.

It turns out that the documentation for what this function actually does is insanely bad. I'm going to upload a patch clarifying what it actually does, which is not intuitive at all.

As for the issue you are seeing, I think it's very interesting. Notice that the box that comes up does have the right dimensions to incorporate all three points, even when you don't specify num_points!

Here is what is going on - I've printed out the vals. All those nan guys will lead to not much of a plot.

sage: sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)],interpolation_type='linear')
[[ -1.00000000e+00              nan              nan              nan               nan              nan]
 [             nan   6.66133815e-16              nan              nan               nan              nan]
 [             nan              nan              nan              nan               nan              nan]
 [             nan              nan              nan   2.00000000e+00               nan              nan]
 [             nan              nan              nan              nan               nan              nan]
 [             nan              nan              nan              nan               nan              nan]]

So you pick your poison here. Either you get something that does have the right points, but then also has everything else be extended by zero, or you don't do that, but the algorithm fails in certain cases. This will happen whenever the "box" in question is a lot bigger than the projection of the actual points to the xy-plane.

sage: sage: list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)]) # bad
sage: sage: list_plot3d([(0,5,1), (5,5,4), (-1,-5,-1)]) # good

I'm good with your work. If you like my changes, sign off as a reviewer and we'll get this in.

[kcrisman]

comment:11

Patchbot: Apply trac_12798-fix_extraneous_points_and_kwds_in_list_plot3d.patch and trac_12798-more-doc.patch.

[kcrisman]

Changed author from Punarbasu Purkayastha to Punarbasu Purkayastha, Karl-Dieter Crisman

[ppurka]

only added #long time to last example

[ppurka]

Changed reviewer from Karl-Dieter Crisman to Karl-Dieter Crisman, Punarbasu Purkayastha

[ppurka]

comment:12

Attachment: trac_12798-more-doc.2.patch.gz

Your patch looks good. The documentation was indeed lacking. I added just one small comment (# long time) in the last example in your patch. Positive review from me.

[jdemeyer]

comment:41

I guess the method sage.plot.plot3d.parametric_surface.ParametricSurface.triangulate() should be fixed not to add any faces of which the coordinates are not all finite numbers. However, I'm having a hard time understanding that method.

Because now we are sending a list of faces to be plotted full of NaNs:

sage: G = list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)],num_points=100)
sage: G.triangulate()
sage: G.face_list()
[...[(1.9696971120698907, 2.71717269474511, nan), (1.9696971120698907, 2.757576737669652, nan), (2.000000142649374, 2.757576737669652, nan), (2.000000142649374, 2.71717269474511, nan)], [(1.9696971120698907, 2.757576737669652, nan), (1.9696971120698907, 2.7979807805941936, nan), (2.000000142649374, 2.7979807805941936, nan), (2.000000142649374, 2.757576737669652, nan)], [(1.9696971120698907, 2.7979807805941936, nan), (1.9696971120698907, 2.8383848235187354, nan), (2.000000142649374, 2.8383848235187354, nan), (2.000000142649374, 2.7979807805941936, nan)], [(1.9696971120698907, 2.8383848235187354, nan), (1.9696971120698907, 2.8787888664432773, nan), (2.000000142649374, 2.8787888664432773, nan), (2.000000142649374, 2.8383848235187354, nan)], [(1.9696971120698907, 2.8787888664432773, nan), (1.9696971120698907, 2.919192909367819, nan), (2.000000142649374, 2.919192909367819, nan), (2.000000142649374, 2.8787888664432773, nan)], [(1.9696971120698907, 2.919192909367819, nan), (1.9696971120698907, 2.959596952292361, 3.9494949494949494), (2.000000142649374, 2.959596952292361, nan), (2.000000142649374, 2.919192909367819, nan)], [(1.9696971120698907, 2.959596952292361, 3.9494949494949494), (1.9696971120698907, 3.000000995216903, nan), (2.000000142649374, 3.000000995216903, nan), (2.000000142649374, 2.959596952292361, nan)]]

I think this is a true bug (and this might even have been the reason for the old default_value).

[ppurka]

comment:42

Replying to @kcrisman:

That is a good question. You can check it out in the code, there is some stuff in graphics.py or plot.py with DOCTEST_MODE. Computing the plot objects is very fast, you are right. And plot3d of course is even different from that.

I had a look at the files. For 3D plot, the DOCTEST_MODE saves the output in every possible file format - tachyon, java3d and jmol. This is evident from the code in sage/plot/plot3d/base.pyx.

[ppurka]

apply after the earlier patches

[ppurka]

comment:43

Attachment: trac_12798-dont_pass_nan_to_ParametricSurface.patch.gz

@jdemeyer: can you test the new patch I have put up. I prevent nans from being passed on to the ParametricSurface class. Since I based it on 5.1beta5 (don't have 5.2alpha0 at home right now), you may apply the previous two patches (either ours, or yours) and then apply this new patch.

Why I opted for this - modifying ParametricSurface has a high probability of introducing more bugs than fixing the one already present. If we modify it, then one option is to put checks for nan or inf in line 588 (as of the 5.1beta5) of parametric_surface.pyx

                    res.x, res.y, res.z = self.f(uu, vv)

But doing so implies that we also need to modify urange, vrange, n and m in that file, and I believe that this requires a major editing of the code.

It seemed better/safer to actually modify list_plot3d instead of ParametricSurface.

[jdemeyer]

comment:44

That extra patch doesn't seem to have changed anything. You can check for yourself with:

sage: G = list_plot3d([(0,0,1), (2,3,4), (-1,-1,-1)],num_points=100)
sage: G.triangulate()
sage: G.face_list()

There should be no nans in that list.

I'm afraid this will need some more substantial work.

[jdemeyer]

Changed reviewer from Karl-Dieter Crisman, Punarbasu Purkayastha to Karl-Dieter Crisman, Punarbasu Purkayastha, Jeroen Demeyer

[jdemeyer]

comment:45

...although, strangely enough, the doctest timeout is gone on Solaris with that patch. I'm still not happy with this patch because of the nans, but I don't mind if somebody wants to give it positive review.

Somebody still needs to test the patch and check whether the output 3D plots still look like they should.

[kcrisman]

comment:46

Yeah, I have to say that I'd be happier if this change was somehow doctested. Is there any place where we can document that the nans have visibly disappeared? (You know what I mean; I realize that's an oxymoron.) Maybe with num_points=4 or something, to make it easy to see.

[kcrisman]

comment:47

Replying to @kcrisman:

Yeah, I have to say that I'd be happier too if this change was somehow doctested. Is there any place where we can document that the nans have visibly disappeared? (You know what I mean; I realize that's an oxymoron.) Maybe with num_points=4 or something, to make it easy to see.

[jdemeyer]

comment:48

Replying to @kcrisman:

Yeah, I have to say that I'd be happier if this change was somehow doctested. Is there any place where we can document that the nans have visibly disappeared? (You know what I mean; I realize that's an oxymoron.)

In fact, I don't know what you mean with "visibly disappeared".

[kcrisman]

comment:49

Yeah, I have to say that I'd be happier if this change was somehow doctested. Is there any place where we can document that the nans have visibly disappeared? (You know what I mean; I realize that's an oxymoron.)

In fact, I don't know what you mean with "visibly disappeared".

Oh. What I mean is that we should be able to have a doctest that would have once output a list with some nans in it, but now doesn't. So we can see that they disappeared... but I don't know how to do that.

[ppurka]

comment:50

Replying to @jdemeyer:

...although, strangely enough, the doctest timeout is gone on Solaris with that patch. I'm still not happy with this patch because of the nans, but I don't mind if somebody wants to give it positive review.

Somebody still needs to test the patch and check whether the output 3D plots still look like they should.

I had a fresh look at this ticket. The last patch that I posted which trims the xvals and yvals on the basis of whether there are nans or not, is actually incorrect. This can be tested on a simple 3D plot:

list_plot3d([(0,0,1), (1,0,0), (0,1,0)], num_points=5)

The output is a square instead of a triangle and the square persists even for num_points=100. This is because we trim the x and y values. Without the third patch, the earlier patches in this ticket actually gives a correct plot (even for such a low value of num_points=5).

As for the presence of nans. This is all correct and there is no contradiction or error here. If you look at what is going on, there is a matrix being created called vals which contains all possible z values corresponding to x in xvals and y in yvals which should appear in our 3D plot. In the simple example above, xvals = [0, 0.25, 0.5, 0.75, 1] and yvals is the same. So all possible points (x,y) with x and y taking values from xvals and yvals, respectively, is the cartesian product xvals x yvals and it will will include points like (0.75, 0.75) or (0.75, 1), etc, which should have no corresponding finite value in vals. All these values in the matrix are set to nan. Prior to this ticket, they were being set to 0.

So, every time the ranges of x and y do not form a square, there will be nans in the matrix vals, and this can not be avoided if you want an accurate plot. So, it is pointless to try and remove nans.

So now, the question is, why does ParametricSurface take so much time only during the doctests and only on Solaris arch. I don't know the answer, and the code in that class (and in IndexFace, etc) is too complex for me and not very well documented.

At this point, what I suggest is to merge the changes regarding the colors, and leave the list_plot3d alone. If it sounds reasonable, then I will open a new ticket regarding the colors and leave this ticket for the future, when the Solaris arch goes out of fashion!

[jdemeyer]

comment:51

Replying to @ppurka:

So, every time the ranges of x and y do not form a square, there will be nans in the matrix vals, and this can not be avoided if you want an accurate plot. So, it is pointless to try and remove nans.

I completely disagree with this. At the end, you're drawing an arbitrary 3D figure given by faces and vertices. This doesn't have to lie over a square grid. Of course, currently the functions are written to require a square grid, but there is no fundamental reason for this. I agree it's not trivial to change this, but don't blame Solaris.

[vbraun]

comment:52

Nan is the standard to denote missing floating point values and should be used. It is IEEE standard and perfectly portable across all architectures where you can reasonably expect to get an accurate answer for floating point operations. Having said that, the special IEEE floating point values (nan, inf, subnormal) have sometimes been extremely slow on ancient processor architectures. For example, see http://www.sonic.net/~jddarcy/Research/fleckmrk.pdf tests an older ultrasparc and finds that nan and inf are fine, but subnormal can be more than three orders of magnitude slower.

On the plus side, modern processors don't really suffer from these childhood diseases any more and it is generally preferable to use IEEE propagation of nans through floating point operations than riddle your code with if/else and the ensuing pipeline stalls. So if the only problem with this ticket is that SPARC is slow then lets just reduce the number of points (doesn't really add anything new to the doctest anyways) and ship it.

[jdemeyer]

comment:53

Replying to @vbraun:

On the plus side, modern processors don't really suffer from these childhood diseases any more and it is generally preferable to use IEEE propagation of nans through floating point operations than riddle your code with if/else and the ensuing pipeline stalls.

Seriously, you're arguing that rendering 3D figures with NaN coordinates is a good thing? I find it hard to believe that rendering a figure with 2000 faces (half of which aren't drawn due to NaNs) is faster than first removing the offending faces and rendering a figure with 1000 faces.

[vbraun]

comment:54

If most of the points are invalid then you should of course use a different base grid, but I think thats not the scope of this ticket. The way I understand it, this ticket is about a fast rectangular base grid that doesn't fall flat on its face if there is a pole somewhere.

[ppurka]

comment:55

Replying to @jdemeyer:

Replying to @vbraun:
Seriously, you're arguing that rendering 3D figures with NaN coordinates is a good thing? I find it hard to believe that rendering a figure with 2000 faces (half of which aren't drawn due to NaNs) is faster than first removing the offending faces and rendering a figure with 1000 faces.

I haven't looked at if it is Sage which is doing the rendering. But if Sage is trying to actually plot/render the points - the nans are specific 3D points (x,y,nan), not entire faces - then it is doing something very inefficient. What I would expect any reasonable plotting routine to do is simply ignore all the points which are not finite real numbers while rendering the plot. This is no different than the procedure of looping through the points and discarding them before providing them to the same plotting routine. Maybe you can have special algorithms for quickly discarding "faces" or larger collections of adjacent points, while generating a set of interpolated points. But this is useful if we control all aspects of 3D plotting in Sage itself.

Note that we are not generating the grid - the grid is being provided by mpl after interpolation. Anyway, the slowdown is architecture specific (there's speedups on intel, as I mentioned before) and triggered only on doctest mode; and if you would like it to stay that way then so be it. Only reasonable way to fix list_plot3d if you don't want to allow nans is to rewrite half of the 3D plotting methods (ParametricSurface), but I don't have the time or inclination to do this now.

[ppurka]

comment:56

@jdemeyer The doctest system has changed since this ticket was opened. So can you apply trac_12798-fix_extraneous_points_and_kwds_in_list_plot3d.patch again and see if it still causes timeouts on solaris?