Spurious warning messages with mgcv and quantreg

[jhmaindonald]

The following runs without problem from the command line:

library(sp)
library(mgcv)
data(meuse)
meuse.gam <- gam(log(lead)~s(elev)+s(dist)+ffreq, data=meuse)

However, when I insert this code into an Sweave file and process
the file using knit(); the following appears in the .tex file that is
generated:

\begin{knitrout}
\definecolor{shadecolor}{rgb}{0.969, 0.969, 0.969}\color{fgcolor}\begin{kframe}
\begin{alltt}
\hlkwd{library}\hlstd{(sp)}
\hlkwd{library}\hlstd{(mgcv)}
\hlkwd{data}\hlstd{(meuse)}
\hlstd{meuse.gam} \hlkwb{<-} \hlkwd{gam}\hlstd{(}\hlkwd{log}\hlstd{(lead)} \hlopt{~} \hlkwd{s}\hlstd{(elev)} \hlopt{+} \hlkwd{s}\hlstd{(dist)} \hlopt{+} \hlstd{ffreq,} \hlkwc{data} \hlstd{= meuse)}
\end{alltt}


{\ttfamily\noindent\color{warningcolor}{\#\# Warning: the matrix is either rank-deficient or indefinite\\\#\# Warning: the matrix is either rank-deficient or indefinite\\\#\# Warning: the matrix is either rank-deficient or indefinite\\\#\# Warning: the matrix is either rank-deficient or indefinite}}\end{kframe}
\end{knitrout}

R version 3.0.1 (2013-05-16)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

locale:
[1] en_AU.UTF-8/en_AU.UTF-8/en_AU.UTF-8/C/en_AU.UTF-8/en_AU.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mgcv_1.7-26  nlme_3.1-111 sp_1.0-13    knitr_1.4.11

loaded via a namespace (and not attached):
[1] evaluate_0.4.7  formatR_0.9     grid_3.0.1      highr_0.2.1     lattice_0.20-23
[6] Matrix_1.0-14   stringr_0.6.2   tools_3.0.1    

A student has reproduced this on a Windows machine.

Such a warning has appeared for any use of the function gam() that I have tried.
I get similar warning messages when using rq() from the quantreg package.

[yihui]

The fact is these warnings indeed happened, but were suppressed by an imperfect approach, namely options(warn = -1).

It happened that someone reported the same problem of the gtools package yesterday (and it was fixed in gtools); here is my code to illustrate the root problem: https://gist.github.com/yihui/6656584

In mgcv, there were two instances of options(warn=-1):

$ cd mgcv
$ grep 'options(warn=-1)' -r .
./R/mgcv.r:  { op <- options(warn=-1) ## don't want to be warned it's not +ve def
./R/plots.r:  { old.warn<-options(warn=-1)

I strongly recommend the author to use suppressWarnings() instead, otherwise you will have to turn off the warnings in knitr using the chunk option warning=FALSE.

[jhmaindonald]

Have you then drawn this to the attention of the authors of the mgcv and quantreg packages?

On 24/09/2013, at 4:28 PM, Yihui Xie notifications@github.com wrote:

The fact is these warnings indeed happened, but were suppressed by an imperfect approach, namely options(warn = -1).

It happened that someone reported the same problem of the gtools package yesterday (and it was fixed in gtools); here is my code to illustrate the root problem: https://gist.github.com/yihui/6656584

In mgcv, there were two instances of options(warn=-1):

$ cd mgcv
$ grep 'options(warn=-1)' -r .
./R/mgcv.r: { op <- options(warn=-1) ## don't want to be warned it's not +ve def
./R/plots.r: { old.warn<-options(warn=-1)
I strongly recommend the author to use suppressWarnings() instead, otherwise you will have to turn off the warnings in knitr using the chunk option warning=FALSE.

—
Reply to this email directly or view it on GitHub.

[yihui]

No, perhaps you can do that?

[jhmaindonald]

Actually, I think it should be raised on r-devel. The issue will affect other packages also.
But I want first to be clear just what the issues are.

The gam() calculations have an internal process by which singularities that arise from an
anticipated over-parameterization are detected and removed. Or to put it another way,
there are redundant basis functions. Use of 'option(warn=-1)' was intended to hide from
the user the warning that is generated in these circumstances -- it serves no useful
purpose and is an annoyance. Because of the special way that knitr handles warnings,
'option(warn=-1)' does not quite do the job when code is processed through knit(). It is
not clear to me why this happens.

The function mroot() has the code fragment:

op <- options(warn=-1) ## don't want to be warned it's not +ve def
L <- chol(A,pivot=TRUE)
options(op) ## reset default warnings

This is a case where the matrix A is non-negative definite by construction. The better
way to handle the issue would, in my view, be to allow an argument to 'chol()' that
turns the warning off. This would address the specific expected cause of the warning,
rather than using a wholesale turning off of all warnings.

vis.gam(), in the file plots.r in the mgcv source, also turns warnings off briefly. The
reason may to allow the specification of a matrix in which the data length "is not a
sub-multiple or multiple of the number of rows".

[yihui]

The reason that knitr can capture the warning even when options(warn = -1) is that knitr evaluates code in withCallingHandlers(), which always capture warnings unless you do invokeRestart("muffleWarning") that kills warnings from the root level. That is also why I recommend suppressWarnings():

> suppressWarnings
function (expr) 
{
    ops <- options(warn = -1)
    on.exit(options(ops))
    withCallingHandlers(expr, warning = function(w) invokeRestart("muffleWarning"))
}

On the other hand, suppressWarnings() also looks more natural and self-explanatory when you want to suppress warnings, whereas for options(warn = -1), you will have to remember what this option warn means.

It may worth a discussion in r-devel whether withCallingHandlers() should respect options(warn = -1). Since it is actually fairly easy for me to do it in the evaluate package, I just did that, and will push a new version to Github later.

[jhmaindonald]

If I were to raise this on r-devel, I'd want to say that I have been discussing
this with you on github.

Maybe however it would come better from you, because there is an issue of
what knitr ought to do. From where I stand, modifying the evaluate package to
respect options(warn=-1) is something that is anyway desirable, at least until
such time as direct use of that option is deprecated or strongly discouraged.
Whether that ought to happen is a somewhat separate issue, and might be
raised as such on r-devel.

What I do not want to do is to set 'warning=FALSE' whenever I execute a code
chunk that calls gam()! There may genuine warnings that I ought to heed.
That is, I judge, now resolved.

[yihui]

I just made the changes in evaluate, and you can test the devel version here:

install.packages('evaluate',,'http://www.rforge.net/')

I guess I'm not going to bother r-devel at the moment. It is not anything urgent.

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.