stanc_options = list(“O1”) not being applied?

[tomshafer]

Describe the bug
I have tried to follow this Discourse post demonstrating stanc’s --O1 flag in cmdstanr, but it seems like the flag is not being passed to stanc.

To Reproduce

lr.stan

data {
  int<lower=1> k;
  int<lower=0> n;
  matrix[n, k] X;
  array[n] int y;
}
parameters {
  vector[k] beta;
  real alpha;
}
model {
  target += std_normal_lpdf(beta | );
  target += std_normal_lpdf(alpha | );
  target += bernoulli_logit_lpmf(y | alpha + X* beta);
}

options("cmdstanr_verbose" = TRUE)

# Simulated data
set.seed(1)
n <- 25000
k <- 10
X <- matrix(rnorm(n * k), ncol = k)
y <- rbinom(n, size = 1, prob = plogis(3 * X[,1] - 2 * X[,2] + 1))
mdata <- list(k = k, n = n, y = y, X = X)

# Non-optimized model
mod_no_opt <- cmdstan_model("lr.stan", exe_file = "no_opt_lr", force_recompile = TRUE)
fit_no_opt <- mod_no_opt$sample(data = mdata, seed = 1, chains = 1, refresh = 500)

# Optimized model
mod_opt <- cmdstan_model("lr.stan", exe_file = "opt_lr", stanc_options = list("O1"), force_recompile = TRUE)
fit_opt <- mod_opt$sample(data = mdata, seed = 1, chains = 1, refresh = 500)

print(fit_no_opt$time()$total)
print(fit_opt$time()$total)

Expected behavior

• fit_opt is faster
• compilation of mod_opt shows --O1 is applied to stanc

Instead, the models run in about equal time and the stanc call doesn’t have O1 applied.

Operating system
macOS 13.4.1

CmdStanR version number
0.5.3

Additional context
CmdStan version 2.32.2

[jgabry]

Hmm, I'm not sure what's going on because when I run this I see a pretty big speed difference:

> print(fit_no_opt$time()$total)
[1] 15.51
> print(fit_opt$time()$total)
[1] 8.13984

@rok-cesnovar Any idea why this wouldn't work for @tomshafer but seems to work fine for me?

[tomshafer]

That’s so interesting—exactly what I was hoping to see, but I’ve tried a few different things since, without success (I’m on a 2.6 GHz i7, which should be OK-ish).

1. Custom make/local

   > print(fit_no_opt$time()$total)
   [1] 16.50736
   > print(fit_opt$time()$total)
   [1] 16.69734
   

2. Empty make/local

   > print(fit_no_opt$time()$total)
   [1] 16.06238
   > print(fit_opt$time()$total)
   [1] 16.03752
   

3. Empty ~/.R/Makevars

   > print(fit_no_opt$time()$total)
   [1] 19.68105
   > print(fit_opt$time()$total)
   [1] 17.33334
   

[rok-cesnovar]

This is really strange! I get the same thing as Jonah, also on a Mac. Are you using the most-recent Github version of cmdstanr? If not, can you try that?

Will post some additional debugging ideas in a bit.

[tomshafer]

OK, I think this was a problem with my CmdStan install. Instead of rebuild_cmdstan(), this time I ran install_cmdstan(overwrite = TRUE, ...) and got a much better result. No idea why, but here’s the deal for posterity. Thanks for trying this out on your end and sorry not to know what caused the issue!

remotes::install_github("stan-dev/cmdstanr")

# Restart session
library(cmdstanr)
options("cmdstanr_verbose" = TRUE)

# Custom make/local
install_cmdstan(overwrite = TRUE, cpp_options = list(
  "CXX = /usr/local/opt/llvm/bin/clang++",
  "STAN_THREADS = true",
  "STAN_NO_RANGE_CHECKS = true",
  "PRECOMPILED_HEADERS = true",
  "CXXFLAGS_OPTIM = -O3 -mtune=native -march=native",
  "CXXFLAGS_OPTIM_TBB = -O3 -mtune=native -march=native",
  "CXXFLAGS_OPTIM_SUNDIALS = -O3 -mtune=native -march=native",
  "CXXFLAGS += -DEIGEN_USE_BLAS -Wno-deprecated-declarations -Wno-deprecated-builtins",
  "LDLIBS += -framework Accelerate"
))

# Restart
library(cmdstanr)
options("cmdstanr_verbose" = TRUE)

# Simulated data
set.seed(1)
n <- 25000
k <- 10
X <- matrix(rnorm(n * k), ncol = k)
y <- rbinom(n, size = 1, prob = plogis(3 * X[,1] - 2 * X[,2] + 1))
mdata <- list(k = k, n = n, y = y, X = X)

# Non-optimized model
mod_no_opt <- cmdstan_model("lr.stan", exe_file = "no_opt_lr", force_recompile = TRUE)
fit_no_opt <- mod_no_opt$sample(data = mdata, seed = 1, chains = 1, refresh = 500)

# Optimized model
mod_opt <- cmdstan_model("lr.stan", exe_file = "opt_lr", stanc_options = list("O1"), force_recompile = TRUE)
fit_opt <- mod_opt$sample(data = mdata, seed = 1, chains = 1, refresh = 500)

> print(fit_no_opt$time()$total)
[1] 16.86377
> print(fit_opt$time()$total)
[1] 7.907304

[rok-cesnovar]

Awesome to hear that it's working now! I am not sure I have any ideas as to what happened.

Will close the issue for now, but don't hesitate to post if the issue comes back. Hoping it doesn't though :)

[tomshafer]

OK, here’s the issue!

Installing from GitHub with remotes passes STANCFLAGS and is fast:

> remotes::install_github("stan-dev/cmdstanr”)
Using github PAT from envvar GITHUB_PAT
Downloading GitHub repo stan-dev/cmdstanr@HEAD
...

> mod_opt <- cmdstan_model("lr.stan", exe_file = "opt_lr", stanc_options = list("O1"), force_recompile = TRUE)
Compiling Stan program...
Running make /var/folders/jc/97flnncx6rl3rq_xv_31xwd1d808dq/T/RtmpaHbPnq/model-e6492c0b5660 \
  "STANCFLAGS +=  --O1  --name='lr_model’”

...

> print(fit_no_opt$time()$total)
[1] 20.05996
> print(fit_opt$time()$total)
[1] 9.156741

Installing with install.packages does not do that, and is slow.

> install.packages("cmdstanr", repos = c("https://mc-stan.org/r-packages/", getOption("repos")))
Installing package into ‘/Users/tom.shafer/Library/R/4.3’
(as ‘lib’ is unspecified)
Warning in install.packages :
  unable to access index for repository https://mc-stan.org/r-packages/bin/macosx/big-sur-x86_64/contrib/4.3:
  cannot open URL 'https://mc-stan.org/r-packages/bin/macosx/big-sur-x86_64/contrib/4.3/PACKAGES'
installing the source package ‘cmdstanr’
...

> mod_opt <- cmdstan_model("lr.stan", exe_file = "opt_lr", stanc_options = list("O1"), force_recompile = TRUE)
Compiling Stan program...
Running make /var/folders/jc/97flnncx6rl3rq_xv_31xwd1d808dq/T/Rtmp64tC66/model-e87d2149cfc5
|

...

> print(fit_no_opt$time()$total)
[1] 16.07749
> print(fit_opt$time()$total)
[1] 16.30384

Does the release not support --O1 and I just missed it, maybe?

[rok-cesnovar]

Great, really appreciate the follow-up! We are working on getting the Github version released ASAP, so this should be resolved in the next few days.