[REVIEW]: BridgeStan: Efficient in-memory access to the methods of a Stan model

[editorialbot]

Submitting author: @roualdes (Edward Roualdes)
Repository: https://github.com/roualdes/bridgestan
Branch with paper.md (empty if default branch): joss
Version: v2.1.2
Editor: @Nikoleta-v3
Reviewers: @salleuska, @saumil-sh
Archive: 10.5281/zenodo.8169248

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/8ecf92ae523d5719bcc2b63ec254945d"><img src="https://joss.theoj.org/papers/8ecf92ae523d5719bcc2b63ec254945d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/8ecf92ae523d5719bcc2b63ec254945d/status.svg)](https://joss.theoj.org/papers/8ecf92ae523d5719bcc2b63ec254945d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@adity-om & @salleuska, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @Nikoleta-v3 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @salleuska

📝 Checklist for @saumil-sh

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.21 s (368.8 files/s, 39910.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Julia                            8            230             19           1071
Python                          10            254            274           1048
JSON                             7              0              0           1037
Markdown                        12            545              0            885
C++                              3             44             19            444
YAML                             5             51              6            402
reStructuredText                 9            175             83            263
R                                4             24             99            215
TOML                             7             44              3            150
C/C++ Header                     4             74            352            135
make                             3             32             22            135
TeX                              1             15              4             88
DOS Batch                        1              8              1             26
C                                2              6              9             23
CSS                              1              3              0             15
-------------------------------------------------------------------------------
SUM:                            77           1505            891           5937
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1174

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None

[Nikoleta-v3]

👋🏼 @roualdes, @adity-om, @salleuska this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements ✅ As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention #5236 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@Nikoleta-v3) if you have any questions/concerns. 😄 🙋🏻

[salleuska]

Review checklist for @salleuska

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/roualdes/bridgestan?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@roualdes) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[Nikoleta-v3]

👋🏻 @adity-om any update on your review? 😄

[roualdes]

@Nikoleta-v3 just a friendly follow up. Anything you know that we don't already know from this thread? Thanks :)

[Nikoleta-v3]

Thank you for the nudge @roualdes and apologises for the delay in my reply. Unfortunately, I don't but I am going to reach out to the reviewers.

[Nikoleta-v3]

@salleuska did you get to complete your review?

[Nikoleta-v3]

👋🏻 @adity-om any update on your review?

[salleuska]

@Nikoleta-v3 and @roualdes I am sorry for the late reply. I had to travel in the past weeks and have unexpectedly limited access to the internet. I still have to check out some things about the software functionalities/docs, but I have some comments about the paper and a couple of issues.

Paper

I think that the writing of the paper could be improved and that there some things that are worth clarifying. I think that the package provides a great tool for Stan users to take advantage of internal Stan methods, allowing users to develop their own algorithms. However, it seems to me that some parts of the paper are more focused on marketing the product rather than presenting the methods the bridgestan package allows access.

Major Comments

1. I would suggest revising the text in "Statement of need" and clarify which methods of Stan models are usually not accessible via the standard Stan interfaces that are instead made available by bridgestan. I also think it would be useful to mention what are potential user cases or if there are already relevant developed algorithms that take advantage of the package.

2. From the text it is not clear to me what is the state of the art, for example:

• line 43. "Using a memory-compatible C interface makes this possible even if the host language (e.g., R) was compiled with a different compiler, something no prior interface which exposed Stan’s log density calculations could allow."
  -> Is there previous work that allows to access to Stan's models methods? If so could you cite that? Or give an idea of the shortcomings of these interfaces?

• line 44. "Existing tools with similar functionalities include JAX (Bradbury et al., 2018) and Turing.jl via the JuliaAD ecosystem " -> What do you mean with similar functionalities? From the sentence it is unclear whether these tools offer access to methods in Stan or instead provide functions for derivatives and calculation of log-densities?

Minor

• line 27: "we find that rstan ranks at number 70, rstantools 179, and rstanarm 502." it is not obvious to a reader what these packages are,

• line 39. "BridgeStan offers efficient, in-memory, from memory allocated within the host language, computations of the log joint density function of a Stan model, itself implemented using highly templated C++ from the Stan math library" -> this sentence is hard to follow; I think it would helpful to elaborate a bit, and possible merge these concepts related to memory in the paragraph at line 53.

• line 47: "The Stan community by and large uses CPU hardware" - this is unclear; maybe "by" is a "buy"?

• There is an abuse of the term "efficient" that I find distracting and I would encourage the authors to remove it when unnecessary.

[salleuska]

Code

For now, I found one error in the R example.

roualdes/bridgestan#121 (comment)

I will check julia and python early next week.

[roualdes]

Thanks so much for the helpful feedback.

@Nikoleta-v3 what is the expected process here? Do we wait until all feedback is in before updating our manuscript? Or do we push updates as the feedback rolls in, and the reviewers are responsible for pulling the latest changes?

[Nikoleta-v3]

Hey @roualdes you can push updates as the feedback rolls in. I do recommend having separate commits for each of the reviewers’ comments (obviously the minor comments you can address in one commit). You can then reply to the comments in this issue and provide the commit hash (in my experience this helps the reviewers’ a lot).

Some people also prefer opening pull requests in the case of major comments.

Now regarding the second reviewer, we have been in touch. Hopefully they will complete their review in the upcoming weeks.

Thank you again for your patience and submitting to JOSS 🙏🏻

[roualdes]

Thanks, @salleuska, for your suggestions. I think your comments have really helped.
Here's a quick record of the changes we've made, based on your feedback so far.

Major Comments

1. Explain user cases, see commit 5d63b03
2. Add a new paragraph about similar offerings from other software, see commit e2cc9ba and commit d8473a9

Minor Comments

• Specify more clearly what those R packages are all about, see commit 2932c77
• clean up sentence about what BridgeStan offers, see commit 9c687e4
• replace phrase by and large, see commit f2589c7
• remove most uses of the word efficient, see commit 06c5513

[WardBrian]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[salleuska]

Many thanks @roualdes! I did play around with Julia and Python and haven't had any issues to report. Also the documentation seems in pretty good shape to me. I'll take a look at the changes and keep you posted!

[Nikoleta-v3]

@editorialbot remove @adity-omfrom reviewers

[editorialbot]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

[saumil-sh]

I approve 👍 this report 📄 and the package 📦 for publication!

[roualdes]

@Nikoleta-v3 please let us know if there is anything else you need from the authors or the reviewers. Thanks.

[Nikoleta-v3]

Thank you all for your work 🙏🏻 I will take one last look at the paper and the package in the next few days, and then we can proceed with the publication. 👍🏻

[Nikoleta-v3]

Hey @roualdes thank you for your patience! I have not further comments. Just two minor typos in the manuscript:

• Line 8: to

• Line 35: I believe it should be aids

At this point could you please:

• (if necessary) Make a new tagged release of your software, and list the version tag of the archived version here. Or let me know if I should just use the release 1.0.0
• Archive the reviewed software in Zenodo or a similar service (e.g., figshare, an institutional repository)
  Check the archival deposit (e.g., in Zenodo) has the correct metadata. This includes the title (should match the paper title) and author list (make sure the list is correct and people who only made a small fix are not on it). You may also add the authors' ORCID.
• Please list the DOI of the archived version here.

I can then move forward with accepting the submission.

[roualdes]

Tagged v2.1.2: https://github.com/roualdes/bridgestan/releases/tag/v2.1.2
Zenodo up to date: https://zenodo.org/record/8169248

[WardBrian]

@editorialbot generate pdf

We made some edits including your above suggestions and ensuring our zenodo and authors list match up with recent contributions as well

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Nikoleta-v3]

@editorialbot set v2.1.2 as version

[editorialbot]

Done! version is now v2.1.2

[Nikoleta-v3]

@editorialbot set 10.5281/zenodo.8169248 as archive

[editorialbot]

Done! archive is now 10.5281/zenodo.8169248

[Nikoleta-v3]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[Nikoleta-v3]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

👋 @openjournals/dsais-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#4423, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[gkthiruvathukal]

All appears to be in good shape here. I'm moving to accept. Now!

[gkthiruvathukal]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Roualdes
  given-names: Edward A.
  orcid: "https://orcid.org/0000-0002-8757-3463"
- family-names: Ward
  given-names: Brian
  orcid: "https://orcid.org/0000-0002-9841-3342"
- family-names: Carpenter
  given-names: Bob
  orcid: "https://orcid.org/0000-0002-2433-9688"
- family-names: Seyboldt
  given-names: Adrian
  orcid: "https://orcid.org/0000-0002-4239-4541"
- family-names: Axen
  given-names: Seth D.
  orcid: "https://orcid.org/0000-0003-3933-8247"
contact:
- family-names: Roualdes
  given-names: Edward A.
  orcid: "https://orcid.org/0000-0002-8757-3463"
doi: 10.5281/zenodo.8169248
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Roualdes
    given-names: Edward A.
    orcid: "https://orcid.org/0000-0002-8757-3463"
  - family-names: Ward
    given-names: Brian
    orcid: "https://orcid.org/0000-0002-9841-3342"
  - family-names: Carpenter
    given-names: Bob
    orcid: "https://orcid.org/0000-0002-2433-9688"
  - family-names: Seyboldt
    given-names: Adrian
    orcid: "https://orcid.org/0000-0002-4239-4541"
  - family-names: Axen
    given-names: Seth D.
    orcid: "https://orcid.org/0000-0003-3933-8247"
  date-published: 2023-07-22
  doi: 10.21105/joss.05236
  issn: 2475-9066
  issue: 87
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 5236
  title: "BridgeStan: Efficient in-memory access to the methods of a
    Stan model"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.05236"
  volume: 8
title: "`BridgeStan`: Efficient in-memory access to the methods of a
  Stan model"

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

[editorialbot]

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.05236 joss-papers#4425
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.05236
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.05236/status.svg)](https://doi.org/10.21105/joss.05236)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.05236">
  <img src="https://joss.theoj.org/papers/10.21105/joss.05236/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.05236/status.svg
   :target: https://doi.org/10.21105/joss.05236

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss