Stricter YAML parsing in Pandoc >= 2.2.2 generates Pandoc WARNINGs

[jennybc]

rmarkdown::pandoc_version()
#> [1] '2.3.1'

^{Created on 2018-11-04 by the reprex package (v0.2.1)}

Session info

devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  os       macOS  10.14.1              
#>  system   x86_64, darwin15.6.0        
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_CA.UTF-8                 
#>  ctype    en_CA.UTF-8                 
#>  tz       America/Vancouver           
#>  date     2018-11-04                  
#> 
#> ─ Packages ──────────────────────────────────────────────────────────────
#>  package     * version    date       lib
#>  assertthat    0.2.0      2017-04-11 [2]
#>  backports     1.1.2      2017-12-13 [2]
#>  base64enc     0.1-3      2015-07-28 [2]
#>  callr         3.0.0      2018-08-24 [1]
#>  cli           1.0.1      2018-09-25 [1]
#>  crayon        1.3.4      2017-09-16 [2]
#>  debugme       1.1.0      2017-10-22 [1]
#>  desc          1.2.0      2018-05-01 [2]
#>  devtools      2.0.1      2018-10-26 [1]
#>  digest        0.6.18     2018-10-10 [1]
#>  evaluate      0.12       2018-10-09 [2]
#>  fs            1.2.6      2018-08-23 [1]
#>  glue          1.3.0      2018-07-17 [1]
#>  htmltools     0.3.6      2017-04-28 [2]
#>  knitr         1.20       2018-02-20 [2]
#>  magrittr      1.5        2014-11-22 [2]
#>  memoise       1.1.0      2017-04-21 [2]
#>  pkgbuild      1.0.2      2018-10-16 [1]
#>  pkgload       1.0.2      2018-10-29 [1]
#>  prettyunits   1.0.2      2015-07-13 [1]
#>  processx      3.2.0      2018-08-16 [1]
#>  ps            1.2.0      2018-10-16 [1]
#>  R6            2.3.0      2018-10-04 [1]
#>  Rcpp          0.12.19    2018-10-01 [1]
#>  remotes       2.0.2      2018-10-30 [1]
#>  rlang         0.3.0.9000 2018-11-01 [1]
#>  rmarkdown     1.10       2018-06-11 [1]
#>  rprojroot     1.3-2      2018-01-03 [2]
#>  sessioninfo   1.1.0.9001 2018-10-26 [1]
#>  stringi       1.2.4      2018-07-20 [1]
#>  stringr       1.3.1      2018-05-10 [2]
#>  testthat      2.0.1      2018-10-13 [1]
#>  usethis       1.4.0.9000 2018-10-27 [1]
#>  withr         2.1.2      2018-03-15 [2]
#>  yaml          2.2.0      2018-07-25 [2]
#>  source                            
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.1)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.1)                    
#>  Github (r-lib/rlang@eb17969)      
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  Github (r-lib/sessioninfo@5b02efa)
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#>  local                             
#>  CRAN (R 3.5.0)                    
#>  CRAN (R 3.5.0)                    
#> 
#> [1] /Users/jenny/resources/R/library
#> [2] /Library/Frameworks/R.framework/Versions/3.5/Resources/library

---

As of version 2.2.2, Pandoc parses YAML with HsYAML, which has a stricter interpretation of YAML 1.2 than the previous approach.

https://github.com/jgm/pandoc/blob/b8573aab292cc7c75b9a7268602799e7abda5f81/changelog#L953-L959

pandoc (2.2.2)

  * Use HsYAML instead of yaml for translations and YAML metadata (#4747).
    yaml wraps a C library; HsYAML is pure Haskell.  Advances #4535.

    Note: HsYAML implements YAML 1.2, in which the valid true
    values are `true`, `True`, `TRUE`.  This means a change in
    the semantics of YAML metadata that could affect users:
    `y`, `yes`, and `on` no longer count as true values.

It's pretty common to see yes used as a true value in .Rmd documents, although perhaps it is always in the output key, which is parsed by Pandoc but not actually used. That YAML is used by rmarkdown itself. It still might be a good idea to shift from yes to TRUE in YAML we produce, in the long run.

Here is the one semi-breakage I see: indentation in a multi-line YAML collection.

Take the YAML from 3.1.10.5 Pandoc arguments in R Markdown: The Definitive Guide

---
title: "Habits"
output:
  html_document:
    pandoc_args: [
      "--title-prefix", "Foo",
      "--id-prefix", "Bar"
    ]
---

If I use rmarkdown::render() or the IDE's Knit button on this, I see these WARNINGs, but the render still works:

[WARNING] Could not parse YAML metadata at line 13 column 1: :7:26: Unexpected '
  '
[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'foo.utf8'

Output created: foo.html

Changing the indentation clears the WARNING:

---
title: "Habits"
output:
  html_document:
    pandoc_args: [
                   "--title-prefix", "Foo",
                   "--id-prefix", "Bar"
                 ]
---

[kevinushey]

I wonder if we should adopt the convention that --prefixed lists are used by default (at least, when a sequence spans multiple lines)? It's certainly loses some of the aesthetics, though...

---
title: "Habits"
output:
  html_document:
    pandoc_args:
      - "--title-prefix=Foo"
      - "--id-prefix=Bar"
---

[cderv]

I don't see the warning anymore with last Pandoc (2.11.0.4) when trying

---
title: "Habits"
output:
  html_document:
    pandoc_args: [
      "--title-prefix", "Foo",
      "--id-prefix", "Bar"
    ]
---

However, this issue is real as Pandoc uses YAML 1.2 but R package yaml uses 1.1.
However, Pandoc may revert back from HsYAML to C yaml based on libyaml: See jgm/pandoc#4535 (comment)

LibYAML support for 1.2 may be coming: yaml/libyaml#20
but that would mean a new R package : vubiostat/r-yaml#76 (comment)

YAML version is something we need to watch out while improving our YAML support in Rmarkdown, and RStudio IDE.

[cderv]

Several updates on this:

• Pandoc has reverted the use of HsYAML in 2.16 https://github.com/jgm/pandoc/blob/master/changelog.md#pandoc-216-2021-10-31 to yaml lib that uses 1.1 version.
• 'Y', 'N', 'Yes', 'No', 'On', 'Off' and others are now parsed again. It needs to be quoted if this is supposed to be a string. Also, 'null' is parsed as a YAML null value

It seems there is not yet issue reported with this change. However, Pandoc 2.16 is not yet in the RStudio release version, we could see more later.

Also, work is on-going for YAML validation in the IDE. This will probably help and solved such issue by helping with formatting and completion.