WISH: support for long vectors in writeBin() and write()

[wlandau]

R >= 3.0.0 supports vectors with over 2^31 - 1 bytes, but writeBin() and write() do not yet.

writeBin(raw(2^31), tempfile())
#> Error in writeBin(raw(2^31), tempfile()): long vectors not supported yet: connections.c:4418

^{Created on 2019-06-15 by the reprex package (v0.3.0)}

[HenrikBengtsson]

R-devel commit r77522 and particularly it's commit message:

Fix PR#17665: read/writeBin(rawConnection())')

caused be to revisit this feature request. Unfortunately, it's still the case in R devel (2019-12-06 r77536; to become R 4.0.0) that;

x <- raw(2^31)
writeBin(x, con = nullfile())
# Error in writeBin(x, con = nullfile()) : 
#  long vectors not supported yet: connections.c:4430

We get the same if we use a raw connection;

x <- raw(2^31)
con <- rawConnection(raw(0L), "w")
writeBin(raw(2^31), con = con)
# Error in writeBin(raw(2^31), con = con) : 
#  long vectors not supported yet: connections.c:4430

Maybe it's worth checking in with R core to see if r77522 was meant to fix that second case.

[wlandau]

Good idea. I just sent a note to r-devel@r-project.org.

[mmaechler]

For a next time : If you had gone reading the above PR#17665 link (and/or the changeset for commit r77536 which includes a regression test), you'd seen what bug was fixed.
... and as I just replied to your R-devel note, this case may be relatively easy to get as well.

[HenrikBengtsson]

Thank you @mmaechler for fixing this in R-devel r77541. It now works:

$ R --vanilla
R Under development (unstable) (2019-12-07 r77541) -- "Unsuffered Consequences"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
...

> x <- raw(2^31)
> writeBin(x, con = nullfile())
> con <- rawConnection(raw(0L), "w")
> writeBin(raw(2^31), con = con)
> y <- rawConnectionValue(con)
> stopifnot(identical(y, x))

PS. One purpose of this issue tracker/wish list is to server as a gatekeeper to decrease the noise on R devel and BugZilla and to lower the workload on R Core. Here, I got literally only a few spare minutes to check back in on this issue and shared that this was still a problem. I should probably have skipped that vague "checking in with R core" suggestion.

[wlandau]

Yes, thank you @mmaechler. This is an exciting enhancement. I will reference the proper svn revision numbers going forward.

[mmaechler]

Thank you, too @wlandau , after all your R-devel@r-project.org message triggered it. BTW, this is not extremely well tested, just basically a few simple / regression tests. If you have real use cases, go ahead running them .. and possibly find if there are more holes to be plugged... C arithmetic with silent integer overflow is a dangerous feature here..