Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update feather serializer; Add parquet serializer #849

Merged
merged 27 commits into from
Mar 24, 2022

Conversation

pachadotdev
Copy link
Contributor

Pull Request

Before you submit a pull request, please do the following:

  • Add an entry to NEWS concisely describing what you changed. OK

  • Add unit tests in the tests/testthat directory. OK

  • Run Build->Check Package in the RStudio IDE, or devtools::check(), to make sure your change did not add any messages, warnings, or errors. OK

The relevant tests were added to these files:

══ Testing test-parse-body.R ═══════════════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 169 ] Done!

══ Testing test-serializer-feather.R ═══════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 6 ] Done!

══ Testing test-serializer-parquet.R ═══════════════════════════════════════════
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 6 ] Done!

In addition, I separated the PR in different pushes (i.e. 1 for content + 1 for NEWS + 1 for tests)

Minimal reproducible example

I created a mini API to test both from browser and in R with download,file

MIni API

library(plumber)

#* Mtcars JSON
#* @serializer json
#* @get /mtcarsjson
function(){
  mtcars
}

# this is thx to good old PR #520

#* Mtcars JSON
#* @serializer csv
#* @get /mtcarscsv
function(){
  mtcars
}

#* Mtcars JSON
#* @serializer feather
#* @get /mtcarsfeather
function(){
  mtcars
}

#* Mtcars JSON
#* @serializer parquet
#* @get /mtcarsparquet
function(){
  mtcars
}

I run this mini API from RStudio with the command plumber::plumb("plumber.R")$run(port = 1234)

Then I run these reproducible tests in a new RStudio window

download.file(
  "http://127.0.0.1:1234/mtcarscsv",
  "mtcars.csv"
)

download.file(
  "http://127.0.0.1:1234/mtcarsfeather",
  "mtcars.feather"
)

download.file(
  "http://127.0.0.1:1234/mtcarsparquet",
  "mtcars.parquet"
)

readr::read_csv("mtcars.csv")
arrow::read_feather("mtcars.feather")
arrow::read_parquet("mtcars.parquet")

All good!

PR task list:

  • Update NEWS
  • Add tests
  • Update documentation with devtools::document()

In addition, I added myself as ctb, after adding #520, #398 and #360.

@pachadotdev
Copy link
Contributor Author

this can be interesting in terms of performance for mid-sized tables
the current PR can reduce the time to move data from one the server running plumber to the laptop by ~50%
ropensci/tradestatistics#49

R/content-types.R Outdated Show resolved Hide resolved
R/serializer.R Outdated Show resolved Hide resolved
R/serializer.R Outdated Show resolved Hide resolved
R/content-types.R Outdated Show resolved Hide resolved
R/content-types.R Outdated Show resolved Hide resolved
R/parse-body.R Outdated Show resolved Hide resolved
tests/testthat/test-serializer-parquet.R Outdated Show resolved Hide resolved
tests/testthat/test-serializer-parquet.R Outdated Show resolved Hide resolved
tests/testthat/test-serializer-feather.R Show resolved Hide resolved
tests/testthat/test-serializer-feather.R Show resolved Hide resolved
tests/testthat/test-parse-body.R Outdated Show resolved Hide resolved
pachadotdev and others added 15 commits January 18, 2022 14:57
Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
Co-authored-by: Barret Schloerke <barret@rstudio.com>
@pachadotdev
Copy link
Contributor Author

@schloerke all ready now !

@pachadotdev
Copy link
Contributor Author

@schloerke all ready now !

hi @schloerke !
any news on this?

@schloerke schloerke changed the title update feather serializers + add parquet serializer Update feather serializer; Add parquet serializer Mar 24, 2022
@schloerke schloerke merged commit 0c21b78 into rstudio:main Mar 24, 2022
schloerke added a commit that referenced this pull request Jun 2, 2022
* main:
  Fix Content-Length header on HTTP responses which forbid it (#760)
  Generate docs and fix existing tests (#860)
  decode static URI before serving (#754)
  Update feather serializer; Add parquet serializer (#849)
  remove author in pkgdown; have website build again
  Fix plumber website. Use tidytemplate (#851)
  Runtime Vignette typo (#848)
  GHA v2 (#842)
  master -> main (#839)
  Create GeoJSON serializer and parser (#830)
  Add details about named functions as router modifiers (#824)
  Parser params should be in a `list()` (#827)
  Test on ubuntu 18/20 and use relative R versions (#828)
  Shared secret rejections are not errors (#809)
  Fix tag endpoint block annotation spelling to `@tag` (#800)
  Add ORCID info (#794)
  Add blog post link to website (#792)
  Use serializer Content-Type header for PlumberEndpoint API spec (#789)
  Use dev version (#790)
  v1.1.0 (#752)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants