Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Method to set unload at a package level #174

Closed
DyfanJones opened this issue Nov 24, 2021 · 1 comment
Closed

Method to set unload at a package level #174

DyfanJones opened this issue Nov 24, 2021 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@DyfanJones
Copy link
Owner

Currently noctua supports AWS Athena unload method through adhoc setting of parameter unload in dbGetQuery , dbSendQuery and dbExecute.

Unfortunetly this means that dplyr functionality doesn't benefit from this feature. By having a method to set unload = TRUE on a package level then dplyr can take advantage of this feature with little implementation on dplyr side of things.

@DyfanJones
Copy link
Owner Author

Implementation seems to be working nicely :D

library(DBI)
library(noctua)
library(dplyr, warn.conflict = FALSE)

con <- dbConnect(athena())

noctua_options(cache = 10, unload = TRUE)

df = dbGetQuery(con, "select * from awswrangler_test.noaa")

noaa = tbl(con, dbplyr::in_schema("awswrangler_test", "noaa"))

noaa %>% filter(!is.na(m_flag))

noctua:::athena_option_env$cache_dt

#>                                 QueryId                                                                                 Query     State StatementType WorkGroup                            UnloadDir
#> 1: 774b6a83-cf53-4a5b-849c-d4c144bcf672 SELECT *\nFROM "awswrangler_test"."noaa"\nWHERE (NOT((("m_flag") IS NULL)))\nLIMIT 11 SUCCEEDED           DML   primary 08313f36-6da9-490f-8ba0-947f14d9a9f8
#> 2: 9a94ae61-8b85-4620-9440-9033990dc83f                                                   select * from awswrangler_test.noaa SUCCEEDED           DML   primary 72c55e13-6770-4b15-8e98-bc77be1e5c8b

DyfanJones added a commit that referenced this issue Nov 24, 2021
feature: add unload to noctua_options to all dplyr unload functionality (#174)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant