Development Data Location?

[James-G-Hill]

Hello, this is more a query rather than bug or request. Many applications depend upon data that's downloaded from an external source such as an SQL database or AirTable. During development of such applications the download time for the dataset can be long enough to feel like an interruption in the workflow. It therefore sometimes seems best to include a simple dataset for development purposes that can be loaded quickly in from disk. In this case, where is the best place to store such data; would you recommend to put it in a sub-directory of the /dev directory & then load in within the 'run_dev.R' file?

[ColinFay]

(Moving this to the Discussion board :) )

Hey @James-G-Hill,

Thanks for your question.

This is indeed something that can be cumbersome while developing. I indeed would suggest using a local extraction of the db so that you don't have to do the query all the time.

Here are some strategies I have adopted in the past:

• When it's a tabular db (SQL), I extract a small portion of the db and use it either as a package dataset (https://r-pkgs.org/data.html), but the simplest way I've found is to write a csv / Rdata inside the inst folder. When inside inst, I read the dataset on the top of the server function with dataset <- read.csv(app_sys("dataset.csv")).

• When it's a NoSQL or graph db, I prefer having my own Docker container with a small piece of the whole db, but I understand it might be more complex. In that case you can also use a small portion of the db that you put in inst as an .Rdata file.

Whatever the approach, I then add a "local" argument to my run_app function, that is FALSE by default, then inside my run_dev.R script I switch this to TRUE. That way, local is TRUE only when developing.

Then on top of my server function, I do:

if (golem::get_golem_option("local")){
  dataset <- read.csv(app_sys("dataset.csv"))
} else {
  dataset <- do_the_external_sql_query()
}

Does that make sense to you?
Let me know if you need me to elaborate more around this idea :)

Cheers,

Colin

[James-G-Hill]

yes Colin, that makes sense. I didn't want to include non-production data in my inst directory. So the option I took was to make adding a dataset via golem_options possible. In the app_server including an if then statement to check whether the app is in production mode or not, then accepting the dataset through golem_options if it's not in production mode. If it is in production mode then just accept the data as usual. Then I modified the run_dev file to read in the data from a subdirectory of dev and pass in the data through run_app in dev mode.

It works for the architecture of my current app because the whole dataset gets loaded in at app_server either way but I'm sure it wouldn't work for all apps.