Add support for data repositories #1

ThierryO · 2018-07-24T12:22:58Z

Add functionality to store and retrieve R data.frames to a git repository as git optimized text files

…rage Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

…orrectly Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Swapping two variables when rewritten a data.frame results in a large diff while the information content of the data hasn't changed. Therefore the variables will be reordered to match the original order. Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

When a line is moved in a file, the resulting diff is a deletion at the original location and an addition at the new location. Changing the order of the observations in a data.frame does not change the information content. Sorting the data before writing avoids unnecessary diffs. Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

…within the sorting variables Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

…tead of "data_repository" Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

ThierryO · 2018-08-22T09:40:31Z

@stijnvanhoey and @florisvdh: can you have another look at this?

Main changes:

use the git_repository class instead of the data_repository class (which has been removed)
allow the user to store data in verbose format (default remains the optimized format)
automatic staging of the changes is now optional
add a new functionrm_data() to remove data objects in bulk
updated vignette

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

dir.exists() is not available in R < 3.2.0 Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

hansvancalster · 2018-08-23T11:50:22Z

vignettes/data-repository.Rmd

+### Storing data
+
+Use `write_delim_git()` to store a `data.frame` into the repository. The function will separate the data and the metadata. The data is stored as a headerless, unquoted tab delimited file with ".tsv" extension and UTF-8 encoding. The metadata is stored in YAML format with ".yml" extension. Therefore any extension given to the `file` will be stripped (with a warning).
+


Just to check if I get it right... the metadata contains the variable names, their type and in case of factors the mapping between levels and labels + the sorting order?

Should attributes of the data.frame be saved as metadata as well? For instance rownames?

IMHO: if rownames are important they should be a variable

hansvancalster · 2018-08-23T11:52:50Z

vignettes/data-repository.Rmd

+
+```{r}
+# undo the remove by resetting to the last commit
+reset(commits(repo)[[1]], "hard")


is this line necessary here?

The alternative is rewritting the data objects. resetting the repo to the last commit is more elegant.

hansvancalster · 2018-08-23T11:53:41Z

vignettes/data-repository.Rmd

+
+### Verbose data storage
+
+`write_delim_git()` will store the data by default in an optimize way in the repository. The downside of this is that the stored data is less human-readable. 


optimize -> optimized

fixed in 51c4d7e

hansvancalster · 2018-08-23T11:56:02Z

vignettes/data-repository.Rmd

+
+### Reading data
+
+Retrieving data is straight forward. Use `read_delim_git` and provide the `file` and the `repo`. The retrieved data is identical to the original data after applying the ordering of variables and observations.


straight forward -> straightforward

fixing in 51c4d7e

hansvancalster

Really useful to have a write and read for data.frames to and from a git repo! Cool!

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

stijnvanhoey

With the concept of data repository cleared out, the functionality is very clear to me, nice work! I have not verified the code itself, but the vignette provides a good introduction. Thanks, looking forward to further use and test it.

florisvdh · 2018-09-04T07:34:39Z

I agree with stijnvanhoey. I did not retest the functions but from the git status output in the vignette they seems to behave very well. Functionality is very straightforward now and at least for me the data handling might become the primary use of the git2r package. I like the way rm_file() now works. Also other changes and enhancements much appreciated.

florisvdh · 2018-09-04T07:40:37Z

I have just one small remark; the title of the vignette still says 'data repository'. It does not bother me though, if you left it on purpose -- it is clear from the title that the focus of the vignette is on data and from the explanation it is clear how it works.

ThierryO added 30 commits July 23, 2018 13:33

add a non exported meta() function which prepares the vectors for sto…

2d000ec

…rage Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Add write_delim_git()

03d8f6d

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Add read_delim_git()

ec2b5fd

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

write_delim_git() and read_delim_git() handle directories with dots c…

e2d0e0e

…orrectly Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

repository() gains a "project" argument

ecbf3f4

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

write_delim_git() and read_delim_git() use the project concept

9086194

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

init() and clone() gain the "project" argument

18b38bb

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

export meta()

7b490df

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

workdir() takes "project" into account when set

a19f3a3

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

move meta() and read_delim_git() and use 4 spaces per tab

66817dc

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

write_delim_git() stages the files

7dfa131

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

status() takes "project" into account

ebcc9c8

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

rm_file() handles data repos

a9349ca

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

add is_data_repo()

6cf0737

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Fix typo

3bfceb9

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

reset() handles data repositories

cdb3e81

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

fix bugs

57df033

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

add unit tests for data repositories

8b90f13

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Create a "data_repository" class

8566c41

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

update unit tests

2f24fcb

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

rm_file() can delete all .tsv. or all .yml files

edf9ee7

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

meta.character() checks for 'NA' values

84c96f9

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

allow the storage of logicals in data repositories

1599206

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

data repositories handle complex data

091f9b0

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

data repositories handle POSIXct timestamps

b550898

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

add draft version of vignette

98514f1

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

write_delim_git() yields a warning in case of duplicate observations …

a60c248

…within the sorting variables Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

don't return the call with data_repository errors

c220999

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Vignette: replace 'format' by 'extension'

04b8ca0

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

ThierryO force-pushed the datarepos branch from 39f797e to 04b8ca0 Compare August 20, 2018 13:26

ThierryO added 9 commits August 20, 2018 19:17

write_delim_git() and read_delim_git() work with "git_repository" ins…

1ae9f9c

…tead of "data_repository" Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

remove the data_repository class

dea6be9

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

add rm_data()

5281e40

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

update vignette

42bddcd

write_delim_git() gains a stage argument

cbc9829

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

write_delim_git() gains a optimize argument

f476e18

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

revert unneeded changes

d78f9f1

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

Add more unit tests

ee881ca

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

More work on the vignette

3fe6a6f

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

ThierryO added 2 commits August 22, 2018 12:01

bugfixes

d26aaf9

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

use file.exists() instead of dir.exists()

24dc997

dir.exists() is not available in R < 3.2.0 Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

hansvancalster reviewed Aug 23, 2018

View reviewed changes

ThierryO added 3 commits August 24, 2018 09:50

fix typos

51c4d7e

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

add recent_commit()

fa8cddb

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

try to debug unit test on recent_commit()

1ddbb14

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

ThierryO force-pushed the datarepos branch from 429dc38 to 1ddbb14 Compare August 29, 2018 12:21

ThierryO added 2 commits August 29, 2018 14:47

use forward slashes when normalising path

b427175

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

clean_data_path() gains a normalize argument

b748146

Signed-off-by: Thierry Onkelinx <thierry.onkelinx@inbo.be>

stijnvanhoey approved these changes Aug 30, 2018

View reviewed changes

ThierryO mentioned this pull request Sep 6, 2018

Storing and retrieving data.frames ropensci/git2r#366

Closed

ThierryO mentioned this pull request Apr 18, 2019

git2rdata, a companion package for git2r ropensci/software-review#263

Closed

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for data repositories #1

Add support for data repositories #1

ThierryO commented Jul 24, 2018

ThierryO commented Aug 22, 2018

hansvancalster Aug 23, 2018

ThierryO Aug 24, 2018

hansvancalster Aug 23, 2018

ThierryO Aug 24, 2018

hansvancalster Aug 23, 2018

ThierryO Aug 24, 2018

hansvancalster Aug 23, 2018

ThierryO Aug 24, 2018

hansvancalster left a comment

stijnvanhoey left a comment

florisvdh commented Sep 4, 2018

florisvdh commented Sep 4, 2018

		### Storing data

		Use `write_delim_git()` to store a `data.frame` into the repository. The function will separate the data and the metadata. The data is stored as a headerless, unquoted tab delimited file with ".tsv" extension and UTF-8 encoding. The metadata is stored in YAML format with ".yml" extension. Therefore any extension given to the `file` will be stripped (with a warning).


		### Verbose data storage

		`write_delim_git()` will store the data by default in an optimize way in the repository. The downside of this is that the stored data is less human-readable.


		### Reading data

		Retrieving data is straight forward. Use `read_delim_git` and provide the `file` and the `repo`. The retrieved data is identical to the original data after applying the ordering of variables and observations.

Add support for data repositories #1

Are you sure you want to change the base?

Add support for data repositories #1

Conversation

ThierryO commented Jul 24, 2018

ThierryO commented Aug 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hansvancalster left a comment

Choose a reason for hiding this comment

stijnvanhoey left a comment

Choose a reason for hiding this comment

florisvdh commented Sep 4, 2018

florisvdh commented Sep 4, 2018