-
Notifications
You must be signed in to change notification settings - Fork 2
/
README.Rmd
264 lines (210 loc) · 11.4 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%",
error=TRUE
)
```
# ebvcube package
<!-- badges: start -->
[](https://CRAN.R-project.org/package=ebvcube)
[](https://github.com/EBVCube/ebvcube/actions/workflows/R.yaml)
[](https://app.codecov.io/gh/EBVcube/ebvcube)
<!-- badges: end -->
This package can be used to easily access the data of the EBV netCDFs which can be downloaded from the
[EBV Data Portal](https://portal.geobon.org/). It also provides some basic visualization. Advanced users can build their own netCDFs following the EBV structure.
## 1. Basis
The EBV netCDF structure is designed to hold Essential Biodiversity Variables. This concept is further described [here](https://geobon.org/ebvs/what-are-ebvs/).
The files are based on the [Network Common Data Format](https://www.unidata.ucar.edu/software/netcdf/) (netCDF). Additionally, it follows the [Climate and Forecast Conventions](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html) (CF, version 1.8) and the [Attribute Convention for Data Discovery](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3) (ACDD, version 1.3).
## 2. Data structure
The structure allows several datacubes per netCDF file. These cubes have four dimensions: longitude, latitude, time and entity, whereby the last dimension can, e.g., encompass different species or groups of species, ecosystem types or other. The usage of hierarchical groups enables the coexistence of multiple data cubes. All cubes share the same dimensions.
The first level (netCDF group) are scenarios, e.g., the modelling for different Shared Socioeconomic Pathways (SSP) scenarios. The second level (netCDF group) are metrics, e.g., the percentage of protected area per pixel and its proportional loss over a certain time span per pixel. All metrics are repeated per scenario, if any are present.
```bash
├── scenario_1
│ ├── metric_1
│ │ └── ebv_cube [lon, lat, time, entity]
│ │
│ └── metric_1
│ └── ebv_cube [lon, lat, time, entity]
│
└── scenario_2
├── metric_1
│ └── ebv_cube [lon, lat, time, entity]
│
└── metric_1
└── ebv_cube [lon, lat, time, entity]
```
Just keep in mind: All EBV netCDF always have a metric. But they may or may not have a scenario. The resulting datacubes hold the data. These datacubes are 4D.
## 2. Installation
You can install the ebvcube packages with:
``` r
#get the current version on CRAN
install.packages('ebvcube')
#install the latest version from GitHub
devtools::install_github('https://github.com/EBVCube/ebvcube/tree/dev')
#troubleshooting for BioConductor packages
#if one of the following packages is not loaded: rhdf5, DelayedArray, HDF5Array
install.packages("BiocManager")
BiocManager::install('rhdf5')
BiocManager::install('DelayedArray')
BiocManager::install('HDF5Array')
```
## 3. Working with the package - a quick intro
The example data set used in this README is a spatial subset (African continent) of the [Local bird diversity (cSAR/BES-SIM)](https://portal.geobon.org/ebv-detail?id=1) data set by Ines Martins.
### 3.1 Take a very first look at the file
With the following two functions you get the metadata of a specific EBV netCDF file.
First we take a look at some basic metadata of that file. The properties encompass much more information!
```{r example}
library(ebvcube)
#set the path to the file
file <- system.file(file.path("extdata", "martins_comcom_subset.nc"), package="ebvcube")
#read the properties of the file
prop_file <- ebv_properties(file, verbose=FALSE)
#take a look at the general properties of the data set - there are more properties to discover!
prop_file@general[c(1, 2, 4)]
slotNames(prop_file)
```
Now let's get the paths to all possible datacubes. The resulting data.frame includes the paths and also descriptions of the metric and/or scenario and/or entity. The paths basically consist of the nested structure of scenario, metric and the datacube.
```{r}
datacubes <- ebv_datacubepaths(file, verbose=FALSE)
datacubes
```
In the next step we will get the properties of one specific datacube - fyi: the result also holds the general file properties from above.
```{r}
prop_dc <- ebv_properties(file, datacubes[1, 1], verbose=FALSE)
prop_dc@metric
```
### 3.2 Plot the data to get a better impression
To discover the spatial distribution of the data, you can plot a map of the datacube that we just looked at. It has 12 timesteps. Here we look at the first one.
```{r}
#plot the global map
dc <- datacubes[2, 1]
ebv_map(file, dc, entity=1, timestep = 1, classes = 9,
verbose=FALSE, col_rev = TRUE)
```
It's nice to see the global distribution, but how is the change of that datacube (non forest birds) over time? Let's take a look at the average. The function returns the values, catch them!
```{r}
#get the averages and plot
averages <- ebv_trend(file, dc, entity=1, verbose=FALSE)
averages
```
It would be cool to have that for other indicators as well? Check out the different options for 'method'.
### 3.3 Read the data from the files to start working
Before you actually load the data it may be nice to get an impression of the value range and other basic measurements.
```{r}
#info for whole dataset
measurements <- ebv_analyse(file, dc, entity=1, verbose=FALSE)
#see the included measurements
names(measurements)
#check out the mean and the number of pixels
measurements$mean
measurements$n
#info for a subset defined by a bounding box
#you can also define the subset by a Shapefile - check it out!
bb <- c(-26, 64, 30, 38)
measurements_bb <- ebv_analyse(file, dc, entity = 1, subset = bb, verbose=FALSE)
#check out the mean of the subset
measurements_bb$mean
measurements_bb$n
```
To access the first three timesteps of the data you can use the following:
```{r}
#load whole data as array for two timesteps
data <- ebv_read(file, dc, entity = 1, timestep = 1:3, type = 'a')
dim(data)
```
You can also get a spatial subset of the data by providing a Shapefile.
```{r}
#load subset from shapefile (Cameroon)
shp <- system.file(file.path('extdata', 'cameroon.shp'), package="ebvcube")
data_shp <- ebv_read_shp(file, dc, entity=1, shp = shp, timestep = c(1, 2, 3), verbose=FALSE)
dim(data_shp)
#very quick plot of the resulting raster plus the shapefile
borders <- terra::vect(shp)
ggplot2::ggplot() +
tidyterra::geom_spatraster(data = data_shp[[1]]) +
tidyterra::geom_spatvector(data = borders, fill = NA) +
ggplot2::scale_fill_fermenter(na.value=NA, palette = 'YlGn', direction = 1) +
ggplot2::theme_classic()
```
Imagine you have a very large dataset but only limited memory. The package provides the possibility to load the data as a DelayedArray. The ebv_write() function helps you to write that data back on disk properly. Look into the manual to obtain more information.
### 3.4 Take a peek on the creation of an EBV netCDF
#### a. Create an empty EBV netCDF (with metadata)
First of all, you have to insert all the metadata in the [EBV Data Portal](https://portal.geobon.org/home) and then use the resulting text file (json format) to create an empty netCDF which complies to the EBV netCDF structure, i.e., it has the correct structure mapped to your data and holds the metadata. Additionally to that (json) text file, the function needs a list of all entities the netCDF (see help page for detailed information) will encompass and geospatial information such as the coordinate reference system.
The example is based on the [Local bird diversity (cSAR/BES-SIM) ](https://portal.geobon.org/ebv-detail?id=1).
```{r}
#paths
json <- system.file(file.path('extdata', 'metadata.json'), package="ebvcube")
new_nc <- file.path(system.file(package="ebvcube"), 'extdata', 'test.nc')
entities <- c('forest bird species', 'non-forest bird species', 'all bird species')
#defining the fillvalue - optional
fv <- -3.4e+38
#create the netCDF
ebv_create(jsonpath = json, outputpath = new_nc, entities = entities,
epsg = 4326, extent = c(-180, 180, -90, 90), resolution = c(1, 1),
fillvalue = fv, overwrite=TRUE, verbose=FALSE)
#needless to say: check the properties of your newly created file to see if you get what you want
#especially the entity_names from the slot general should be checked to see if your csv was formatted the right way
print(ebv_properties(new_nc, verbose=FALSE)@general$entity_names)
#check out the (still empty) datacubes that are available
dc_new <- ebv_datacubepaths(new_nc, verbose=FALSE)
print(dc_new)
```
Hint: You can always take a look at your netCDF in [Panoply](https://www.giss.nasa.gov/tools/panoply/) provided by NASA. That's very helpful to understand the structure.
#### b. Add your data to the EBV NetCDF
In the next step you can add your data to the netCDF from GeoTiff files or in-memory objects (matrix/array). You need to indicate the datacubepath the data belongs to. You can add your data timestep per timestep, in slices or all at once. You can simply add more data to the same datacube by changing the timestep definition.
```{r}
#path to tif with data
root <- system.file(file.path('extdata'), package="ebvcube")
tifs <- c('entity1.tif', 'entity2.tif', 'entity3.tif')
tif_paths <- file.path(root, tifs)
#adding the data
entity <- 1
for (tif in tif_paths){
ebv_add_data(filepath_nc = new_nc,
metric = 1,
entity = entity,
timestep=1:3,
data = tif,
band = 1:3,
verbose = FALSE)
entity <- entity + 1
}
```
#### c. Add missing attributes to datacube
Ups! So you did a mistake and want to change the attribute?! No problem. Just use the upcoming function to change it.
```{r}
ebv_attribute(new_nc, attribute_name='units', value='Percentage', levelpath=dc_new[1, 1])
#check the properties one more time - perfect!
print(ebv_properties(new_nc, dc_new[1, 1], verbose=FALSE)@ebv_cube$units)
```
In this case the levelpath corresponds to the datacube path. But you can also alter attributes at the metric or scenario level. See the manual for more info.
## 4. Cite package
```{r}
citation('ebvcube')
```
## List of all functions
|Functionality|Function|Description|
|:---|:---|:---|
|Basic access|ebv_datacubepaths|Get all available data cubes in the netCDF|
||ebv_properties|Get all the metadata of the netCDF|
||ebv_download|Download EBV netCDFs from the EBV Portal|
|Data access|ebv_read|Read the data|
||ebv_read_bb|Read a spatial subset given by a bounding box|
||ebv_read_shp|Read a spatial subset given by a Shapefile|
||ebv_analyse|Get basic measurements of the data|
||ebv_write|Write manipulated data back to disc|
|Data visualization|ebv_map|Plot a map of the specified data slice|
||ebv_trend|Plot the temporal trend |
|Data creation|ebv_create|Create a new EBV netCDF|
||ebv_create_taxonomy|Create a new EBV netCDF with taxonomy info|
||ebv_metadata|Create the EBV metadata text file (JSON)|
||ebv_add_data|Add data to the new netCDF|
||ebv_attribute|Change an attribute value|