04-harmonisation.Rmd

# Harmonisation with synthetic data {#harmonisation}

In this section we describe how to harmonise synthetic data on the client side. This assumes that you have used one of the previous methods to generate your synthetic data set.

Recall that we are aiming to use synthetic data on the client side to design harmonisation algorithms, and then implement these on Opal on the server side using the real data. This removes the need for the user to have full access to the data. Harmonisation algorithms can be implemented in Opal using MagmaScript (JavaScript with some additional functions) without having full access to the data. The idea is that writing JavaScript on the client side, having full access to the synthetic data, is easier than writing the code on the server side with only access to summaries.

Additional steps for harmonisation after generation of synthetic data are:

1. With the synthetic data on the client side, the user can view the data and develop their code. They will be able to see the how the data changes as the code is run.
2. When the code is complete, it can be run on the serve side using the real data.

In detail, the steps proposed are:

1. Start a JavaScript session on the client side
2. Load the synthetic data into the session
3. Write and test JavaScript code in the session against the synthetic data
4. When happy, copy the code into Opal to generate the harmonised data

```{r echo=FALSE, fig.cap="Prototyping DataSHIELD harmonisation using synthetic data on Javascript"}
knitr::include_graphics(rep("images/dssynthetic_harm.png"))

```

## Getting set up

First we start a JavaScript session and load the additional MagmaScript functionality that is found in Opal. We also load our synthetic data into the JavaScript session.
```{r start V8}
library(V8)
ct2 = v8()
ct2$source("https://raw.githubusercontent.com/tombisho/dsSyntheticClient/main/MagmaScript.min.js")
synth_data = read.csv(file = "data/synth_data.csv")
ct2$assign("synth_data", synth_data)
```

We then go into the JavaScript v8 console.
```{r eval=FALSE}
ct2$console()

```

## Experiment with a single row
A MagmaScript function grabs the first row of data. We can then write some JavaScript to operate on that single row and show the result:
```{javascript}
var $ = MagmaScript.MagmaScript.$.bind(synth_data[0]);

if ($('y3age').value() > 25 ){
  out = 1
} else {
  out = 0
}
```
## Test on whole dataset
Now we test our code against the whole dataset. This is done by:

1. Defining the script as a string assigned to a variable
2. Execute this script in a loop through each row of data
3. Each time capture the output

```{javascript}

myScript = `
if ($('y3age').value() > 25 ){
  out = 1
} else {
  out = 0
}
`

var my_out = [];

for (j = 0; j < synth_data.length; j++){
  my_out.push(MagmaScript.evaluator(myScript, als_syn[j]))
}
exit
```
And pull the results into R for inspection:

```{r eval=FALSE}
my_out = ct2$get("my_out")

synth_data_harm = synth_data
synth_data_harm$my_var = my_out
```

## Run the code on the real data

If we are happy with the code, we can paste it directly into the Opal *script* interface so that it can be executed on the real data:

```{r echo=FALSE, fig.cap="Script editor in Opal"}
knitr::include_graphics(rep("images/opal_script.PNG"))

```

This will generate a harmonised variable in the view on Opal which can be used in analyses. The summary statistics of the harmonised data can be checked to make sure the harmonisation is working correctly.

A similar process could be conducted in a platform like MOLGENIS.