Feature/move more doc (#47)

* Update README and documentation

* Add some DFS waffle

* Add DFS references

* Correct Julia commas

README.md

@@ -1,4 +1,6 @@
 # AccordDaTools
+
+## Description
 Collection of ACCORD DA tools. So far we have jbdiagnose and associated plotting. 
 Also available is DFS calculation and associated plotting.
 
@@ -8,7 +10,9 @@ Desirables:
 - obstat
 - Other ...
 
-## Installation
+## Getting started
+
+## Installing
 ```
 cd AccordDaTools
 mkdir build
@@ -23,64 +27,22 @@ Add DA Tools to your PATH:
 export PATH=/path/to/da_tools/bin:$PATH
 ```
 
-## Tools
-### Structure functions
-#### Jb diagnose
-Full documentation is available [here](doc/tools/jbdiagnose.md).
+## Documentation
+The documentation for _Accord DA tools_ is available on [https://accord-nwp.github.io/AccordDaTools](https://accord-nwp.github.io/AccordDaTools).
 
-Help/usage:
-```
-jbdiagnose -h
-```
+Contributions adding/correcting the documenation are welcome via using Pull Request.
 
-Example:
-```
-jbdiagnose -b jb_data/stabfiltn_IRELAND75L65_jbconv.bal -c jb_data/stabfiltn_IRELAND75L65_jbconv.cv -g 15000 -l harmL65 -e IRELAND75L65
-```
+## Help
+TBC
 
-#### plotjbbal
-Help/usage:
-```
-plotjbbal -h
-```
+## Authors
+TBC
 
-Example:
-```
-plotjbbal -t stdv -p QQ -r ./ -e IRELAND75L65
-```
+## Version history
+TBC
 
-#### plotjbdiag
-Help/usage:
-```
-plotjbdiag -h
-```
+## License
+TBC
 
-Example:
-```
-plotjbdiag -l 50 -t vercor -p QQ -r ./ -e IRELAND75L65 
-```
-
-### DFS
-The dfscomp tool reads the (ASCII) data from an unperturbed CCMA and a perturbed CCMA.
-
-#### Calculate DFS
-Help/usage:
-```
-scripts/dfscomp.sh -h
-```
-
-Example:
-```
-scripts/dfscomp.sh -u $DTG_CCMA_unpert.dat -p $DTG_CCMA.dat -o dfs.dat 
-```
-
-#### Plot DFS
-Help/usage:
-```
-python3 scripts/plotdfs.py -h
-```
-
-Example:
-```
-python3 scripts/plotdfs.py -i dfs.dat
-```
+## Acknowledgements
+TBC

doc/make.jl

@@ -3,9 +3,10 @@ using Documenter
 
 pages = [
    "Jb diagnose" => "jbdiagnose.md",
-   "cv header list" => "cvheaderlist.md",
-   "VarBc Coeff" => "varbccoeff.md",
-   "DIACOV" => "diacov.md"
+   "cv header" => "cvheaderlist.md",
+   "DIACOV" => "diacov.md",
+   "DFS" => "dfs.md",
+   "VarBc Coeff" => "varbccoeff.md"
 ]
 
 prettyurls = get(ENV, "CI", nothing) == "true" 

doc/src/dfs.md

@@ -1,15 +1,57 @@
-#DFS
 
-## DFS computation
+# Degree of Freedom Signal
+
+"Degree of Freedom Signal" (DFS) is a used to quantify the influence of observational data on the analysis in numerical weather prediction (NWP) and other geophysical systems. It provides a measure of the information content of the observations with respect to the analysis.
+
+### Definition
+The DFS is defined as the trace of the product of the observation sensitivity matrix and the observation error covariance matrix. Mathematically, it can be expressed as:
+
+\[ \text{DFS} = \text{Tr}(\mathbf{K} \mathbf{H}) \]
+
+where:
+- \(\mathbf{K}\) is the Kalman gain matrix, which represents how much weight is given to the observations in the assimilation process.
+- \(\mathbf{H}\) is the observation operator, which maps the model state variables to the observed variables.
+
+### Interpretation
+1. **Information Content**: DFS indicates how much the observations have influenced the analysis. A higher DFS means that the observations have a significant impact on the analysis, providing more information.
+2. **Observational Weight**: It reflects the relative weight of the observations compared to the background information (prior model state). A higher DFS suggests that the observations are trusted more compared to the model forecast.
+3. **Data Quality and Quantity**: The DFS can help assess the quality and the effective number of independent observations. High DFS values can indicate good quality data or a large number of observations contributing to the analysis.
+
+### Importance in Data Assimilation
+- **Performance Monitoring**: DFS can be used to monitor and evaluate the performance of the data assimilation system. By tracking the DFS over time, one can identify periods where observations are particularly influential or where the system might rely too heavily on the background model.
+- **Observation System Design**: In designing observational networks, DFS can help determine the most valuable locations and times for taking observations to maximize their impact on the analysis.
+- **Error Diagnostics**: By analyzing the DFS, one can diagnose potential issues with observation errors, model errors, or the assimilation scheme itself.
+
+### Practical Example
+In NWP, suppose you assimilate satellite radiance data into a global atmospheric model. The DFS would tell you how much the satellite data has altered the model's initial conditions. If the DFS is high, the satellite data has significantly adjusted the model state, indicating that the data is highly informative. Conversely, a low DFS might suggest that the observations are not very influential, possibly due to high observational errors or redundancy with existing data.
+
+### References
+ - Chapnik, B., Desroziers, G., Rabier, F., & Talagrand, O. (2006). Diagnosis and tuning of observational error statistics in a quasi-operational data assimilation setting. *Quarterly Journal of the Royal Meteorological Society, 132*(616), 543-565. https://doi.org/10.1256/qj.05.82
+ - Cardinali, C., Pezzulli, S., & Andersson, E. (2004). Influence-matrix diagnostic of a data assimilation system. *Quarterly Journal of the Royal Meteorological Society, 130*(603), 2767-2786. https://doi.org/10.1256/qj.03.205
+
+
+## Calculate DFS
+
 The dfscomp tool reads the (ASCII) data from an unperturbed CCMA and a perturbed CCMA.
 
 Help/usage:
 ```
-dfscomp -h
+scripts/dfscomp.sh -h
+```
+
+Example:
+```
+scripts/dfscomp.sh -u $DTG_CCMA_unpert.dat -p $DTG_CCMA.dat -o dfs.dat
+```
+
+## Plot DFS
+Help/usage:
+```
+python3 scripts/plotdfs.py -h
 ```
 
 Example:
 ```
-dfscomp -p CCMA_2022050112.dat -u CCMA_2022050112_unpert.dat -o dfs2022050112.dat
+python3 scripts/plotdfs.py -i dfs.dat
 ```
 

doc/src/jbdiagnose.md

@@ -4,6 +4,43 @@
 A tool to diagnose HARMONIE structure function files and a comparison between structure functions for different AROME domains
 Converted from [original](assets/jbdiagnose.pdf) by Nils Gustafsson
 
+
+
+## jbdiagnose
+
+Help/usage:
+```
+jbdiagnose -h
+```
+
+Example:
+```
+jbdiagnose -b jb_data/stabfiltn_IRELAND75L65_jbconv.bal -c jb_data/stabfiltn_IRELAND75L65_jbconv.cv -g 15000 -l harmL65 -e IRELAND75L65
+```
+
+## plotjbbal
+Help/usage:
+```
+plotjbbal -h
+```
+
+Example:
+```
+plotjbbal -t stdv -p QQ -r ./ -e IRELAND75L65
+```
+
+## plotjbdiag
+Help/usage:
+```
+plotjbdiag -h
+```
+
+Example:
+```
+plotjbdiag -l 50 -t vercor -p QQ -r ./ -e IRELAND75L65
+```
+
+
 ## 1 Introduction
 The development and implementation of the HARMONIE data assimilation are now in a quite advanced stage with pre-operational testing at most of the HIRLAMmember weather services. At the same time, the technical and scientific knowledge about various components of this data assimilation is not so widely spread and a joint effort is needed forthe validation of these locally installed system. One of the important components of the HARMONIE data assimilation is the background error statistics. A tool for diagnostics of the background error statistics directly from the background error statistics files has therefore been developed and applied to the recently developed statistics files for HARMONIE at mesoscale (AROME)resolution (2.5 km grid resolution) at SMHI, met.no, FMI, DMI, KNMI and AEMET. This note describes the diagnostic tool (jbdiagnose) and presents results from a comparison of the different statistics from the AROME implementations. Since this diagnostic software is quite fresh, there may certainly be coding errors that could affect the results. Any comments on the comparison and the graphs presented here are therefore most welcome.