From 09b873ff47173a74e2c0af483297971f1c504f8e Mon Sep 17 00:00:00 2001 From: k204229 Date: Fri, 28 Jun 2024 17:30:41 +0200 Subject: [PATCH] fix: remove doubled copied presentation for freva plugin jup notebook --- ...ay 2 - Introduction to Freva Plugins.ipynb | 1637 ----------------- 1 file changed, 1637 deletions(-) diff --git a/talks/Freva_Workshop_2024/day 2/Day 2 - Introduction to Freva Plugins.ipynb b/talks/Freva_Workshop_2024/day 2/Day 2 - Introduction to Freva Plugins.ipynb index 687f8fc..0e0d9ec 100644 --- a/talks/Freva_Workshop_2024/day 2/Day 2 - Introduction to Freva Plugins.ipynb +++ b/talks/Freva_Workshop_2024/day 2/Day 2 - Introduction to Freva Plugins.ipynb @@ -1637,1643 +1637,6 @@ "" ] }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# Freva: Plugin Development\n", - "\n", - "\"freva\n", - "\n", - "**Martin Bergemann, Bijan Fallah, Andrej Fast, Mostafa Hadizadeh, \n", - "Christopher Kadow, Etor E. Lucio-Eceiza, Felix Örtel, Manuel Reis, and many others…**\n", - "\n", - "Deutsches Klimarechenzentrum @ CLINT in Data Analysis Dpt. (+ Data Management Dpt.*)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# Overview of a Freva plugin\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Why should I build a Freva plugin?\n", - "It allows you to be part of \"The Freva Cycle\"\n", - "\n", - "\"parts\"\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Structure of a Plugin\n", - "
\n", - "
\n", - "

There are 3 main building blocks in a plugin:

\n", - "
    \n", - "
  • Wrapper: connects the tool to Freva
    • \n", - "
  • Tool: scientific code
    • \n", - "
  • Installer: they install the necessary libraries
    • \n", - "
\n", - "
\n", - "
\n", - " \"parts\"\n", - "
\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Structure of a Plugin: The Wrapper\n", - "
\n", - "
\n", - "

The connection to Freva:

\n", - "
    \n", - "
  • Written in python 3
    • \n", - "
  • Needs to follow a series of standards
    • \n", - "
\n", - "
\n", - "
\n", - " \"parts\"\n", - "
\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Structure of a Plugin: The tool\n", - "
\n", - "
\n", - "

The scientific code:

\n", - "
    \n", - "
  • Written in any* language (python, R, C, FORTRAN, a mix...)
    • \n", - "
  • A single script or multiple scripts and/or folders (subroutines, libraries, etc.).
    • \n", - "
\n", - "
\n", - "
\n", - " \"parts\"\n", - "
\n", - "
\n", - "*caveat: we encourage free software" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Structure of a Plugin: Installer\n", - "
\n", - "
\n", - "

Installs the plugin environment:

\n", - "
    \n", - "
  • Usually: Makefile (instructions) + .yaml file (libraries)
    • \n", - "
  • It can also be more elaborated: e.g. binary compilation, installation of subroutines...
    • \n", - "
\n", - "
\n", - "
\n", - " \"parts\"\n", - "
\n", - "
\n", - "\n", - "**NOTE:** It replaces the dependencies on the HPC (e.g. `modules`) and makes it more transferable to other e.g. institutions" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# Git and Gitlab" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## What is git? \"freva\n", - "Is a free and open source *Distributed* **Version Control System**:\n", - "- **Version Control System**: a system that tracks and provides control over changes to source code, documentation and configuration files.\n", - "- *Distributed*: a form of version control in which the complete codebase, including its full history, is mirrored on every developer's computer\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## What is gitlab? \"freva\n", - "\n", - "
\n", - "
\n", - "
  • GitLab is a web-based tool: provides version control, issue tracking, CI/CD etc.
    • \n", - "
  • DKRZ hosts its own framework at https://gitlab.dkrz.de
    • \n", - "
    \n", - "
    \n", - " \"freva\n", - "
    \n", - "
    \n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Gitlab and Freva:\n", - "\n", - "- One central local repository for the plugged plugins\n", - "- Other local repositories for different developers\n", - "- One remote repository for the plugin at gitlab\n", - "\"freva\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Gitlab and Freva: location of the plugins\n", - "\n", - "- Under [https://gitlab.dkrz.de/freva/plugins4freva](https://gitlab.dkrz.de/freva/plugins4freva) project:\n", - "\"freva\n", - "- Other projects have also their own, e.g.: \n", - " - climxtreme: https://gitlab.dkrz.de/bm1159/plugins4freva\n", - " - regiklim-nukleus: https://gitlab.dkrz.de/ch1187/plugins4freva\n", - "- some plugins can be found in different projects (see [forks](https://docs.gitlab.com/ee/user/project/repository/forking_workflow.html)).\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Gitlab and Freva: cloning a repository\n", - "To download a repository there are 2 ways: via `https` and via `ssh`:\n", - "\"freva\n", - "- `https`: straightforward, requires to enter (dkrz) username and password (usually). \n", - "- `ssh`: convenient, requires to provide a ssh key previously.\n", - "\n", - "In both cases:\n", - "\n", - "```\n", - "$ ssh @levante.dkrz.de\n", - "> cd /path/to/my/work/\n", - "> git clone https://gitlab.dkrz.de/freva/plugins4freva/plugin_workshop/example_plugin.git\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Gitlab and Freva: setting your SSH key\n", - "1. generate a ssh a private and public (`.pub`) key locally (e.g. laptop or levante):\n", - "```\n", - "$ ssh-keygen -t ed25519\n", - "$ cd ~/.ssh/\n", - "$ ls\n", - "id_ed25519 id_ed25519.pub\n", - "```\n", - "2. copy the **public** key: \n", - "```\n", - "$ cat id_ed25519.pub\n", - "ssh-ed25519 AAAAC3NzaC1lFDI1YTE5AAAAICUqpXx8AyrNH7HQgPSq9LuB7sTEmXP4ODqOnaHVPPQ6 username@levante\n", - "```\n", - "3. add the public key in gitlab (`user>preferences>ssh keys`): \"freva" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Gitlab and Freva: creating a new project\n", - "- If we already had an existing Git repository:\n", - "```shell\n", - "$ cd NEW_PLUGIN # for example\n", - "$ git remote rename origin old-origin\n", - "$ git remote add origin git@gitlab.dkrz.de:freva/plugins4freva/new_plugin.git\n", - "$ git push -u origin --all\n", - "$ git push -u origin --tags\n", - "```\n", - "\n", - "- If we did not already have an existing Git repository:\n", - "```shell\n", - "$ cd NEW_PLUGIN # for example\n", - "$ git init\n", - "$ git remote add origin git@gitlab.dkrz.de:freva/plugins4freva/new_plugin.git\n", - "# create your new files and folders\n", - "$ git add .\n", - "$ git commit -m \"Initial commit of my new project\"\n", - "$ git push -u origin master\n", - "```\n", - "\n", - "Now you could (hypothetically) go to https://gitlab.dkrz.de/freva/plugins4freva/new_plugin" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# License of Plugins" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## What kind of License should I follow?\n", - "\n", - "[The Freva code](https://github.com/FREVA-CLINT/freva/blob/main/LICENSE.md) follows a [BSD3 license](https://opensource.org/license/bsd-3-clause), a permissive open-source license with minimal restrictions:\n", - "- It allows for wide usage, including in proprietary software.\n", - "- it imposes very few restrictions on how the software can be redistributed.\n", - "\n", - "In principle, any derived product from Freva (i.e. plugins), should follow this license. *Unless* stated otherwise by the developer.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## What kind of License should I follow?\n", - "\n", - "- We encourage open-source software to foster collaboration within and outside your projects.\n", - "- That said, common courtesy should be expected, e.g.:\n", - " - reference to the first manuscript where the developer used the plugin (the plugin documentation is useful for this!).\n", - " - contacting him/her in case there is no such manuscript." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# Connecting your IDE / Code Editor to a remote session using SSH" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## 1. VS-Code \"freva\n", - "\n", - "Open a new window in VS Code. There are several ways to connect to a remote session, as indicated in the picture below:\n", - "\n", - "![Image depicting multiple ways to start a remote session in VS Code](media/vscode/01_connect_to_remote.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\n", - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Clicking either of the buttons opens a prompt at the top of the window, giving you several options to start a remote session:\n", - "\n", - "![Image depicting the prompt that opens when clicking one of the \"Connect to...\" buttons in the new VS Code window](media/vscode/02_remote_connection_options.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Click \"Connect Current Window to Host...\" (\"Connect to Host...\" opens a new window for the connection).\n", - "\n", - "You can type in the address of the remote host that you want to connect to or, in case the host is specified in your ssh config file (.ssh/config or ssh_config), select them from the list of hosts. To connect to levante, enter [k-number@]levante.dkrz.de :\n", - "\n", - "![Image depicting the prompt to enter a remote SSH host, with k202195@levante.dkrz.de manually entered](media/vscode/03_remote_host_entered.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "The first time connecting to a host, you will be prompted to specify the OS of the remote host (i.e. Linux / Windows / macOS).\n", - "\n", - "Depending on the authentication set up for the remote host, you might be prompted to type in a password. Once the connection has been established, your can open a folder on the remote host, like you would do on your local machine (by pressing Ctrl + K, Ctrl + O or by pressing \"Open Folder\"):\n", - "\n", - "![Image depicting how to open a folder once connected to the remote host, similar to opening a local folder](media/vscode/04_opening_remote_folder.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "You may be prompted again to type in the password at this point. The window should refresh and give you access to the remote folder that you selected in the previous step, similar to what can be seen in the following image:\n", - "\n", - "\"freva" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "From here you can use the editor to work with remote files basically as you would with files on your local machine. Any terminals opened in this window will also connect to the remote host.\n", - "\n", - "For detailed documentation on remote development using VS Code, including development on dev containers or WSL, click [here](https://code.visualstudio.com/docs/remote/remote-overview)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## 2. PyCharm \"freva\n", - "\n", - "\n", - "> 😱\n", - ">Remote Development is currently in beta and only supported for the Professional Version of PyCharm!\n", - "\n", - "Open your Plugins (Ctrl+Alt+S) and confirm that \"Remote Development Gateway\" is installed and that it is enabled:\n", - "\n", - "![Image depicting a list of installed PyCharm plugins, with the \"Remote Development Gateway\" being installed and enabled.](media/pycharm/01_remote_development_gateway.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "To open a remote session in PyCharm, go to the welcome screen and click on the \"Remote Development\" Tab in the left side bar. This should give you a number of options, including the option to connect to a remote session using SSH:\n", - "\n", - "![Image depiciting the welcome screen of PyCharm Professional, with the \"Remote Development\" Tab highlighted using red arrows.](media/pycharm/02_remote_development_tab.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "To establish a new SSH Connection, click \"New Connection\" below the SSH Connection option. This should open the following dialogue:\n", - "\n", - "![Image depicting the dialogue that opens when creating a new SSH connection on PyCharm. Includes fields for the username, the host name or address, and the port that should be used connected to.](media/pycharm/03_ssh_connection_dialogue.jpg)\n", - ">⚠\n", - ">Note that PyCharm currently only supports connecting to remote hosts running Linux." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Again, depending on the type of authentication set up by the remote host, you may be prompted to enter a password to establish the remote connection.\n", - "\n", - "In the next step (pending the successful connection to the remote host), you will be asked to set the IDE version (i.e. the version of the IDE that will be installed on the remote host) and the project directory (i.e. the directory on the remote host that contains the project files that you will be working with):\n", - "\n", - "![Image depicting the PyCharm window that prompts you to set the IDE Version to be installed on the remote host, as well as the project directory that you will be working in](media/pycharm/04_choose_ide_and_project.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Click \"Download IDE and Connect\". The installation of the selected IDE on the remote host might take a while. Once the installation is complete, a window containing the remote IDE should open.\n", - "\n", - "![Image depicting the remote IDE window, after successfully installing it to the remote host](media/pycharm/05_remote_IDE_successfully_opened.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "The remote IDE gives you the ability to work with project files, much like you would with local project files. Any terminal sessions started in this IDE will also be connected to the remote host.\n", - "\n", - "For detailed documentation on remote development using PyCharm, click [here](https://www.jetbrains.com/help/pycharm/remote-development-starting-page.html)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## 3. R Studio \"freva\n", - "\n", - "There is currently only one method of using R Studio on a remote host that is officially supported by R, namely by connecting to an instance of R Studio Server running on the remote host. The previously available alternative, by creating a remote connection in [R Studio Desktop Pro](https://docs.posit.co/ide/desktop-pro/2022.02.1+461.pro1/remote-desktop.html), has been removed since version [2023.06.2](https://docs.posit.co/ide/desktop-pro/remote_desktop/remote_desktop.html).\n", - "\n", - "However, R Studio Server is not available on Levante, so creating a remote development environment instance this way is not possible. However, it is possible to remotely develop and execute R-code by using the [VS Code R Extension](https://code.visualstudio.com/docs/languages/r). " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "First, connect to Levante as outlined in the VS Code section of this guide. Once the connection is established, you will want to install the R (`REditorSupport.r`) and R Debugger (`RDebugger.r-debugger`) Extensions, as shown below:\n", - "![Image depicting R VS Code Plugins](media/rstudio/01_rstudio_vscode_extensions.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Make sure that you have installed the extensions to the remote host, and not your local machine, by going to Extensions (Ctrl+Shift+X):\n", - "\n", - "![Image showing that R VS Code Plugins where installed on Levante](media/rstudio/02_rstudio_extensions_on_remote.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "In a terminal that is connected to levante via ssh (e.g. in the VS Code window connected to levante), type in the following command:\n", - "```bash\n", - "$ module load r\n", - "```\n", - "The result should be something along the lines of:\n", - "```bash\n", - "[k202195@levante3 ~]$ module load r\n", - "Autoloading openjdk/17.0.0_35-gcc-11.2.0\n", - "\n", - "Loading r/4.1.2-gcc-11.2.0\n", - " Loading requirement: openjdk/17.0.0_35-gcc-11.2.0\n", - "[k202195@levante3 ~]$\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "We'll need to find out, where R is located on the remote host. We do this by using the command:\n", - "```bash\n", - "$ which R\n", - "```\n", - "The output should be something like:\n", - "```bash\n", - "[k202195@levante3 ~]$ which R\n", - "/sw/spack-levante/r-4.1.2-eprwea/bin/R\n", - "[k202195@levante3 ~]$\n", - "```\n", - "Copy this path to your clipboard." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Install the [httpgd](https://github.com/nx10/httpgd) package to embed live R graphics into the VS Code environment. You can do this by running the following commands in a terminal on the remote host:\n", - "```bash\n", - "# Start and R command line session\n", - "$ R\n", - "```\n", - "The following command is executed in the R command line session\n", - "```R\n", - "install.packages(\"httpgd\")\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Open the settings (Ctrl + ,) and type `r.rterm.linux` in the search bar as shown in the image below. Make sure that you paste the previously copied path into the field, as depicted:\n", - "\n", - "![Image depicting how to set the path for the R Terminal in VS Code](media/rstudio/03_rterm_linux_settings.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "Once that is done, open any R file on the remote host. Once an R terminal is ready, you could either select the code or put the cursor at the beginning or ending of the code you want to run, press (Ctrl+Enter), and then code will be sent to the active R terminal.\n", - "\n", - "If you want to run an entire R file, open the file in the editor, and press Ctrl+Shift+S and the file will be sourced in the active R terminal:\n", - "\n", - "![Image depicting some R code being run remotely on Levante and the resulting interactive plots](media/rstudio/04_opening_and_running_r_file_in_vscode.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "\"freva\n", - "
    \n", - "
    \n", - "
    \n", - "\n", - "For more information on working with R in VS Code, click [here](https://code.visualstudio.com/docs/languages/r)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# What is a Freva Wrapper?" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "

    Freva Wrapper Main Classes

    \n", - "\n", - "
    \n", - "
    \n", - "
      \n", - "
    • PluginAbstract Class Functions
      • \n", - "
      \n", - "
    • parameters.py Classes
      • \n", - "
      \n", - "
    • _ConfigWrapper Class
      • \n", - "
    \n", - "
    \n", - "
    \n", - " \"What\n", - "
    \n", - "
    " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Special Variables:\n", - "\n", - "| Variables | Description |\n", - "|--------------------|------------------------------------------------------------------------|\n", - "| USER_BASE_DIR | Absolute path to the central directory for this user. |\n", - "| USER_OUTPUT_DIR | Absolute path to where the plugin outputs for this user are stored. |\n", - "| USER_PLOTS_DIR | Absolute path to where the plugin plots for this user are stored. |\n", - "| USER_CACHE_DIR | Absolute path to the cached data (temp data) for this user. |\n", - "| USER_UID | The users’ User Identifier. |\n", - "| SYSTEM_DATE | Current date in the form YYYYMMDD (e.g. 20120130). |\n", - "| SYSTEM_DATETIME | Current date in the form YYYYMMDD_HHmmSS (e.g. 20120130_101123). |\n", - "| SYSTEM_TIMESTAMP | Milliseconds since epoch (i.e. e.g. 1358929581838). |\n", - "| SYSTEM_RANDOM_UUID | A random Universal Unique Identifier (e.g. 912cca21-6364-4f46-9b03-4263410c9899). |\n", - "\n", - "Example: \n", - "\n", - "```bash\n", - "output_file='$USER_OUTPUT_DIR/myfile_${SYSTEM_DATETIME}blah.nc'\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: class_basedir\n", - "```python\n", - "class PluginAbstract(abc.ABC):\n", - " \"\"\"Base class that is used as a template for all Freva plugins.\n", - " Any api wrapper class defining Freva plugins must inherit from this class.\"\"\"\n", - "\n", - " @deprecated_method(\"PluginAbstract\", \"class_basedir\")\n", - " def getClassBaseDir(self) -> Optional[str]:\n", - " \"\"\"Deprecated method for class_basedir.\"\"\"\n", - "\n", - " def class_basedir(self) -> str:\n", - " \"\"\"Get absolute path to the module defining the plugin class.\"\"\"\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: call\n", - "```python\n", - "class PluginAbstract(abc.ABC):\n", - " \"\"\"Base class that is used as a template for all Freva plugins.\n", - " Any api wrapper class defining Freva plugins must inherit from this class.\"\"\"\n", - "\n", - " def call(\n", - " self,\n", - " cmd_string: Union[str, list[str]],\n", - " check: bool = True,\n", - " **kwargs,\n", - " ) -> sub.Popen[Any]:\n", - " \"\"\"Run command with arguments and return a CompletedProcess instance.\"\"\"\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: call and class_basedir - Example\n", - "```python\n", - "from evaluation_system.api import plugin\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " def run_tool(self, config_dict=None)\n", - " self.call(\n", - " f\"{self.class_basedir}/main_code.sh {config_dict}\"\n", - " )\n", - " # or\n", - " self.call(\n", - " f\"{Path(__file__).parent}/main_code.sh {config_dict}\"\n", - " )\n", - " # or\n", - " self.call(\n", - " f\"{os.getenv(USER_BASE_DIR)}/main_code.sh {config_dict}\"\n", - " ) \n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: add_output_to_databrowser\n", - "```python\n", - "class PluginAbstract(abc.ABC):\n", - " @deprecated_method(\"PluginAbstract\", \"add_output_to_databrowser\")\n", - " def linkmydata(self, *args, **kwargs): # pragma: no cover\n", - "\n", - " def add_output_to_databrowser(\n", - " self,\n", - " plugin_output: os.PathLike,\n", - " project: str, # user-\n", - " product: str, # project.product\n", - " *,\n", - " model: str = \"freva\",\n", - " institute: Optional[str] = None,\n", - " ensemble: str = \"r1i1p1\",\n", - " time_frequency: Optional[str] = None,\n", - " variable: Optional[str] = None,\n", - " experiment: Optional[str] = None,\n", - " index_data: bool = True,\n", - " ) -> Path:\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: add_output_to_databrowser - Example\n", - "```python\n", - "from evaluation_system.api import plugin\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " def run_tool(self, config_dict=None)\n", - " self.add_output_to_databrowser(\n", - " plugin_output=config_dict[\"outputdir\"],\n", - " project=config_dict.get(\"project\", \"default_project\"),\n", - " product=config_dict.get(\"product\", \"default_product\"),\n", - " model=config_dict.get(\"model\", \"default_model\"),\n", - " institute=config_dict.get(\"institute\", \"default_institute\"),\n", - " time_frequency=\"mon\",\n", - " )\n", - "```\n", - "\n", - "### How to find it in Freva:\n", - "\n", - "```bash\n", - "freva databrowser product=project.product project=user-\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: prepare_output\n", - "```python\n", - "class PluginAbstract(abc.ABC):\n", - " \"\"\"Base class that is used as a template for all Freva plugins.\n", - " Any api wrapper class defining Freva plugins must inherit from this class.\"\"\"\n", - "\n", - " @deprecated_method(\"PluginAbstract\", \"prepare_output\")\n", - " def prepareOutput(self, *args) -> dict[str, dict[str, str]]:\n", - " \"\"\"Deprecated method for :class:`prepare_output`. \"\"\"\n", - "\n", - " def prepare_output(\n", - " self, output_files: Union[str, list[str], dict[str, dict[str, str]]]\n", - " ) -> dict[str, dict[str, str]]:\n", - " \"\"\"Prepare output for files supposedly created.\"\"\"\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# PluginAbstract Class\n", - "## Function: prepare_output - Example\n", - "```python\n", - "from evaluation_system.api import plugin\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " def run_tool(self, config_dict=None)\n", - " return self.prepare_output(config_dict[\"outputdir\"])\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# ParameterType Class\n", - "```python\n", - "class ParameterType(initOrder):\n", - " \"\"\"Base class for all parameter types.\n", - " All available parameter types inherit from this class. The class creates\n", - " a parameter object holding the following user defined information.\n", - " \"\"\"\n", - " def __init__(\n", - " self,\n", - " name=None,\n", - " default=None,\n", - " mandatory=False,\n", - " max_items=1,\n", - " item_separator=\",\",\n", - " regex=None,\n", - " version=1,\n", - " help=\"No help available.\",\n", - " print_format=\"%s\",\n", - " impact=Parameter.Impact.affects_values,\n", - " ):\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# ParameterDictionary Class\n", - "```python\n", - "class ParameterDictionary(dict):\n", - " \"\"\"Directory holding all plugin parameters for a ``Freva`` plugin.\n", - " This class behaves like a built-in ``dict`` with additional features.\n", - " The most prominent feature is that the order of added items is preserved,\n", - " as opposed to a normal build-in ``dict``.\n", - " \"\"\"\n", - " def __init__(self, *parameters: ParameterType) -> None:\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "# _ConfigWrapper class\n", - "```python\n", - "class _ConfigWrapper:\n", - " \"\"\"Convenience class that helps to dynamically set the location of the\n", - " evaluation system config file.\"\"\"\n", - " _env: str = \"EVALUATION_SYSTEM_CONFIG_FILE\"\n", - " def __init__(\n", - " self,\n", - " default_file: str,\n", - " ):\n", - " self.default_file = default_file\n", - " def __fspath__(self) -> str:\n", - " return os.environ.get(self._env, self.default_file)\n", - " def __repr__(self) -> str:\n", - " return self.__fspath__()\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "

    Freva Wrapper Structure

    \n", - "\n", - "
    \n", - "
    \n", - "
      \n", - "
    • Plugin Details
      • \n", - "
      \n", - "
    • Plugin Parameters
      • \n", - "
      \n", - "
    • Plugin Methods
      • \n", - "
    \n", - "
    \n", - "
    \n", - " \"wrapper's\n", - "
    \n", - "
    " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Freva Wrapper Structure: 1. Details\n", - "```python\n", - "from evaluation_system.api import plugin\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " __version__ = (0, 0, 0)\n", - " \"\"\"\n", - " def __version__(self) -> tuple[int, int, int]:\n", - " # 3-value tuple representing the plugin version.\n", - " \"\"\"\n", - " __short_description__: str = \"template plugin\"\n", - " __category__ = \"something\"\n", - " \"\"\"\n", - " def __category__(self) -> str:\n", - " # Optional category this plugin belongs to.\n", - " return \"\"\n", - " \"\"\"\n", - " __tags__ = [\"something\", \"somethingelse\"]\n", - " \"\"\"\n", - " def __tags__(self) -> list[str]:\n", - " # Optional tags, that are the plugin can be described with.\n", - " return [\"\"]\n", - " \"\"\"\n", - " tool_developer: Optional[str] = \"someonename someonefamily\"\n", - " __long_description__: Optional[str] = \"template plugin long\"\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Freva Wrapper Structure: 2. Parameters\n", - "```python\n", - "from evaluation_system.api.parameters import ParameterDictionary, String, Bool\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " __parameters__ = ParameterDictionary(\n", - " String(ParameterType),\n", - " \"\"\"\n", - " Example:\n", - " String(\n", - " name=\"string\",\n", - " default=\"None\",\n", - " help=(\n", - " \"some helps\"\n", - " ),\n", - " )\n", - " \"\"\"\n", - " Bool(ParameterType),\n", - " \"\"\"\n", - " Example:\n", - " Bool(\n", - " name=\"checkbox\",\n", - " default=True,\n", - " )\n", - " \"\"\"\n", - " )\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Freva Wrapper Structure: 2. Parameters\n", - "```python\n", - "from evaluation_system.api.parameters import ParameterDictionary, SolrField, InputDirectory, String\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " __parameters__ = ParameterDictionary( \n", - " InputDirectory(String),\n", - " \"\"\"\n", - " Example:\n", - " InputDirectory(\n", - " name=\"inputdirectory\",\n", - " ),\n", - " \"\"\"\n", - " SolrField(String),\n", - " \"\"\"\n", - " class SolrField(String):\n", - " # A parameter using solr for finding valid values.\n", - " def __init__(\n", - " self,\n", - " *args,\n", - " facet: str,\n", - " group: int = 1,\n", - " multiple: bool = False,\n", - " predefined_facets: Optional[list[str]] = None,\n", - " editable: bool = True,\n", - " **kwargs,\n", - " ): \n", - " \"\"\"\n", - " )\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Freva Wrapper Structure: 2. Parameters\n", - "```python\n", - "from evaluation_system.api.parameters import ParameterDictionary, String, CacheDirectory, SelectField, Directory\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " __parameters__ = ParameterDictionary(\n", - " Directory(String),\n", - " CacheDirectory(Directory),\n", - " SelectField(String),\n", - " \"\"\"\n", - " class SelectField(String):\n", - " # Select field to select parameter from predefined values.\n", - " def __init__(self, options: dict[str, str], *args, **kwargs):\n", - " Example:\n", - " SelectField(\n", - " options={\"first\": \"1st\", \"second\": \"2nd\"},\n", - " ),\n", - " \"\"\"\n", - " Integer(ParameterType),\n", - " \"\"\"\n", - " Example:\n", - " Integer(\n", - " name=\"integer\",\n", - " default=1,\n", - " )\n", - " \"\"\"\n", - " )\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Freva Wrapper Structure: 3. Methods\n", - "```python\n", - "from evaluation_system.api import plugin\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " def run_tool(self, config_dict=None):\n", - " self.call(f\"python template_plugin.py arg1 arg2 {config_dict}\")\n", - " self.call(f\"template_runner.sh {config_dict}\")\n", - " self.add_output_to_databrowser(\n", - " output_dir,\n", - " project=config_dict[\"project\"], # user-\n", - " product=config_dict[\"product\"], # f\"{config_dict[\"project\"]}.{config_dict[\"project\"]}\"\n", - " institute=config_dict[\"institute\"],\n", - " model=config_dict[\"model\"],\n", - " experiment=config_dict[\"experiment\"],\n", - " time_frequency=config_dict[\"time_frequency\"],\n", - " variable=config_dict[\"variable\"],\n", - " ensemble=config_dict[\"ensemble\"],\n", - " )\n", - " return self.prepare_output(config_dict[\"outputdir\"])\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Recommended: Keep track of errors\n", - "\n", - "```python\n", - "from evaluation_system.api import plugin\n", - "import json\n", - "class TemplatePlugin(plugin.PluginAbstract):\n", - " def run_tool(self, config_dict=None):\n", - " json_file = Path(config_dict[\"cache\"]) / \"out.json\"\n", - " with json_file.open(\"w\") as json_f:\n", - " json.dump(config_dict, json_f, indent=4)\n", - " with errorfile.open(\"w\") as error_f:\n", - " self.call(\n", - " f\"{self.class_basedir}/main_runner.sh {json_file}\",\n", - " stderr=error_f,\n", - " )\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "### Additional: How to deal with JSON files in the main script of plugin\n", - "#### Python:\n", - "```python\n", - "import json\n", - "import sys\n", - "with open(sys.argv[1]) as f:\n", - " config_dict = json.load(f)\n", - "```\n", - "#### Bash\n", - "```bash\n", - "_jq(){\n", - " jq -r ${1} ${2} | sed \"s/null/None/g;s/true/True/g;s/false/False/g\"\n", - "\n", - "}\n", - "export CACHE=\"$(_jq .cache $1)\"\n", - "export OUTPUTDIR=\"$(_jq .outputdir $1)\"\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "### Wrapper Illustration\n", - "\n", - "![First Wrapper](media/wrapper/First_wrapper.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "### Wrapper Illustration\n", - "\n", - "![Second Wrapper](media/wrapper/Second_wrapper.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "### Wrapper Illustration\n", - "\n", - "![Third Wrapper](media/wrapper/Third_wrapper.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "### Wrapper Illustration\n", - "\n", - "![Fourth Wrapper](media/wrapper/Fourth_wrapper.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# How To Install All Needed Libraries Of A Plugin" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Why the need?\n", - "\n", - "- Formerly, we relied on HPC modules to run plugins' software.\n", - "- Less than ideal when migrating to newer HPC/other institutions!\n", - "- Now, all necessary software installed in a `conda/mamba` environment at **each** repo folder." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "

    Steps to Install Necessary Packages

    \n", - "\n", - "
    \n", - "
    \n", - "
      \n", - "
    • Define an Env YAML file
      • \n", - "
      \n", - "
    • Install env via Conda
      • \n", - "
      \n", - "
    • Activate Environment
      • \n", - "
      \n", - "
    • Alternative: Makefile
      • \n", - "
    \n", - "
    \n", - "
    \n", - " \"install\n", - "
    \n", - "
    " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Define an Env YAML file\n", - "e.g. `plugin-env.yml`:\n", - "```yaml\n", - "channels:\n", - " - conda-forge\n", - "dependencies:\n", - " - r-base\n", - " - pip\n", - "```\n", - "## Install env via Conda\n", - "- The environment **must** be located at the same level as the wrapper file!\n", - "- The environment **must** be located named `plugin_env/` so Freva recognizes it!\n", - "```bash\n", - "mamba env create --prefix ./plugin_env -f plugin-env.yml \n", - "```\n", - "## Activate Environment\n", - "```bash\n", - "conda activate plugin_env/bin/activate\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## More flexible approach via Makefile\n", - "\n", - "We can define a `Makefile` as below and then after definition of `environment.yml`, install all the dependencies listed above via `make` commands. But first and foremost, we need to ensure, we have `make` installed on our machine.\n", - "\n", - "```Makefile\n", - "env:\n", - "\t@# Installs a conda environment with R and the IDF package\n", - "\t@if [ -d \"./plugin_env\" ]; then \\\n", - "\t\trm -rf ./plugin_env; \\\n", - "\tfi\n", - "\tconda env create --prefix ./plugin_env -f environment.yml --force\n", - "\tRscript -e 'install.packages(\"IDF\", repos=\"https://CRAN.R-project.org/\")'\n", - "cartopy:\n", - "\t@# Download various cartopy shape-files\n", - "\t@if [ -d \"./cartopy\" ]; then \\\n", - "\t\trm -rf ./cartopy; \\\n", - "\tfi\n", - "\t./plugin_env/bin/python3 assets/cartopy_download.py physical cultural\n", - ".PHONY: env cartopy\n", - "```\n", - "\n", - "* Using `mamba` instead of using `conda` can make the way of installation faster.\n", - "\n", - "You can take a look at a plugin template [here](https://gitlab.dkrz.de/freva/plugins4freva/plugintemplate)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "slide" - } - }, - "source": [ - "# How To Plug a Tool into Freva" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "

    Plug a Tool into Freva

    \n", - "\n", - "
    \n", - "
    \n", - "
      \n", - "
    • Plug into Freva via CLI
      • \n", - "
      \n", - "
    • Plug into Freva via Freva-Web
      • \n", - "
      \n", - "
    \n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Plug a tool into Freva\n", - "\n", - "\n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Plug a tool into Freva\n", - "\n", - "In this step we need to clone a plugin in the user directory of freva-dev and replace `` with your `k-number` or `b-number` username:\n", - "\n", - "```sh\n", - "ssh @levante.dkrz.de\n", - "#After login to Levante\n", - "cd work/ch1187/clint/freva-dev/share/work//freva-dev/\n", - "git clone https://gitlab.dkrz.de/freva/plugins4freva/plugin_workshop/example_plugin.git\n", - "```\n", - "In this step you need to install the dependencies. Have a look at the previous section (How To Install All Needed Libraries Of A Plugin)\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Plug a tool into Freva\n", - "\n", - "\n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "slideshow": { - "slide_type": "subslide" - } - }, - "source": [ - "## Plug a tool into Freva\n", - "\n", - "\n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    \n", - " \"plug\n", - "
    \n", - "
    " - ] - }, { "cell_type": "markdown", "metadata": {