scripts/02P_WOA_netcdf_to_zarr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c0460f2a-1a64-4d9e-93de-940b13970b72",
   "metadata": {},
   "source": [
    "# Extracting WOA23 data using FishMIP regional model boundaries\n",
    "**Author:** Tormey Reimer  \n",
    "**Date:** 2024-08-27  \n",
    "**Edited by:** Denisse Fierro Arcos  \n",
    "**Last edited on:** 2024-09-04  \n",
    "  \n",
    "This script transforms the original WOA files in `netcdf` format downloaded in the [01R_download_WOA_data.R](01R_download_WOA_data.R) script, and then converts them to a cloud-optimised `zarr` file.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3111c729-db73-4f3d-892d-27149d909268",
   "metadata": {},
   "source": [
    "## Setting working directory\n",
    "Remember to change the working directory below to the location of the scripts in your own local machine. Update the `your_path` variable below before continuing with the next chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9b7ff0d5-797d-40bb-ba58-4e553dc608dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "your_path = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ad33014e-6f29-43df-9e72-aabcd6c38d3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir(os.path.join(your_path, 'processing_WOA_data/scripts'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e9308c6-5df5-4739-aa00-d1c29c65f689",
   "metadata": {},
   "source": [
    "## Loading libraries\n",
    "We will load published Python libraries as well as our custom-made `useful_functions` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "52b71ba6-708d-457f-bc54-48325b24b4cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.distributed import Client\n",
    "from glob import glob\n",
    "import useful_functions as uf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e2313d9-501a-4454-9b06-737002c47db9",
   "metadata": {},
   "source": [
    "## Starting a cluster\n",
    "This will allow us to automatically parallelising tasks on large datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e5d68105-be15-4c01-beec-bf3d26d0fc6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(threads_per_worker = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393ff90e-9439-4bca-96e0-f844c32fa45c",
   "metadata": {},
   "source": [
    "## Defining basic variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a8337bba-1a80-422f-b3af-f468a8eb5d86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining path to WOA_data directory\n",
    "WOA_path = '/g/data/vf71/WOA_data/global/'\n",
    "\n",
    "#Get a list of all temperature files in the WOA directory\n",
    "WOA_temp = sorted(glob(os.path.join(WOA_path, '*/*_t*.nc')))\n",
    "#Get a list of all salinity files in the WOA directory\n",
    "WOA_sal = sorted(glob(os.path.join(WOA_path, '*/*_s*.nc')))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832550bf-0454-46ab-97a5-7fd479e80ada",
   "metadata": {},
   "source": [
    "## Processing monthly WOA data\n",
    "These files contain the month as two digits in the filename. For more information, you can refer to WOA documentation [here](https://www.ncei.noaa.gov/data/oceans/woa/WOA23/DOCUMENTATION/WOA23_Product_Documentation.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30471982-beb1-43bb-be21-1e8d71df66b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Selecting monthly data for temperature\n",
    "WOA_temp_month = [f for f in WOA_temp if '00' not in f]\n",
    "temp_out = os.path.join(WOA_path, 'woa23_month_clim_mean_temp_1981-2010.zarr')\n",
    "uf.netcdf_to_zarr(WOA_temp_month, 't_an', temp_out)\n",
    "\n",
    "# Selecting monthly data for salinity\n",
    "WOA_sal_month = [f for f in WOA_sal if '00' not in f]\n",
    "sal_out = os.path.join(WOA_path, 'woa23_month_clim_mean_sal_1981-2010.zarr')\n",
    "uf.netcdf_to_zarr(WOA_sal_month, 's_an', sal_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1052f040-a0ff-4b44-b838-7fe27066b17b",
   "metadata": {},
   "source": [
    "## Processing climatological mean over entire period\n",
    "These files can be identified because the include `00` in their filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76c9e2fc-a2de-4ff5-a17e-f7c47c99dce9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Climatological temperature mean\n",
    "WOA_temp = [f for f in WOA_temp if '00' in f]\n",
    "temp_out = os.path.join(WOA_path, 'woa23_clim_mean_temp_1981-2010.zarr')\n",
    "uf.netcdf_to_zarr(WOA_temp, 't_an', temp_out)\n",
    "# Number of observations (temperature)\n",
    "obs_temp_out = os.path.join(WOA_path, 'woa23_number_obs_temp_1981-2010.zarr')\n",
    "uf.netcdf_to_zarr(WOA_temp, 't_dd', obs_temp_out)\n",
    "\n",
    "# Climatological salinity mean\n",
    "WOA_sal = [f for f in WOA_sal if '00' in f]\n",
    "sal_out = os.path.join(WOA_path, 'woa23_clim_mean_sal_1981-2010.zarr')\n",
    "uf.netcdf_to_zarr(WOA_sal, 's_an', sal_out)\n",
    "# Number of observations (salinity)\n",
    "obs_sal_out = os.path.join(WOA_path, 'woa23_number_obs_salt_1981-2010.zarr')\n",
    "uf.netcdf_to_zarr(WOA_sal, 's_dd', obs_sal_out)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:analysis3-24.04] *",
   "language": "python",
   "name": "conda-env-analysis3-24.04-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}