Part 2 - Image Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cornerstone Project - Eric Rodrigues de Carvalho\n",
    "\n",
    "This notebook is a series of codes that serves as one of the primary tools to execute my cornerstone project as an undergraduated student called \"SUPER-RESOLUÇÃO DE IMAGENS EM\n",
    "TOMOGRAFIA COMPUTADORIZADA DE BAIXA\n",
    "DOSAGEM: COMPARAÇÃO DE MÉTODOS DE\n",
    "APRENDIZADO PROFUNDO\" (SUPER-RESOLUTION OF IMAGES IN\n",
    "COMPUTED TOMOGRAPHY OF LOW\n",
    "DOSAGE: COMPARISON OF METHODS OF\n",
    "DEEP LEARNING).\n",
    "\n",
    "The main goals of this notebook are:\n",
    "\n",
    "1. After using chaiNNer as specified in the thesis, execute metrics in a subset of images to evaluate DL models performance in both no-reference and full-reference metrics and also its runtime.\n",
    "2. Plot examples for visual comparision.\n",
    "\n",
    "### Warning\n",
    "\n",
    "##### This notebook only process the results after chaiNNer generates them. If you haven't done this, go back to \"Part 1 - Data and chaiNNer preparation .ipynb\" and follow the Section 3 steps.\n",
    "\n",
    "##### If you followed all Sections from Part 1, you can \"Run All\" this notebook after reading all information below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Super-Resolution Model Evaluation Process\n",
    "\n",
    "This script is designed to evaluate the performance of multiple image super-resolution models by calculating several image quality metrics between the super-resolved images generated by these models and the ground truth high-resolution images. The models being evaluated include DAT, ESRGAN, HAT, SRCNN, and SwinIR, all of which aim to reconstruct high-quality images from lower-resolution observations.\n",
    "\n",
    "#### 1. **Input Preparation**\n",
    "\n",
    "- **Observation Images**: These are the low-resolution test images that will be super-resolved by the different models. The observation images are stored in a designated directory and follow a structured naming convention.\n",
    "- **Ground Truth Images**: The corresponding high-resolution images are used as references for comparison. These images are essential for calculating the quality metrics and are stored in a separate directory, also following a structured naming format.\n",
    "\n",
    "#### 2. **Super-Resolved Images from Models**\n",
    "\n",
    "- Each model has its own folder where the super-resolved images generated by that model are stored. These images are compared against the ground truth to determine how well each model reconstructs the high-resolution details from the low-resolution observations.\n",
    "\n",
    "#### 3. **Metrics Calculation**\n",
    "\n",
    "To assess the quality of the super-resolved images, the following metrics are calculated:\n",
    "\n",
    "- **PSNR (Peak Signal-to-Noise Ratio)**: This measures how closely the super-resolved image resembles the ground truth in terms of pixel intensity.\n",
    "- **SSIM (Structural Similarity Index)**: SSIM evaluates how similar the structure of the image is compared to the ground truth, focusing on contrast, luminance, and texture.\n",
    "- **LPIPS (Learned Perceptual Image Patch Similarity)**: LPIPS assesses the perceptual quality of the image, accounting for how humans perceive image differences, using deep learning.\n",
    "- **NIQE (Natural Image Quality Evaluator)**: This is a no-reference metric, meaning it doesn’t require a ground truth image. It assesses the quality of the super-resolved image based on statistical properties.\n",
    "- **NRQM (No-Reference Quality Metric)**: Like NIQE, this is a no-reference metric but focuses on perceptual image quality without requiring a reference image.\n",
    "- **PI (Perceptual Index)**: This is a combination of the NIQE and NRQM scores and provides an overall perceptual quality index for the image.\n",
    "  The metric values, along with the computation times, are recorded for analysis.\n",
    "\n",
    "#### 4. **Processing Flow**\n",
    "\n",
    "- **Preprocessing**: The observation and ground truth images are loaded from their respective directories and converted to arrays for further processing.\n",
    "- **Metric Computation**: For each pair of images (observation/model output vs. ground truth), the defined metrics are computed. In some cases, like LPIPS, NIQE, and NRQM, the images are first converted to PyTorch tensors.\n",
    "- **Result Storage**: The calculated metric values, along with the corresponding image IDs and model names, are saved to a structured list. Additionally, the runtime for each metric calculation is recorded for performance analysis.\n",
    "\n",
    "#### 5. **Data Export**\n",
    "\n",
    "- Once all images and models are processed, the results are compiled into a pandas DataFrame, which is then exported as a CSV file. This CSV contains detailed information on each metric for each image and model, providing a comprehensive evaluation of the models’ performance.\n",
    "\n",
    "#### 6. **Output Analysis**\n",
    "\n",
    "- The final DataFrame allows for further analysis of how each model performs across different images and metrics. By analyzing this data, one can determine which models excel in different aspects of super-resolution, such as perceptual quality, structural accuracy, or overall similarity to the ground truth.\n",
    "\n",
    "This evaluation process not only provides quantitative insights into each model's performance but also highlights trade-offs between various metrics, allowing for informed decisions when choosing a model for specific tasks.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "This script evaluates the performance of different image super-resolution models\n",
    "by calculating various image quality metrics, such as PSNR, SSIM, LPIPS, NIQE, NRQM,\n",
    "and Perceptual Index (PI), between the super-resolved images generated by the models\n",
    "and the ground truth high-resolution images.\n",
    "\"\"\"\n",
    "\n",
    "import os\n",
    "import glob\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "import skimage.metrics\n",
    "import torch\n",
    "from lpips import LPIPS\n",
    "import pyiqa\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "root_path = \"./\"\n",
    "\n",
    "# Paths to the folders containing the observation images, ground truth images, and model-generated images.\n",
    "observation_path = root_path+ 'Dataset/observation_test_images'\n",
    "ground_truth_path = root_path+ 'Dataset/ground_truth_test_images'\n",
    "model_paths = {\n",
    "    'DAT': root_path+'Images/DAT',\n",
    "    'ESRGAN': root_path+'Images/ESRGAN',\n",
    "    'HAT': root_path+'Images/HAT',\n",
    "    'SRCNN': root_path+'Images/SRCNN',\n",
    "    'SwinIR': root_path+'Images/SwinIR'\n",
    "}\n",
    "\n",
    "# Define the metrics to calculate for evaluation\n",
    "metrics = ['PSNR', 'SSIM', 'LPIPS', 'NIQE', 'NRQM', 'PI']\n",
    "\n",
    "# Initialize the LPIPS model for perceptual similarity (AlexNet backbone)\n",
    "lpips_model = LPIPS(net='alex')\n",
    "\n",
    "# Initialize the NIQE and NRQM models using pyiqa for no-reference image quality assessments\n",
    "niqe_model = pyiqa.create_metric('niqe')\n",
    "nrqm_model = pyiqa.create_metric('nrqm')\n",
    "\n",
    "\n",
    "def pil_to_tensor(image):\n",
    "    \"\"\"\n",
    "    Converts a PIL image to a PyTorch tensor.\n",
    "\n",
    "    Args:\n",
    "        image (PIL.Image): Input PIL image.\n",
    "\n",
    "    Returns:\n",
    "        torch.Tensor: Image converted to a 4D tensor with shape (1, C, H, W).\n",
    "    \"\"\"\n",
    "    image = np.array(image).astype(np.float32) / \\\n",
    "        255.0  # Normalize to [0, 1] range\n",
    "    image = torch.tensor(image).permute(2, 0, 1)  # Change to [C, H, W] format\n",
    "    image = image.unsqueeze(0)  # Add batch dimension [1, C, H, W]\n",
    "    return image\n",
    "\n",
    "\n",
    "def calculate_pi(niqe, nrqm):\n",
    "    \"\"\"\n",
    "    Calculates the Perceptual Index (PI) metric based on NIQE and NRQM scores.\n",
    "\n",
    "    PI = (10 - NRQM) / 2 + NIQE / 2\n",
    "\n",
    "    Args:\n",
    "        niqe (float): NIQE score of the image.\n",
    "        nrqm (float): NRQM score of the image.\n",
    "\n",
    "    Returns:\n",
    "        float: Perceptual Index score.\n",
    "    \"\"\"\n",
    "    return (10 - nrqm) / 2 + niqe / 2\n",
    "\n",
    "\n",
    "def calculate_metrics(image1, image2):\n",
    "    \"\"\"\n",
    "    Calculates multiple image quality metrics (PSNR, SSIM, LPIPS, NIQE, NRQM, PI)\n",
    "    between two images.\n",
    "\n",
    "    Args:\n",
    "        image1 (np.array): Ground truth image as a NumPy array.\n",
    "        image2 (np.array): Test or model-generated image as a NumPy array.\n",
    "\n",
    "    Returns:\n",
    "        dict: A dictionary with metric names as keys and tuples of (metric_value, runtime) as values.\n",
    "    \"\"\"\n",
    "    metrics_data = {}\n",
    "\n",
    "    # PSNR (Peak Signal-to-Noise Ratio)\n",
    "    start_time = time.time()\n",
    "    psnr = skimage.metrics.peak_signal_noise_ratio(image1, image2)\n",
    "    metrics_data['PSNR'] = (psnr, time.time() - start_time)\n",
    "\n",
    "    # SSIM (Structural Similarity Index)\n",
    "    start_time = time.time()\n",
    "    ssim = skimage.metrics.structural_similarity(\n",
    "        image1, image2, channel_axis=-1)\n",
    "    metrics_data['SSIM'] = (ssim, time.time() - start_time)\n",
    "\n",
    "    # Convert images to tensors for LPIPS, NIQE, and NRQM calculation\n",
    "    observation_tensor = pil_to_tensor(observation_img)\n",
    "    ground_truth_tensor = pil_to_tensor(ground_truth_img)\n",
    "\n",
    "    # LPIPS (Learned Perceptual Image Patch Similarity)\n",
    "    start_time = time.time()\n",
    "    lpips_score = lpips_model(ground_truth_tensor, observation_tensor).item()\n",
    "    metrics_data['LPIPS'] = (lpips_score, time.time() - start_time)\n",
    "\n",
    "    # NIQE (Natural Image Quality Evaluator)\n",
    "    start_time = time.time()\n",
    "    niqe_score = niqe_model(observation_tensor).item()\n",
    "    metrics_data['NIQE'] = (niqe_score, time.time() - start_time)\n",
    "\n",
    "    # NRQM (No-Reference Quality Metric)\n",
    "    start_time = time.time()\n",
    "    nrqm_score = nrqm_model(observation_tensor).item()\n",
    "    metrics_data['NRQM'] = (nrqm_score, time.time() - start_time)\n",
    "\n",
    "    # Perceptual Index (PI)\n",
    "    start_time = time.time()\n",
    "    pi_score = calculate_pi(niqe_score, nrqm_score)\n",
    "    metrics_data['PI'] = (pi_score, time.time() - start_time)\n",
    "\n",
    "    return metrics_data\n",
    "\n",
    "\n",
    "# Get a list of all observation images in the dataset and sort by filename to ensure consistency\n",
    "observation_images = glob.glob(os.path.join(observation_path, '*.bmp'))\n",
    "observation_images.sort()\n",
    "\n",
    "# Select the first 150 observation images\n",
    "selected_observation_images = observation_images\n",
    "\n",
    "# Extract the image IDs from the filenames (assuming filenames follow a specific format)\n",
    "image_ids = [int(os.path.basename(image).split('.')[0].split('_')[-1])\n",
    "             for image in selected_observation_images]\n",
    "\n",
    "# Initialize an empty list to store results\n",
    "data = []\n",
    "\n",
    "# Process the observation images and calculate metrics\n",
    "for observation_image, observation_number in zip(selected_observation_images, image_ids):\n",
    "    # Find the corresponding ground truth image\n",
    "    ground_truth_image = os.path.join(\n",
    "        ground_truth_path, \n",
    "        f'ground_truth_test_{observation_number:04d}.bmp'\n",
    "    )\n",
    "\n",
    "    # Open the observation and ground truth images as PIL images\n",
    "    observation_img = Image.open(observation_image)\n",
    "    ground_truth_img = Image.open(ground_truth_image)\n",
    "\n",
    "    # Convert the images to NumPy arrays\n",
    "    observation_array = np.array(observation_img)\n",
    "    ground_truth_array = np.array(ground_truth_img)\n",
    "\n",
    "    # Calculate metrics for the observation image\n",
    "    observation_metrics = calculate_metrics(\n",
    "        ground_truth_array, observation_array)\n",
    "    for metric, (value, runtime) in observation_metrics.items():\n",
    "        data.append([observation_number, 'Observation',\n",
    "                    metric, value, runtime])\n",
    "\n",
    "# Process images from each model and calculate metrics\n",
    "for model, model_path in model_paths.items():\n",
    "    for observation_number in image_ids:\n",
    "        # Find the corresponding model-generated and ground truth images\n",
    "        model_image = os.path.join(\n",
    "            model_path,\n",
    "            f'observation_test_{observation_number:04d}.bmp'\n",
    "        )\n",
    "        ground_truth_image = os.path.join(\n",
    "            ground_truth_path,\n",
    "            f'ground_truth_test_{observation_number:04d}.bmp'\n",
    "        )\n",
    "\n",
    "        # Open the model and ground truth images as PIL images\n",
    "        observation_img = Image.open(model_image)\n",
    "        ground_truth_img = Image.open(ground_truth_image)\n",
    "\n",
    "        # Convert the images to NumPy arrays\n",
    "        observation_array = np.array(observation_img)\n",
    "        ground_truth_array = np.array(ground_truth_img)\n",
    "\n",
    "        # Calculate metrics for the model-generated image\n",
    "        model_metrics = calculate_metrics(\n",
    "            ground_truth_array, observation_array)\n",
    "        for metric, (value, runtime) in model_metrics.items():\n",
    "            data.append([observation_number, model, metric, value, runtime])\n",
    "\n",
    "# Convert the collected data into a DataFrame for analysis and export\n",
    "df = pd.DataFrame(\n",
    "    data, columns=['Image_ID', 'Model', 'Metric', 'Value', 'Runtime'])\n",
    "\n",
    "# Save the DataFrame to a CSV file for further use\n",
    "df.to_csv(root_path+'metrics_results.csv', index=False)\n",
    "\n",
    "# Print the DataFrame to view the results\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Group by 'Model' and 'Metric' and calculate the mean and standard deviation of 'Value'\n",
    "aggregated_df = df.groupby(['Model', 'Metric']).agg(\n",
    "    {'Value': ['mean', 'std']}\n",
    ")\n",
    "\n",
    "# Print the aggregated DataFrame\n",
    "print(aggregated_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Group by 'Model' and 'Metric' and calculate the mean and standard deviation of 'Value'\n",
    "aggregated_df = df.groupby(['Model', 'Metric']).agg(\n",
    "    {'Value': ['mean', 'std']}\n",
    ")\n",
    "\n",
    "# Get the unique metrics\n",
    "metrics = df['Metric'].unique()\n",
    "\n",
    "# Initialize the dictionary to store the best models\n",
    "best_results = {}\n",
    "\n",
    "# Iterate over each metric\n",
    "for metric in metrics:\n",
    "    # If the metric is PSNR, SSIM, or NRQM, find the model with the highest mean value\n",
    "    if metric in ['PSNR', 'SSIM', 'NRQM']:\n",
    "        best_model = aggregated_df.loc[(\n",
    "            slice(None), metric), ('Value', 'mean')].idxmax()[0]\n",
    "    # Otherwise, find the model with the lowest mean value\n",
    "    else:\n",
    "        best_model = aggregated_df.loc[(\n",
    "            slice(None), metric), ('Value', 'mean')].idxmin()[0]\n",
    "    # Store the best model for the current metric\n",
    "    best_results[metric] = best_model\n",
    "\n",
    "# Print the best models for each metric\n",
    "print('Best results:')\n",
    "for metric, model in best_results.items():\n",
    "    print(f'{metric}: {model}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Group by 'Model' and 'Metric' and calculate the mean and standard deviation of 'Value'\n",
    "aggregated_df = df.groupby(['Model', 'Metric']).agg(\n",
    "    {'Value': ['mean', 'std']}\n",
    ")\n",
    "\n",
    "# Print the aggregated DataFrame\n",
    "print(aggregated_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path + 'metrics_results.csv')\n",
    "\n",
    "# Set up the grid for 3 rows and 2 columns\n",
    "fig, axes = plt.subplots(3, 2, figsize=(12, 18))\n",
    "axes = axes.flatten()\n",
    "\n",
    "# Define the metrics\n",
    "metrics = df['Metric'].unique()\n",
    "\n",
    "# Loop through each metric\n",
    "for i, metric in enumerate(metrics):\n",
    "    # Select the axis\n",
    "    ax = axes[i]\n",
    "\n",
    "    # Filter the data for the current metric\n",
    "    data = df[df['Metric'] == metric]\n",
    "\n",
    "    # Create the boxplot\n",
    "    boxplot = sns.boxplot(x='Model', y='Value', data=data, ax=ax)\n",
    "\n",
    "    # Customize colors for specific models\n",
    "    for j, artist in enumerate(ax.patches):\n",
    "        # Extract the corresponding model for the current box\n",
    "        model = data['Model'].unique()[j]\n",
    "\n",
    "        if model == 'ESRGAN':\n",
    "            artist.set_facecolor('lightgreen')\n",
    "\n",
    "    # Set titles and labels\n",
    "    ax.set_title(f'Diagrama de Caixa da {metric}')\n",
    "    ax.set_xlabel('Modelo')\n",
    "    ax.set_ylabel('Valor')\n",
    "\n",
    "# Adjust layout\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample plot for visual comparision\n",
    "\n",
    "This script visualizes the super-resolved images generated by various models (DAT, ESRGAN, HAT, SRCNN, and SwinIR) alongside the low-resolution observation image and the corresponding ground truth high-resolution image. The process involves loading images from pre-specified directories, displaying them in a side-by-side comparison, and saving these visualizations as high-resolution PNG files.\n",
    "\n",
    "Steps:\n",
    "\n",
    "Define Paths: Paths to the observation, ground truth, and model-generated images are specified. Each model has its own directory where its super-resolved images are stored.\n",
    "Image Loading: The observation image and the ground truth image are located and loaded. Similarly, for each model, the super-resolved images corresponding to the observation are loaded.\n",
    "Visualization: Two sets of visualizations are created:\n",
    "The ground truth image is saved individually.\n",
    "The observation and model images are displayed in a 2x3 grid for comparison, with the observation image and each model’s output in its own subplot.\n",
    "Saving and Displaying: The plots are saved as PNG files with high resolution, and the second plot (observation + model images) is displayed for immediate inspection.\n",
    "This approach allows for a clear visual comparison of how different models reconstruct the super-resolved images relative to the ground truth.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "def visualize_model_comparisons(image_id):\n",
    "    \"\"\"\n",
    "    Visualizes the super-resolved images generated by different models, comparing them\n",
    "    with the observation image and the ground truth image. Saves the visualizations\n",
    "    as PNG files.\n",
    "\n",
    "    Parameters:\n",
    "    -----------\n",
    "    image_id : int\n",
    "        The ID of the image to visualize (used for both observation and ground truth images).\n",
    "\n",
    "    Process:\n",
    "    1. Load the observation image and ground truth image.\n",
    "    2. Load the super-resolved images from five different models: DAT, ESRGAN, HAT, SRCNN, and SwinIR.\n",
    "    3. Display the ground truth image individually and save it.\n",
    "    4. Create a 2x3 grid to visualize the observation image alongside the super-resolved images from each model.\n",
    "    5. Save the comparison as a PNG image and display the plot.\n",
    "    \"\"\"\n",
    "    # Define the paths to the different folders\n",
    "    observation_path = root_path+ 'Dataset/observation_test_images'\n",
    "    ground_truth_path = root_path+ 'Dataset/ground_truth_test_images'\n",
    "    model_paths = {\n",
    "        'DAT': root_path+ 'Images/DAT',\n",
    "        'ESRGAN': root_path+ 'Images/ESRGAN',\n",
    "        'HAT': root_path+ 'Images/HAT',\n",
    "        'SRCNN': root_path+ 'Images/SRCNN',\n",
    "        'SwinIR': root_path+ 'Images/SwinIR'\n",
    "    }\n",
    "\n",
    "    # Find and load the observation image\n",
    "    observation_image = os.path.join(\n",
    "        observation_path, f'observation_test_{image_id:04d}.bmp')\n",
    "    observation_img = Image.open(observation_image)\n",
    "\n",
    "    # Find and load the ground truth image\n",
    "    ground_truth_image = os.path.join(\n",
    "        ground_truth_path, f'ground_truth_test_{image_id:04d}.bmp')\n",
    "    ground_truth_img = Image.open(ground_truth_image)\n",
    "\n",
    "    # Find and load the model-generated images\n",
    "    model_images = {}\n",
    "    for model, model_path in model_paths.items():\n",
    "        model_image = os.path.join(\n",
    "            model_path, f'observation_test_{image_id:04d}.bmp')\n",
    "        model_images[model] = Image.open(model_image)\n",
    "\n",
    "    # Save the ground truth image individually\n",
    "    fig_gt, ax_gt = plt.subplots(figsize=(5, 5))\n",
    "    ax_gt.imshow(ground_truth_img)\n",
    "    ax_gt.set_title('Ground Truth', fontsize=14)\n",
    "    ax_gt.axis('off')\n",
    "    plt.savefig(f'ground_truth_image_{image_id}.png', dpi=300, bbox_inches='tight')\n",
    "    plt.close(fig_gt)  # Close the figure after saving\n",
    "\n",
    "    # Create a new figure for the remaining images (Observation + Models)\n",
    "    fig, axs = plt.subplots(2, 3, figsize=(15, 10))\n",
    "\n",
    "    # Flatten the array of axes for easier iteration\n",
    "    axs = axs.flatten()\n",
    "\n",
    "    # Define a consistent font size for titles\n",
    "    title_fontsize = 14\n",
    "\n",
    "    # Display the observation image\n",
    "    axs[0].imshow(observation_img)\n",
    "    axs[0].set_title('Observation', fontsize=title_fontsize)\n",
    "    axs[0].axis('off')\n",
    "\n",
    "    # Display the model images\n",
    "    for i, (model, image) in enumerate(model_images.items()):\n",
    "        axs[i+1].imshow(image)\n",
    "        axs[i+1].set_title(model, fontsize=title_fontsize)\n",
    "        axs[i+1].axis('off')\n",
    "\n",
    "    # Turn off any unused axes\n",
    "    for j in range(len(model_images) + 1, len(axs)):\n",
    "        axs[j].axis('off')\n",
    "\n",
    "    # Adjust spacing between the subplots\n",
    "    plt.subplots_adjust(wspace=0.1, hspace=0.3)\n",
    "\n",
    "    # Save the comparison figure with high resolution\n",
    "    plt.savefig(f'comparison_models_image_{image_id}.png', dpi=300, bbox_inches='tight')\n",
    "\n",
    "    # Show the plot for inspection\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "# Example usage\n",
    "visualize_model_comparisons(image_id=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prompt: Get the metrics for image_id=16 from metrics_results.csv\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Filter the DataFrame for image_id = 16\n",
    "image_16_metrics = df[df['Image_ID'] == 16]\n",
    "\n",
    "# Print the metrics for image_id = 16\n",
    "print(image_16_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Define metrics where higher values are better\n",
    "higher_is_better_metrics = ['PSNR', 'SSIM', 'NRQM']\n",
    "\n",
    "# Define metrics where lower values are better\n",
    "lower_is_better_metrics = ['LPIPS', 'NIQE', 'PI']\n",
    "\n",
    "# Normalize metrics\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "# Apply normalization based on whether higher or lower values are better\n",
    "for metric in higher_is_better_metrics + lower_is_better_metrics:\n",
    "    df_metric = df[df['Metric'] == metric]\n",
    "    df_metric[['Value']] = scaler.fit_transform(df_metric[['Value']])\n",
    "\n",
    "    if metric in lower_is_better_metrics:\n",
    "        df_metric[['Value']] = 1 - df_metric[['Value']]\n",
    "\n",
    "    df.update(df_metric)\n",
    "\n",
    "# Group by 'Model' and 'Metric' and calculate the mean of 'Value'\n",
    "aggregated_df = df.groupby(['Model', 'Metric'])['Value'].mean().reset_index()\n",
    "\n",
    "# Pivot the DataFrame to have 'Metric' as columns and 'Model' as rows\n",
    "pivot_df = aggregated_df.pivot(index='Metric', columns='Model', values='Value')\n",
    "\n",
    "# Calculate the difference between ESRGAN and Observation for each metric\n",
    "pivot_df['diff'] = pivot_df['ESRGAN'] - pivot_df['Observation']\n",
    "\n",
    "# Calculate the sum of the differences for each image ID\n",
    "df['diff_sum'] = df.groupby('Image_ID')['Value'].transform(\n",
    "    lambda x: x.sum() if df['Model'].iloc[0] == 'ESRGAN' else -x.sum())\n",
    "\n",
    "# Sort the IDs by 'diff_sum' and get the second-largest difference\n",
    "sorted_diff_df = df.groupby('Image_ID')['diff_sum'].sum().reset_index()\n",
    "sorted_diff_df = sorted_diff_df.sort_values(by='diff_sum', ascending=False)\n",
    "\n",
    "# Get the second-largest difference\n",
    "second_max_diff_id = sorted_diff_df.iloc[1]['Image_ID']\n",
    "\n",
    "print(f\"The ID with the second-largest difference is: {second_max_diff_id}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "def visualize_model_comparisons(image_id):\n",
    "    \"\"\"\n",
    "    Visualizes the super-resolved images generated by different models, comparing them\n",
    "    with the observation image and the ground truth image. Saves the visualizations\n",
    "    as PNG files.\n",
    "\n",
    "    Parameters:\n",
    "    -----------\n",
    "    image_id : int\n",
    "        The ID of the image to visualize (used for both observation and ground truth images).\n",
    "\n",
    "    Process:\n",
    "    1. Load the observation image and ground truth image.\n",
    "    2. Load the super-resolved images from five different models: DAT, ESRGAN, HAT, SRCNN, and SwinIR.\n",
    "    3. Display the ground truth image individually and save it.\n",
    "    4. Create a 2x3 grid to visualize the observation image alongside the super-resolved images from each model.\n",
    "    5. Save the comparison as a PNG image and display the plot.\n",
    "    \"\"\"\n",
    "    # Define the paths to the different folders\n",
    "    observation_path = root_path+'Dataset/observation_test_images'\n",
    "    ground_truth_path = root_path+'Dataset/ground_truth_test_images'\n",
    "    model_paths = {\n",
    "        'DAT': root_path+'Images/DAT',\n",
    "        'ESRGAN': root_path+'Images/ESRGAN',\n",
    "        'HAT': root_path+'Images/HAT',\n",
    "        'SRCNN': root_path+'Images/SRCNN',\n",
    "        'SwinIR': root_path+'Images/SwinIR'\n",
    "    }\n",
    "\n",
    "    # Find and load the observation image\n",
    "    observation_image = os.path.join(\n",
    "        observation_path, f'observation_test_{image_id:04d}.bmp')\n",
    "    observation_img = Image.open(observation_image)\n",
    "\n",
    "    # Find and load the ground truth image\n",
    "    ground_truth_image = os.path.join(\n",
    "        ground_truth_path, f'ground_truth_test_{image_id:04d}.bmp')\n",
    "    ground_truth_img = Image.open(ground_truth_image)\n",
    "\n",
    "    # Find and load the model-generated images\n",
    "    model_images = {}\n",
    "    for model, model_path in model_paths.items():\n",
    "        model_image = os.path.join(\n",
    "            model_path, f'observation_test_{image_id:04d}.bmp')\n",
    "        model_images[model] = Image.open(model_image)\n",
    "\n",
    "    # Save the ground truth image individually\n",
    "    fig_gt, ax_gt = plt.subplots(figsize=(5, 5))\n",
    "    ax_gt.imshow(ground_truth_img)\n",
    "    ax_gt.set_title('Ground Truth', fontsize=14)\n",
    "    ax_gt.axis('off')\n",
    "    plt.savefig(f'ground_truth_image_{image_id}.png', dpi=300, bbox_inches='tight')\n",
    "    plt.close(fig_gt)  # Close the figure after saving\n",
    "\n",
    "    # Create a new figure for the remaining images (Observation + Models)\n",
    "    fig, axs = plt.subplots(2, 3, figsize=(15, 10))\n",
    "\n",
    "    # Flatten the array of axes for easier iteration\n",
    "    axs = axs.flatten()\n",
    "\n",
    "    # Define a consistent font size for titles\n",
    "    title_fontsize = 14\n",
    "\n",
    "    # Display the observation image\n",
    "    axs[0].imshow(observation_img)\n",
    "    axs[0].set_title('Observation', fontsize=title_fontsize)\n",
    "    axs[0].axis('off')\n",
    "\n",
    "    # Display the model images\n",
    "    for i, (model, image) in enumerate(model_images.items()):\n",
    "        axs[i+1].imshow(image)\n",
    "        axs[i+1].set_title(model, fontsize=title_fontsize)\n",
    "        axs[i+1].axis('off')\n",
    "\n",
    "    # Turn off any unused axes\n",
    "    for j in range(len(model_images) + 1, len(axs)):\n",
    "        axs[j].axis('off')\n",
    "\n",
    "    # Adjust spacing between the subplots\n",
    "    plt.subplots_adjust(wspace=0.1, hspace=0.3)\n",
    "\n",
    "    # Save the comparison figure with high resolution\n",
    "    plt.savefig(f'comparison_models_image_{image_id}.png', dpi=300, bbox_inches='tight')\n",
    "\n",
    "    # Show the plot for inspection\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "# Example usage\n",
    "visualize_model_comparisons(image_id=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Define metrics where higher values are better\n",
    "higher_is_better_metrics = ['PSNR', 'SSIM', 'NRQM']\n",
    "\n",
    "# Define metrics where lower values are better\n",
    "lower_is_better_metrics = ['LPIPS', 'NIQE', 'PI']\n",
    "\n",
    "# Normalize metrics\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "# Apply normalization based on whether higher or lower values are better\n",
    "for metric in higher_is_better_metrics + lower_is_better_metrics:\n",
    "    df_metric = df[df['Metric'] == metric]\n",
    "    df_metric[['Value']] = scaler.fit_transform(df_metric[['Value']])\n",
    "\n",
    "    if metric in lower_is_better_metrics:\n",
    "        df_metric[['Value']] = 1 - df_metric[['Value']]\n",
    "\n",
    "    df.update(df_metric)\n",
    "\n",
    "# Group by 'Model' and 'Metric' and calculate the mean of 'Value'\n",
    "aggregated_df = df.groupby(['Model', 'Metric'])['Value'].mean().reset_index()\n",
    "\n",
    "# Pivot the DataFrame to have 'Metric' as columns and 'Model' as rows\n",
    "pivot_df = aggregated_df.pivot(index='Metric', columns='Model', values='Value')\n",
    "\n",
    "# Calculate the difference between each model and the 'Observation' model for each metric\n",
    "for model in pivot_df.columns:\n",
    "    if model != 'Observation':\n",
    "        pivot_df[f'diff_{model}'] = pivot_df[model] - pivot_df['Observation']\n",
    "\n",
    "# Calculate the sum of the absolute differences for each image ID\n",
    "df['diff_sum'] = df.groupby('Image_ID').apply(lambda x: (\n",
    "    x['Value'] - x[x['Model'] == 'Observation']['Value'].mean()).abs().sum()).reset_index(level=0, drop=True)\n",
    "\n",
    "# Find the ID with the smallest difference\n",
    "min_diff_id = df.loc[df['diff_sum'].idxmin(), 'Image_ID']\n",
    "\n",
    "print(f\"The ID with the smallest difference is: {min_diff_id}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "from PIL import Image\n",
    "\n",
    "\n",
    "def visualize_model_comparisons(image_id):\n",
    "    \"\"\"\n",
    "    Visualizes the super-resolved images generated by different models, comparing them\n",
    "    with the observation image and the ground truth image. Saves the visualizations\n",
    "    as PNG files.\n",
    "\n",
    "    Parameters:\n",
    "    -----------\n",
    "    image_id : int\n",
    "        The ID of the image to visualize (used for both observation and ground truth images).\n",
    "\n",
    "    Process:\n",
    "    1. Load the observation image and ground truth image.\n",
    "    2. Load the super-resolved images from five different models: DAT, ESRGAN, HAT, SRCNN, and SwinIR.\n",
    "    3. Display the ground truth image individually and save it.\n",
    "    4. Create a 2x3 grid to visualize the observation image alongside the super-resolved images from each model.\n",
    "    5. Save the comparison as a PNG image and display the plot.\n",
    "    \"\"\"\n",
    "    # Define the paths to the different folders\n",
    "    observation_path = root_path+'Dataset/observation_test_images'\n",
    "    ground_truth_path = root_path+'Dataset/ground_truth_test_images'\n",
    "    model_paths = {\n",
    "        'DAT': root_path+'Images/DAT',\n",
    "        'ESRGAN': root_path+'Images/ESRGAN',\n",
    "        'HAT': root_path+'Images/HAT',\n",
    "        'SRCNN': root_path+'Images/SRCNN',\n",
    "        'SwinIR': root_path+'Images/SwinIR'\n",
    "    }\n",
    "\n",
    "    # # Define the paths to the different folders\n",
    "    # observation_path = f'{path}/Dataset/observation_test_images'\n",
    "    # ground_truth_path = f'{path}/Dataset/ground_truth_test_images'\n",
    "    # model_paths = {\n",
    "    #     'DAT': f'{path}/Images/Observation/DAT',\n",
    "    #     'ESRGAN': f'{path}/Images/Observation/ESRGAN',\n",
    "    #     'HAT': f'{path}/Images/Observation/HAT',\n",
    "    #     'SRCNN': f'{path}/Images/Observation/SRCNN',\n",
    "    #     'SwinIR': f'{path}/Images/Observation/SwinIR'\n",
    "    # }\n",
    "\n",
    "    # Find and load the observation image\n",
    "    observation_image = os.path.join(\n",
    "        observation_path, f'observation_test_{image_id:04d}.bmp')\n",
    "    observation_img = Image.open(observation_image)\n",
    "\n",
    "    # Find and load the ground truth image\n",
    "    ground_truth_image = os.path.join(\n",
    "        ground_truth_path, f'ground_truth_test_{image_id:04d}.bmp')\n",
    "    ground_truth_img = Image.open(ground_truth_image)\n",
    "\n",
    "    # Find and load the model-generated images\n",
    "    model_images = {}\n",
    "    for model, model_path in model_paths.items():\n",
    "        model_image = os.path.join(\n",
    "            model_path, f'observation_test_{image_id:04d}.bmp')\n",
    "        model_images[model] = Image.open(model_image)\n",
    "\n",
    "    # Save the ground truth image individually\n",
    "    fig_gt, ax_gt = plt.subplots(figsize=(5, 5))\n",
    "    ax_gt.imshow(ground_truth_img)\n",
    "    ax_gt.set_title('Ground Truth', fontsize=14)\n",
    "    ax_gt.axis('off')\n",
    "    plt.savefig(f'ground_truth_image_{image_id}.png', dpi=300, bbox_inches='tight')\n",
    "    plt.close(fig_gt)  # Close the figure after saving\n",
    "\n",
    "    # Create a new figure for the remaining images (Observation + Models)\n",
    "    fig, axs = plt.subplots(2, 3, figsize=(15, 10))\n",
    "\n",
    "    # Flatten the array of axes for easier iteration\n",
    "    axs = axs.flatten()\n",
    "\n",
    "    # Define a consistent font size for titles\n",
    "    title_fontsize = 14\n",
    "\n",
    "    # Display the observation image\n",
    "    axs[0].imshow(observation_img)\n",
    "    axs[0].set_title('Observation', fontsize=title_fontsize)\n",
    "    axs[0].axis('off')\n",
    "\n",
    "    # Display the model images\n",
    "    for i, (model, image) in enumerate(model_images.items()):\n",
    "        axs[i+1].imshow(image)\n",
    "        axs[i+1].set_title(model, fontsize=title_fontsize)\n",
    "        axs[i+1].axis('off')\n",
    "\n",
    "    # Turn off any unused axes\n",
    "    for j in range(len(model_images) + 1, len(axs)):\n",
    "        axs[j].axis('off')\n",
    "\n",
    "    # Adjust spacing between the subplots\n",
    "    plt.subplots_adjust(wspace=0.1, hspace=0.3)\n",
    "\n",
    "    # Save the comparison figure with high resolution\n",
    "    plt.savefig(f'comparison_models_image_{image_id}.png', dpi=300, bbox_inches='tight')\n",
    "\n",
    "    # Show the plot for inspection\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "# Example usage\n",
    "visualize_model_comparisons(image_id=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prompt: Get the metrics for image_id=1 from metrics_results.csv\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Filter the DataFrame for image_id = 1\n",
    "image_1_metrics = df[df['Image_ID'] == 1]\n",
    "\n",
    "# Print the metrics for image_id = 1\n",
    "print(image_1_metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(root_path+'metrics_results.csv')\n",
    "\n",
    "# Define metrics where higher values are better\n",
    "higher_is_better_metrics = ['PSNR', 'SSIM', 'NRQM']\n",
    "\n",
    "# Define metrics where lower values are better\n",
    "lower_is_better_metrics = ['LPIPS', 'NIQE', 'PI']\n",
    "\n",
    "# Normalize metrics\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "# Apply normalization based on whether higher or lower values are better\n",
    "for metric in higher_is_better_metrics + lower_is_better_metrics:\n",
    "    df_metric = df[df['Metric'] == metric]\n",
    "    df_metric[['Value']] = scaler.fit_transform(df_metric[['Value']])\n",
    "\n",
    "    if metric in lower_is_better_metrics:\n",
    "        df_metric[['Value']] = 1 - df_metric[['Value']]\n",
    "\n",
    "    df.update(df_metric)\n",
    "\n",
    "# Create a DataFrame to store whether Observation is better for each ID\n",
    "better_observation_df = pd.DataFrame()\n",
    "\n",
    "# Iterate over each unique Image_ID\n",
    "for image_id in df['Image_ID'].unique():\n",
    "    # Filter data for the current Image_ID\n",
    "    id_df = df[df['Image_ID'] == image_id]\n",
    "\n",
    "    # Create a pivot table to compare Observation with other models\n",
    "    pivot_df = id_df.pivot(index='Metric', columns='Model', values='Value')\n",
    "\n",
    "    if 'Observation' in pivot_df.columns:\n",
    "        # Check if Observation is better than all other models for each metric\n",
    "        all_better = True\n",
    "        for model in pivot_df.columns:\n",
    "            if model != 'Observation':\n",
    "                # Check if Observation is better than this model for all metrics\n",
    "                if not ((pivot_df['Observation'] >= pivot_df[model]).all() or\n",
    "                        (pivot_df['Observation'] <= pivot_df[model]).all()):\n",
    "                    all_better = False\n",
    "                    break\n",
    "\n",
    "        # Record the result\n",
    "        better_observation_df = pd.concat([better_observation_df, pd.DataFrame(\n",
    "            {'Image_ID': [image_id], 'Observation_Better': [all_better]})], ignore_index=True)\n",
    "\n",
    "# Find IDs where Observation is better than all models\n",
    "better_observation_ids = better_observation_df[better_observation_df['Observation_Better']]['Image_ID']\n",
    "\n",
    "print(f\"IDs where Observation wins against all models: {list(better_observation_ids)}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}