Merge pull request #3 from gatheluck/feature/202104/gatheluck/renew-c…

…odes

Update API

.github/workflows/ci.yaml

@@ -0,0 +1,52 @@
+name: CI
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+jobs:
+  test:
+    name: Run tests with mypy, black and pytest
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.9]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Set up Python ${{ matrix.python-version  }}
+        uses: actions/setup-python@v1
+        with:
+          python-version: ${{ matrix.python-version  }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install poetry
+          poetry install --no-interaction
+      - name: Run mypy test
+        run: poetry run mypy fhmap
+
+      - name: Run black test
+        run: poetry run black --check fhmap tests
+
+      - name: Run isort test
+        run: poetry run isort --check-only .
+
+      - name: Run flake8 test
+        run: poetry run flake8 .
+
+      - name: Run pytest with coverage check
+        run: poetry run pytest tests --cov=./fhmap --cov-report=xml
+
+      - name: Upload coverage to Codecov
+        if: ${{ matrix.python-version==3.9 }}
+        uses: codecov/codecov-action@v1
+        with:
+          file: ./coverage.xml
+          fail_ci_if_error: true

.gitignore

@@ -1,6 +1,11 @@
 __pycache__
+.coverage
+.mypy_cache
 .pytest_cache
+.python-version
+.venv
 .vscode
 .DS_Store
-logs
-data/*
+coverage.xml
+data/*
+outputs

README.md

@@ -1,69 +1,147 @@
-# FourierHeatmap
+# FourierHeatmap (latest release: v0.2.0)
+
+[![CI](https://github.com/gatheluck/FourierHeatmap/workflows/CI/badge.svg)](https://github.com/gatheluck/FourierHeatmap/actions?query=workflow%3ACI)
+[![codecov](https://codecov.io/gh/gatheluck/FourierHeatmap/branch/master/graph/badge.svg?token=17KIZNS046)](https://codecov.io/gh/gatheluck/FourierHeatmap)
 
 This is an unofficial pytorch implementation of Fourier Heat Map which is proposed in the paper, [A Fourier Perspective on Model Robustness in Computer Vision [Yin+, NeurIPS2019]](https://arxiv.org/abs/1906.08988). 
 
 Fourier Heat Map allows to investigate the sensitivity of CNNs to high and low frequency corruptions via a perturbation analysis in the Fourier domain.
 
 <img src="samples/FourierHeatmap-Teaser.png" height="300px">
 
+## News
+We release v0.2.0. API is renewed and some useful libraries (e.g. [hydra](https://hydra.cc/docs/intro/)) are added.
+
+Previous version is still available as [v0.1.0](https://github.com/gatheluck/FourierHeatmap/tree/v0.1.0).
+
 ## Requirements
+This library requires following as a pre-requisite.
+- python 3.9+
+- poetry
+
+Note that I run the code with Ubuntu 20, Pytorch 1.8.1, CUDA 11.0.
+
+## Installation
+This repo uses [poetry](https://python-poetry.org/) as a package manager. 
+The following code will install all necessary libraries under `.venv/`.
+
+```
+git clone git@github.com:gatheluck/FourierHeatmap.git
+cd FourierHeatmap
+pip install poetry  # If you haven't installed poetry yet.
+poetry install
+```
+
+## Setup
 
-You will need the following to run the codes:
-- Python 3.7+
-- PyTorch 1.4+
-- TorchVision
-- numpy
-- click
-- tqdm
-- matplotlib
-- seaborn
+### Dataset
 
-Note that I run the code with Ubuntu 18, Pytorch 1.4.0, CUDA 10.1.
+This codes expect datasets exist under `data/`. For example, if you want to evaluate Fourier Heat Map for ImageNet, please set up like follows:
+
+```
+FourierHeatmap
+├── data
+│	└── imagenet
+│		├── train/
+│		└── val/
+```
 
 ## Usage
 
-### Generating Fourier base function
+### Visualizing Fourier basis
 
-The script `fhmap/fourier_base.py`
-generates Fourier base function based on given indeces; `(i,j)`. 
-Example code:
+The script `fhmap/fourier/basis.py` generates Fourier base functions. For example:
 
 ```
-cd fhmap
-python fourier_base.py
+poetry run python fhmap/fourier/basis.py
 ```
 
-will generate 32x32 types 2D Fourier basis and save as an image under `logs/fourier_base_32x32.png`. The generated image should be like follows. 
+will generate 31x31 2D Fourier basis and save as an image under `outputs/basis.png`. The generated image should be like follows. 
 
-<img src="samples/fourier_base_32x32.png" height="300px">
+<img src="samples/basis_31x31.png" height="300px">
 
 ### Evaluating Fourier Heat Map
 
-The script `apps/eval.py`
+The script `fhmap/apps/eval_fhmap.py`
 eveluate Fourier Heat Map for a model. For example:
 
 ```
-cd apps
-python eval.py -a resnet56 -w [WEIGHT_PATH]  -d cifar10 --h_map_size 31 --w_map_size 31 -k 1 -l [LOG_DIR] --eps 32
+poetry run python fhmap/apps/eval_fhmap.py dataset=cifar10 arch=resnet56 weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0
 ```
 
-will generate 31x31 Fourier Heat Map for ResNet56 on Cifar-10 dataset and save as an image under `LOG_DIR/fhmap.png`. The generated image should be like follows. 
+will generate 31x31 Fourier Heat Map for ResNet56 on CIFAR-10 dataset and save as an image under `outputs/eval_fhmap/`. The generated image should be like follows. 
 
 <img src="samples/cifar10_resnet56_natural.png" height="300px">
 
-Note that currectly both `h_map_size` and `w_map_size`  should be odd number because of the symetry of Fourier basis.
 
-### Evaluating your model
+Note that the L2 norm size (=eps) of Fourier basis use in original paper is following:
+| dataset | eps
+---- | ----
+| CIFAR-10 | 4.0
+| ImageNet | 15.7
+
+## Evaluating custom dataset and model
+
+
+
+### Evaluating your custom dataset 
+
+If you want to evaluate Fourier Heat Map on your custom dataset, please refer follwing instraction.
 
-Currently supported architectures are as follows;
+- Implement `YourCustomDatasetStats` class: 
+	- This class holds basic dataset information.
+	- `YourCustomDatasetStats` class should inherit from original `DatasetStats` class in `factory/dataset` module and also shoud be placed in `factory/dataset` module. 
+	- For details, please refer to the `Cifar10Stats` class in `factory/dataset` module.
 
-- Alexnet
-- VGG: 16, 16 with BN, 19, 19 with BN
-- ResNet: 18, 34, 50, 56, 101, 152
-- WideResNet: 16, 28, 40
+- Implement `YourCustomDataModule` class:
+	- This class is responsible for preprocess, transform (includes adding Fourier Noise to image) and create test dataset.
+	- `YourCustomDataModule` class should inherit from `BaseDataModule` class in `factory/dataset` module and also shoud be placed in `factory/dataset` module. 
+	- For details, please refer to the `Cifar10DataModule` class in `factory/dataset` module.
 
-If you want evaluate Fourier Heat Map of your model, you can use 
-`AddFourierNoise`  class in `fhmap/fourier_heatmap.py`  like classes in `torchvision.transforms`. 
+- Implement `YourCustomDatasetConfig` class:
+	- This class is needed for applying [hydra](https://hydra.cc/)'s [dynamic object instantiation](https://hydra.cc/docs/patterns/instantiate_objects/overview) to dataset class.
+	- `YourCustomDatasetConfig` class should inherit from `DatasetConfig` class in `schema/dataset` module and also shoud be placed in `schema/dataset` module. Please add `YourCustomDatasetConfig` to `schema/__init__`.
+	- For details, please refer to the `Cifar10Config` class in `schema/dataset` module.
+
+- Add option for your custom dataset:
+	- Lastly, please add the config of your custom dataset to `ConfigStore` class by adding a follwing line to `apps/eval_fhmap`.
+
+	```
+	cs.store(group="dataset", name="yourcustomdataset", node=schema.YourCustomDatasetConfig)
+	```
+
+Now, you will be able to call your custom dataset like following.
+
+```
+poetry run python fhmap/apps/eval_fhmap.py dataset=yourcustomdataset arch=resnet50 weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0
+```
+
+### Evaluating your custom architecture (model)
+
+If you want to evaluate Fourier Heat Map on your custom architecture (model), please refer follwing instraction.
+
+- Implement `YourCustomArch` class:
+	- Please implement class or function which return your custom architecture. The custom architecture have to subclass of `torch.nn.module`.
+	- For details, please refer to the `factory/archs/resnet` module.
+
+- Implement `YourCustomArchConfig` class:
+	- This class is needed for applying [hydra](https://hydra.cc/)'s [dynamic object instantiation](https://hydra.cc/docs/patterns/instantiate_objects/overview) to architecture class.
+	- `YourCustomArchConfig` class should inherit from `ArchConfig` class in `schema/arch` module and also shoud be placed in `schema/arch` module. Please add `YourCustomArchConfig` to `schema/__init__`.
+	- For details, please refer to the `Resnet56Config` class in `schema/arch` module.
+	- If you want to use architectures which is provided by other libs like [pytorch](https://github.com/pytorch/pytorch) or [timm](https://github.com/rwightman/pytorch-image-models), please refere to the `Resnet50Config` class in `schema/arch` module.
+
+- Add option for your custom architecture:
+	- Lastly, please add the config of your custom architecture to `ConfigStore` class by adding a follwing line to `apps/eval_fhmap`.
+
+	```
+	cs.store(group="arch", name="yourcustomarch", node=schema.YourCustomArchConfig)
+	```
+
+Now, you will be able to call your custom arch like following.
+
+```
+poetry run python fhmap/apps/eval_fhmap.py dataset=cifar10 arch=yourcustomarch weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0
+```
 
 ## References