Here's slides for Shape from Shading algorithms introduction and the IO and design choice of evaluation process: https://docs.google.com/presentation/d/1orHIz7jBYpMmg4C48Fh-Rspw64hOq1gSVDNbC3nZbG0/edit?usp=sharing
Our pipeline contains threes steps:
- dataset registry
- codebase registry
- evaluation
First of all you need to download the dataset and code you try to evaluate.
You have to prepare the dataset by preprocessing the data into supported format(.mat file or image file). And you need to write a API file for the code s.t the tool could recognize the codebase and dataset.
The first two steps should be done manually. But in the end, you should register the them using the sfs dsm --register
and sfs cbm --register
commands provided by the tool.
Once you've register the legal dataset and codebase, you can view them and evaluating them using the sfs run
command.
Each time you run the run
command, If the codebase is bug-free, the evaluation result will be saved under metrics_result
folder in the format of excel file combined with image filess.
You can always add new dataset and codebase, just remember to register them.
You need two things:
- A unix-based OS
- updated version of Matlab installed
Another thing strongly suggested:
- A Python>3.6 virtual environment
You can use anaconda/venv/virtualenv or similar tools, but here's a easy example, type them in the terminal:
python3 -m venv venv
source ./venv/bin/activate
Then you need to open the Matlab and type matlabroot
in the console. Copy and paste the whole path, it might looks like this:
'/Applications/MATLAB_R2022b.app'
Now run the script: bash install.sh path_to_matlabroot
Notice it will prompt you for a path to Matlab root. Follow the guide of prompt
If you want to look at example code/dataset/metrics result, here's link: https://github.com/HendrickZhou/SFSEVAL_example_dataset_codebase
You need to preprocess the data into either image file or .mat file with your own effort.
And your next goal is to write a descriptor.json
file inside a new dataset folder's root, providing all the important information about this dataset. Name your dataset properly, and move them into the sfsevaluator/dataset
folder in this repository. A legal structure:
\sfsevaluator
\sfseval
\codebase
\dataset
\my_new_dataset_no_1
- descriptor.json
- other_data_files_in_this_dataset
\my_new_dataset_no_2
- descriptor.json
- other_data_files_in_this_dataset
- misc_files
After you're done with the json file and move it to the right location, next run this command:
sfs dsm --register your_new_dataset_name
Now you should be able to view all registered dataset with this command:
sfs dsm --list
Here's an example:
[
{
"image": {
"img": "frame-000001.color.png",
"mask": "frame-000001.mask.png",
"pose": "frame-000001.pose.txt"
},
"intrinsic": {
"type":"mat",
"path":"colorIntrinsics.mat"
},
"ground_truth": {
"type": null,
"path": null
},
"shape_prior": {
"type": "png",
"path": "frame-000001.depth.png"
},
"light": null
},
{
"image": {
"img": "frame-000002.color.png",
"mask": null,
"pose": "frame-000002.pose.txt"
},
"intrinsic": {
"type":"mat",
"path":"colorIntrinsics.mat"
},
"ground_truth": {
"type": null,
"path": null
},
"shape_prior": {
"type": "png",
"path": "frame-000002.depth.png"
},
"light": null
}
]
This is a dataset contains two images, each of the image is a json object in a json array. And the name-value pairs contains the information of each important data input for a SFS algorithm.
You can use this one as an example, but here's several rules for this file:
- if the dataset doesn't contains certain data, like ground truth data, that fields should be null. But write it like this:
"ground_truth": {
"type": null,
"path": null
}
DONNOT write it like this:
"ground_truth": null
AKA you still need to list all the fields, and mark each field as null.
-
The
path
field should contain a RELATIVE path to the data in your folder! -
The
type
field only support two types of data:mat
and image typepng
,jpg
,tiff
etc. But basically, if it's notmat
type, it will be recognized as image type, so you can also writeimg
in this field for an image. -
The
light
component is by default, a.mat
file, since it only should be a 9x3 or 9x1 vector(Not 3x9 1x9!). A use example:
"light":"L.mat"
-
The 'pose' field in
image
field is actually useless, don't expect it to do anything. -
EACH
.mat
FILE SHOULD ONLY CONTAINS ONE VARIABLE! Don't squeeze multiple matlab variable into one.mat
file, the tool will only read the FIRST variable in the.mat
file! Remember this when you're preprocessing the dataset.
At this point, we only support algorithm writte in MATLAB!! But other languages could be supported with some effort.
Similar to database management, download the code, write an API file, move it to /codebase
folder, and register it with:
sfs cbm --register sfs_codebase_name
.
function output = SFS_API(img, mask, light, intrinsic, shape_prior)
% img: a matrix of 3/1d, type double
% mask: a matrix of 1d, type double
% shape_prior: a matrix of 1d, type double, or NAN; optional
% intrinsic: a matrix of 3*3, type double
% light: a matrix of 9*1/9*3, type double, or NAN
% output is a structure with: depth, normal
- if a dataset doesn't provide camera intrisince data, the
intrinsic
argument of your api should expect a 3x3 identity matrix produced by the tool. output.depth
should be a 2d matrix!,ouput.normal
should be a 3d matrix.- If you need to use different hyperparameters, modify it in your own API file!
Once all done with the register, run the command
sfs run --ds_name your_dataset_name --cb_name your_codebase_name --tag your_tags_to_this_evluation
or you can run:
sfs run --ds_id your_dataset_id --cb_id your_codebase_id --tag your_tags_to_this_evluation
The id can be found by running sfs cbm --list
and sfs dsm --list
commands.
If you're running on a dataset with multiple images, a optional argument --seq_idx
can be used to indicate which image you want to use(starting from 0). If not provided, by default the idx is 0. Example:
sfs run --ds_id your_dataset_id --cb_name your_codebase_name --seq_idx 1
The purpose of tag is to distinguish the evaluation with different hyperparameters. For example, you can run an algorithm with some virtual hyperparameter a=1. Next time you want to run it with a=2. You can use --tag a_1
and --tag a_2
for example.
If the dataset doesn't provide ground truth data, of course no quantitative metrics can be calculated, only depth image and normal will be saved. Otherwise, two metrics will be calculated by default: MAE & RMSE.
Result will be saved in a tabular format(excel), and image data cell has a hyperlink to the image saved in correpsonding image folder.
When you run into trouble, you need to identify what's on this program's side and what's not. Typical user side errors:
- The dataset is not correctly constructed.
- The API for the algorithm is not bug free
- The original code is not bug free, or this algorithm can't hanlde certain type of input data.
Most of the time bugs come from the wrong data format, the debug method is to add log or print information on your API file or their code to see what's wrong.
If you're certain that the bugs don't come from your side of work, here's the hack to debugging the our code:
In sfseval/run.py
around line 81, add this line: import pdb;pdb.set_trace()
and check it in pdb!
If you want to add more types of metric calculation, modify these places in /sfseval/cal_metrics.py
:
- Add your new field in
MetricDataObject: __init__
function - Add new field to END of
MetricsDB: heading
class variable(line 40) - Add two new line in
MetricsDB: get_metrics
function. And add a new function in this file for calculating your metrics (see what MAE did)