Transfer Learning for Anime Characters

Warning: This repository size is quite big (approx. 100 MB) since it includes training and test images.

Introduction

This repository is the continuation of Flag #15 - Image Recognition for Anime Characters.

In Flag #15, we can see that Transfer Learning works really well with 3 different anime characters: Nishikino Maki, Kotori Minami, and Ayase Eli.

In this experiment, we will try to push Transfer Learning further, by using 3 different anime characters which have hair color similarity: Nishikino Maki, Takimoto Hifumi, and Sakurauchi Riko.

This experiment has 3 main steps:

1. Utilize lbpcascade_animeface to recognize character face from each images
2. Resize each images to 96 x 96 pixels
3. Split images into training & test before creating the final model

raw directory contains 36 images for each characters (JPG & PNG format). The first 30 images are used for training while the last 6 images are used for test.

As an example, we got the following result after applying Step 1 (cropped directory is shown at the right side):

lbpcascade_animeface can detect character faces with an accuracy of around 83%. Failed images are stored in raw (unrecognized) for future improvements.

Since we have 3 characters and 6 test images for each which are not part of training, resized_for_test contains 18 images in total. Surprisingly, almost all characters are detected properly!

Update (Nov 13, 2017): See animeface-2009 section below, which push face detection accuracy to 93%.

Requirements

• OpenCV (https://github.com/opencv/opencv)
• TensorFlow (https://github.com/tensorflow/tensorflow)

Steps

1. The following command is used to populate cropped directory.

$ python bulk_convert.py raw/[character_name] cropped

2. The following command is used to populate resized_for_training & resized_for_test directory.

$ python bulk_resize.py cropped/[character_name] resized

After running the step above, you can decide how many images will be used in resized_for_training and how many images will be used in resized_for_test.

3. Re-train the Inception model by using transfer learning:

$ bazel-bin/tensorflow/examples/image_retraining/retrain --image_dir ~/transfer-learning-anime/resized_for_traning/
$ bazel build tensorflow/examples/image_retraining:label_image

4. At this point, the model is ready to use. We can run the following command to get the classification result:

$ bazel-bin/tensorflow/examples/image_retraining/label_image --graph=/tmp/output_graph.pb --labels=/tmp/output_labels.txt --output_layer=final_result:0 --image=$HOME/transfer-learning-anime/resized_for_test/[character name]/[image name]

If everything works properly, you will get the classification result. See TensorFlow Documentation for more options.

Optionally, sample model can be downloaded by running download_model.sh script inside models (example) directory.

Result Analysis

Initially, we run the experiment with 2 characters: Nishikino Maki and Takimoto Hifumi.

INFO:tensorflow:2017-11-10 08:50:36.151387: Step 3999: Train accuracy = 100.0%
INFO:tensorflow:2017-11-10 08:50:36.151592: Step 3999: Cross entropy = 0.002191
INFO:tensorflow:2017-11-10 08:50:36.210147: Step 3999: Validation accuracy = 100.0% (N=100)
INFO:tensorflow:Final test accuracy = 92.9% (N=14)

The result is as the following:

Image	Classification	OK/NG
	nishikino maki (score = 0.99874) takimoto hifumi (score = 0.00126)	OK
	nishikino maki (score = 0.75519) takimoto hifumi (score = 0.24481)	OK
	nishikino maki (score = 0.99513) takimoto hifumi (score = 0.00487)	OK
	nishikino maki (score = 0.98629) takimoto hifumi (score = 0.01371)	OK
	nishikino maki (score = 0.99723) takimoto hifumi (score = 0.00277)	OK
	nishikino maki (score = 0.99695) takimoto hifumi (score = 0.00305)	OK

Image	Classification	OK/NG
	takimoto hifumi (score = 0.63084) nishikino maki (score = 0.36916)	OK
	takimoto hifumi (score = 0.99728) nishikino maki (score = 0.00272)	OK
	takimoto hifumi (score = 0.99972) nishikino maki (score = 0.00028)	OK
	takimoto hifumi (score = 0.98852) nishikino maki (score = 0.01148)	OK
	takimoto hifumi (score = 0.99456) nishikino maki (score = 0.00544)	OK
	takimoto hifumi (score = 0.96630) nishikino maki (score = 0.03370)	OK

From the result above, 10 out of 12 have threshold > 0.95, while the lowest threshold is 0.63.

At this point, I decided to add Sakurauchi Riko, which is known for its similarity to Nishikino Maki.

INFO:tensorflow:2017-11-10 13:13:59.270717: Step 3999: Train accuracy = 100.0%
INFO:tensorflow:2017-11-10 13:13:59.270912: Step 3999: Cross entropy = 0.005526
INFO:tensorflow:2017-11-10 13:13:59.328139: Step 3999: Validation accuracy = 100.0% (N=100)
INFO:tensorflow:Final test accuracy = 80.0% (N=15)

With 3 similar characters, the result is as the following:

Image	Classification	OK/NG
	nishikino maki (score = 0.99352) sakurauchi riko (score = 0.00612) takimoto hifumi (score = 0.00036)	OK
	nishikino maki (score = 0.47391) sakurauchi riko (score = 0.37913) takimoto hifumi (score = 0.14696)	OK
	nishikino maki (score = 0.95976) sakurauchi riko (score = 0.02797) takimoto hifumi (score = 0.01227)	OK
	nishikino maki (score = 0.88851) sakurauchi riko (score = 0.07526) takimoto hifumi (score = 0.03623)	OK
	nishikino maki (score = 0.99025) sakurauchi riko (score = 0.00766) takimoto hifumi (score = 0.00209)	OK
	nishikino maki (score = 0.96782) sakurauchi riko (score = 0.02783) takimoto hifumi (score = 0.00435)	OK

As you can see above, the similarity between Nishikino Maki and Sakurauchi Miko starts to lower down the confidence level of the resulted model. Nevertheless, all classifications are still correct, where 4 out of 6 maintain the threshold of > 0.95.

Image	Classification	OK/NG
	takimoto hifumi (score = 0.86266) nishikino maki (score = 0.13632) sakurauchi riko (score = 0.00102)	OK
	takimoto hifumi (score = 0.87614) sakurauchi riko (score = 0.12334) nishikino maki (score = 0.00051)	OK
	takimoto hifumi (score = 0.99964) sakurauchi riko (score = 0.00023) nishikino maki (score = 0.00013)	OK
	takimoto hifumi (score = 0.99417) nishikino maki (score = 0.00472) sakurauchi riko (score = 0.00110)	OK
	takimoto hifumi (score = 0.94923) sakurauchi riko (score = 0.04842) nishikino maki (score = 0.00235)	OK
	takimoto hifumi (score = 0.96029) sakurauchi riko (score = 0.02822) nishikino maki (score = 0.01150)	OK

Interestingly, the addition of 3rd character increases the confidence level of several Takimoto Hifumi testcases (see 1st and 4th result). Overall, this character can be easily differentiated compared to the other two.

Image	Classification	OK/NG
	sakurauchi riko (score = 0.98747) takimoto hifumi (score = 0.01054) nishikino maki (score = 0.00199)	OK
	sakurauchi riko (score = 0.96840) takimoto hifumi (score = 0.02895) nishikino maki (score = 0.00265)	OK
	sakurauchi riko (score = 0.97713) nishikino maki (score = 0.02167) takimoto hifumi (score = 0.00119)	OK
	sakurauchi riko (score = 0.90159) nishikino maki (score = 0.06989) takimoto hifumi (score = 0.02852)	OK
	sakurauchi riko (score = 0.99713) takimoto hifumi (score = 0.00184) nishikino maki (score = 0.00103)	OK
	sakurauchi riko (score = 0.79957) nishikino maki (score = 0.19310) takimoto hifumi (score = 0.00733)	OK

From this experiment, it seems that the current bottleneck is located at Step 1 (face detection), which have the overall accuracy of 83% in face detection.

animeface-2009

nagadomi/animeface-2009 provides another method of face detection. 13 out of 21 unrecognized images are now recognized in cropped (unrecognized) directory.

Current found limitations: it seems the script requires more memory and slower to run compared to lbpcascade_animeface.xml.

Image	Classification	OK/NG
	nishikino maki (score = 0.99296) sakurauchi riko (score = 0.00694) takimoto hifumi (score = 0.00010)	OK
	nishikino maki (score = 0.93702) sakurauchi riko (score = 0.04017) takimoto hifumi (score = 0.02281)	OK
	nishikino maki (score = 0.99406) sakurauchi riko (score = 0.00565) takimoto hifumi (score = 0.00030)	OK

Image	Classification	OK/NG
	takimoto hifumi (score = 0.99242) nishikino maki (score = 0.00431) sakurauchi riko (score = 0.00327)	OK
	takimoto hifumi (score = 0.99596) sakurauchi riko (score = 0.00403) nishikino maki (score = 0.00001)	OK
	takimoto hifumi (score = 0.98369) sakurauchi riko (score = 0.01498) nishikino maki (score = 0.00133)	OK
	takimoto hifumi (score = 0.99796) sakurauchi riko (score = 0.00189) nishikino maki (score = 0.00015)	OK
	takimoto hifumi (score = 0.99601) nishikino maki (score = 0.00335) sakurauchi riko (score = 0.00064)	OK
	takimoto hifumi (score = 0.99960) sakurauchi riko (score = 0.00029) nishikino maki (score = 0.00011)	OK
	takimoto hifumi (score = 0.99995) nishikino maki (score = 0.00004) sakurauchi riko (score = 0.00001)	OK

Image	Classification	OK/NG
	sakurauchi riko (score = 0.84480) nishikino maki (score = 0.12101) takimoto hifumi (score = 0.03419)	OK
	sakurauchi riko (score = 0.94310) nishikino maki (score = 0.04296) takimoto hifumi (score = 0.01393)	OK
	sakurauchi riko (score = 0.96176) takimoto hifumi (score = 0.03217) nishikino maki (score = 0.00607)	OK

Since this method gives better result in detecting anime character face and classification still works with almost the same result, the overall face detection accuracy is now around 93%.

License

is created by nagadomi/lbpcascade_animeface.

Copyright for all images are owned by their respective creators.