2019-2020 Spring CSE4097 - Graduation Project I
3 different blink detection methods are examined in the scope of graduation thesis as a side project. We were planning to uses these methods to extract blink-based features, but we gave up on these later.
-
simple thresholding: Eye-aspect-ratio based model with constant thresholds which defines a proper blink behaviour.
-
adaptive model: Same approach above with adaptive thresholds.
-
machine learning model: Completely different experiment with a machine learning approach.
-
Basic model is an implementation of paper by Kazemi and Sullivan (2014). It uses Dlib's "get_frontal_face_detector" (HoG and SVM) to detect faces and "shape_predictor" (ensemble of regression trees) to detect facial landmarks. Then it uses EAR formula to get eye openness.
-
After calculating EAR values, blinks are detected by using 3 constant thresholds:
# eye aspect ratio to indicate blink EAR_THRESHOLD = 0.21 # number of consecutive frames the eye must be below the threshold EAR_CONSEC_FRAMES = 3 # how many frames we should skip at the beggining SKIP_FIRST_FRAMES = 0 -
Building a pipeline to read videos and annotations of TalkingFace dataset, to evaluate the simple model on it and to write / to read output files.
-
Analyzing blink detection problem with timeseries approach. The goal is to make 3 thresholds adaptive to any situation.
-
To make the threshold SKIP_FIRST_FRAMES adaptive, converging slope of linear fitting line is used.
-
Instead of using a constant value for EAR_THRESHOLD, outlier detection is used. 3 different methods IQR, confidence intervals and limiting z values are experimented.
-
To overcome the issue with false blinks while yawning and smiling, Error of EAR-EWMA (exponentially weighted moving average) is used insted of direct usage of EAR values in outlier detection.
-
For the third threshold EAR_CONSEC_FRAMES, the relation between significant values of PACF plot and blink durations, is examined.
-
Stationarity analysis, Dickey-Fuller test, rolling mean, exponentially decaying and time-shifting and confidence intervals are used to build a handcrafted function to estimate significant values of PACF.
-
Another method, grid search with ARIMA, is tested. Best parameters of ARIMA is found then p value is chosen as significant value of PACF.
-
Other experiments on exploring RSI (Relative Strength Indicator), CasualImpact.
-
After all of these experiments, the most efficient adaptive model is decided as a pipeline of
EWMA + outlier detection with z values + guessing significant points of PACF.
- For this part Eyeblink8 dataset is processed to get EAR values and annotations to train ML models in classification phase.
-
For this part, the ML model described in the paper by Soukupova and Cech (2016) is implemented. Output dataframes of part-1 is used. (n x 13) matrix of training set is constructed. Columns are defining
13 frame window (6 previous frames + current frame + 6 next frames)
Then the model is evaluated on same test set (TalkingFace).
-
Another SVM implementation by https://github.com/rezaghoddoosian/Early-Drowsiness-Detection is tested and compared with our results. But it was an old version of SVC (scikit<0.18) and its predict() method gives error so manual implementation of rbf prediction method is developed. (see https://stackoverflow.com/questions/28593684/replication-of-scikit-svm-srv-predictx)
- Comparison of 4 models: simple model, adaptive model, ml model, ml model from RLDD.
- Utility functions based on "eye_blink_detection_1_simple_model.ipynb".
- Utility functions based on "eye_blink_detection_2_adaptive_model.ipynb".
- Utility functions based on "eye_blink_detection_3_ml_model_part1" and "eye_blink_detection_3_ml_model_part2".