Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Version 3.0 #39

Merged
merged 47 commits into from
Oct 14, 2019
Merged

Version 3.0 #39

merged 47 commits into from
Oct 14, 2019

Conversation

Tobias-Fischer
Copy link
Owner

@Tobias-Fischer Tobias-Fischer commented Oct 7, 2019 •
edited
Loading

This pull request contains several major changes:

  • Use 3DDFA landmark extractor (https://github.com/cleardusk/3DDFA) as opposed to https://github.com/1adrianb/face-alignment because of the increased speed (note that we change the code so that Python 2 is still supported for use in ROS)
  • Refactor the code to disentangle ROS-independent code from ROS-specific code
  • Provide a standalone (non-ROS) version of the code
  • Provide helper script to download all required model files (this was done using ROS catkin scripts before)
  • Improve inference speed by doing inference in batches rather than per subject for landmark extraction as well as gaze estimation.
  • Run ensembles in parallel, so that running the 4-model ensemble is now very fast, too.
  • Tensorflow 2.0 compatibility

TODO before merging:

  • Update README to include the new libraries; make sure their LICENSE is included
  • Make sure 3DDFA works as well as the previous landmark extraction method
  • Run this branch on one of our wheelchairs to make sure that the changes are not breaking with the demos/research that is ongoing.
  • Decide whether the eye-blink work should be merged alongside this branch, or release a version with eye-blink support later (@twarz / Incorporate blink estimation #35) <- This will be merged separately.

TODO after merging:

Massive thanks to @ngageorange for a lot of input.

ahmed-alhindawi and others added 30 commits September 24, 2019 12:13
@ahmed-alhindawi
Moved to using 3DDFA landmark extractor as it's faster
958af7f 
Removed timings
@ahmed-alhindawi
Merge branch 'master' into modularised
169efb8
@Tobias-Fischer
Update README.md
0310800
@Tobias-Fischer
Python2 compatibility, add 3DDFA to list of libs
ea06a45
@Tobias-Fischer
Merge branch 'master' into modularised
68bea4f
@Tobias-Fischer
Remove duplicated code in trackers
4327acd
@Tobias-Fischer
Some cleanup
cd9ad83
@Tobias-Fischer
Further cleanup
6472831
@Tobias-Fischer
Add second pass with refined bounding box
abfe23d
@Tobias-Fischer
Rename tracker files, more cleanup
9accc6f
@Tobias-Fischer
Continued cleanup
4af2d8c
@Tobias-Fischer
Cleanup of estimate_gaze.py, make some methods static for easy use ou…
864a1fc 
…tside the class
@Tobias-Fischer
Make some methods in extract_landmarks static for easier use outside
6c1d6d0
@Tobias-Fischer
Bugfix: resize head_pose_image before visualizing rather than after
ffe9005
@Tobias-Fischer
Add script to estimate gaze from images + sample images
209b063
@Tobias-Fischer
Moving imports in specific functions to allow running in non-ROS env
4eb296c
@Tobias-Fischer
Split estimate_gaze into non-ROS (base) and ROS part
2006fb1
@Tobias-Fischer
Split extract_landmarks into non-ROS (base) and ROS part
1a724a3
@Tobias-Fischer
Simplify estimate_gaze_standalone to use restructured base classes
763faa8
@Tobias-Fischer
Remove last few references to ROS in non-ROS version
2ac4591
@Tobias-Fischer
Add README file for standalone script, and helper script to download …
16783a9 
…models
@Tobias-Fischer
Remove unnecessary dependencies, allow running standalone script from…
9f1be40 
… any path
@Tobias-Fischer
Minor fix
7896b50
@Tobias-Fischer
Update README.md
78503a4
@ahmed-alhindawi
batch everything commit:
6d7f213 
- batch landmark extraction onto GPU
- batch eye gaze estimation for multiple subjects
- modified ensembles to run on GPU in parallel using Keras (now run almost as quickly as single model file)
- various small bug fixes
@ahmed-alhindawi
spelling fixes
8aa1bdb
@ahmed-alhindawi
Removed face_alignment as a dependency and imported S3FD as a facedet…
5f1af23 
…ector within RT-GENE

Made the face-encoding face tracker the default if no "use_face_encoding_tracker" variable is declared
@ahmed-alhindawi
created SFD directory
bcc5be9
@Tobias-Fischer
Add additional model files to gitignore
c7fd338
@Tobias-Fischer
Update README files, mostly related to licenses
82e97df
Tobias-Fischer and others added 12 commits October 8, 2019 10:39
@Tobias-Fischer
Add LICENSE files and license information to each file
9cbbee0
@Tobias-Fischer
Minor fixes
92d363a
@ahmed-alhindawi
Fundamentally, this commit changes the SolvePnP from an iterative pro…
332c894 
…cess to DLS which facilitates the removal of the ExtrinsicGuess and associated variables

Also removed link/non-rotated-link setup, instead deferring it to the camera setup process
General refactoring and small bug-fixes
@ahmed-alhindawi
changed default Model_Size parameter to a meaningful value given our …
9646c2e 
…new pose estimator
@Tobias-Fischer
Tensorflow 2 compatibility (using upgrade script) - Fix #40
92f9a1d
@Tobias-Fischer
Further fixes for TF2, change keras -> tensorflow.keras
15e4d6f
@Tobias-Fischer
Batch inference for standalone RT-GENE version
7e7688c
@Tobias-Fischer
Update readme to reflect we are now using tf.keras
1c1ea6d
@Tobias-Fischer
Fixes for TF compatibility
b982e34
@Tobias-Fischer
Use single model by default
8d1fcd2
@Tobias-Fischer
Cleanup
0d058cd
@Tobias-Fischer
interpupillary distance not used in launch anymore, add face_encoding…
7da647c 
…_threshold to launch file
@Tobias-Fischer
Copy link
Owner Author

Tobias-Fischer commented Oct 10, 2019 •
edited
Loading

Breaking changes:

  • rename rgb_frame_id to ros_tf_frame in launch file, publisher is now in start_webcam.launch
  • interpupillary_distance is now set via dynamic_reconfigure rather than in the launch file
  • SequentialTracker is deprecated; to achieve a similar behavior use FaceEncodingTracker with a large face_encoding_threshold (see launch file)

Tobias-Fischer and others added 5 commits October 10, 2019 15:02
@Tobias-Fischer
Update start_webcam.launch
2ba0ab4
@ahmed-alhindawi
cleaned estimate_gaze.launch
d96e8c2 
made the visualisation of the pose frames optional; setting visualise_headpose/visualise_eyepose to False gains ~10% performance improvement
@ahmed-alhindawi
exposed visualisation parameters into the launch file
fd4239e
@Tobias-Fischer
Sanity check added
4f5f13a
@Tobias-Fischer
Move ROS related landmark class into scripts folder
22dbc8c
@Tobias-Fischer Tobias-Fischer merged commit 43f6c47 into master Oct 14, 2019
@Tobias-Fischer Tobias-Fischer deleted the modularised branch October 14, 2019 09:03
@sahilrider
Copy link

sahilrider commented Oct 15, 2019 •
edited
Loading

Hey, What's the eye-blink work?

@Tobias-Fischer
Copy link
Owner Author

Hi @sahilrider,
We will integrate a blink detection algorithm that we will present at the ICCV2019 gaze workshop (https://gazeworkshop.github.io/#accepted). Please check back in the following weeks for more details.

Best, Tobias

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Provide standalone (non-ROS) script Improve computational speed for multi person estimation
3 participants
@Tobias-Fischer @sahilrider @ahmed-alhindawi