Refine HPO

[eunwoosh]

Summary

This PR includes some refinements for HPO. Details are as below:

• Terminate HPO trials when main process gets SIGINT or SIGTERM
• HPO is suspended if HPO trials exists abnormally more than three times
• Remove unused model weight after HPO is done
• Fix a bug that HPO trial process waits to receive response infinitely if main process exits abnormally
• Fix a bug that HPO trails early stopped sometimes when HPO with multi-GPU

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0