Add Graceful Recovery Baseline Test

[bjee19]

Proposed changes

Problem: We want to have a test that checks for how well NGF recovers from container failures.

Solution: Added a manual test that checks for when the nginx-gateway container fails, when the nginx container fails, when the NGF Pod restarts due to a node restart with cleaning up of resources, and when the NGF Pod restarts due to a node restart without cleaning up of resources.

Closes #951

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[bjee19]

Would like specific feedback on the last two test cases.

• Can't think of a good name for them
• The goal was to test the behavior of the NGF Pod on node restart and the easiest way I saw to do that was to do it on a Kind cluster and restart the docker container which is running the cluster. However, I did some digging around and it looks like docker restart sends a SIGTERM signal but if the processes don't cleanup in 10ish seconds it will forcibly terminate. This forced termination is what I'm testing when I don't specifically drain and delete a node (3rd test case) and is the test that usually fails. Is it alright that I included both cases?

[kate-osborn]

Nice work! Very thorough instructions 👍

Can you create a directory under tests for this test? I know there's only one file, but I think it will be good to match the structure of the other testing PRs.

See my review for a few comments and suggestions.

[pleshakov]

Hi @bjee19 Please see a few major and minor requests and comments.
Big one - let's use SIGKILL because SIGTERM kicks in graceful, not abrupt (failure), termination.

[kate-osborn]

🚀

[pleshakov]

good stuff 👍