feat(tests): isolated integration tests

[pmalek]

What this PR does / why we need it:

This PR tries to introduce a concept of isolated integration tests. The basic principles of those are:

• The idea of a shared kind cluster from the traditional integration tests is retained
• kubernetes-sigs/e2e-framework is used for tests and Features (more details in updated TESTING.md) setup and teardown
  • Thanks to it we can inject things like ktf's cluster for the purpose of deploying addons in test.
  • This allows us to install kong addon - Kong Gateway installation via the helm chart - per test (or rather per feature, but that's a detail)
  • This OTOH allows to
    • Run tests in parallel being sure that no test conflicts with another one
    • Not need to "lock a mutex" for a listener port
• Thanks to e2e-framework's features like test Feature's labels we're able to filter the tests not only via names but also by labels and only those tests that were filtered will have their setup run
  • The reason we need to use Features is BeforeEachFeature and AfterEachFeature are not skipped even though feature labels indicate so kubernetes-sigs/e2e-framework#330 which for some reason doesn't allow use to use a shared setup like tenv.BeforeEach[Test|Feature]

---

• An extension of this new proposal could be a merge of the existing system of integration tests and the new one where based on the test Feature labels we'll be able to use a shared kong instance (installed once in kong namespace) or a dedicated one (installed per Feature in its own dedicated namespace)

Special notes for your reviewer:

Added a separate section in TESTING.md

So far the new suite is in the test/integration/isolated.

---

What do we gain by this change in terms of numbers?

Non scientific comparison of relevant test suites:

• dbless expressions router suite before this change: https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6510664946/job/17690016290
  • tests run: 21m52s
• dbless expressions router suite after this change: https://github.com/Kong/kubernetes-ingress-controller/actions/runs/6511109860/job/17737909158
  • tests run: 16m16s

NOTE: this moved the tests around, from the above mentioned suite, to isolated-dbless-expression-router suite (which run for: 6m0s) but the change is that in the isolated suite tests can run in parallel without any interference so the more tests we add there the more benefits we'll see.

[codecov]

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (d12fbef) 77.9% compared to head (24da4ba) 76.8%.
Report is 26 commits behind head on main.

Files	Patch %	Lines
internal/util/test/controller_manager.go	42.4%	15 Missing and 4 partials ⚠️
internal/util/test/kustomize_consts.go	75.8%	4 Missing and 3 partials ⚠️
internal/cmd/rootcmd/run.go	0.0%	6 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #4823     +/-   ##
=======================================
- Coverage   77.9%   76.8%   -1.2%     
=======================================
  Files        168     170      +2     
  Lines      18899   19050    +151     
=======================================
- Hits       14736   14637     -99     
- Misses      3336    3583    +247     
- Partials     827     830      +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[czeslavo]

🎖️ I like the idea very much, especially the structure it can enforce and the flexibility it gives us in terms of injecting various dependencies into a test body. That promotes writing concise and well-readable test cases (or at least makes that easier).

I took a first pass of the review. Will take another tomorrow. 🖖

[mlavacca]

This looks great, @pmalek!

I have one question though - we are running in isolation the UDPRoute tests only. Which other tests do you think we can run in such an isolated environment? Do we have something we should avoid?

[pmalek]

@mlavacca

I have one question though - we are running in isolation the UDPRoute tests only. Which other tests do you think we can run in such an isolated environment? Do we have something we should avoid?

I don't have everything written down because at the time I put up this PR I wasn't sure everyone would be on board with this approach but hey! It seems now that we're mostly OK with this we can plan ahead.

My very high level plan for this is:

• merge this PR
• address Tests: can't use e2e-framework's [After|Before]EachFeature to get conditional setup / teardown run #4847
• reconsider if we want to use kong per feature (current approach mostly due to Tests: can't use e2e-framework's [After|Before]EachFeature to get conditional setup / teardown run #4847) or kong per test (as in Go test)
• migrate all integration tests 1 by 1 to the new (isolated) framework
  • during this migrate decide if the test should be isolated (get its own kong instance and namespace) or use a shared installation of kong (installed in default namespace)

[mlavacca]

@mlavacca

I have one question though - we are running in isolation the UDPRoute tests only. Which other tests do you think we can run in such an isolated environment? Do we have something we should avoid?

I don't have everything written down because at the time I put up this PR I wasn't sure everyone would be on board with this approach but hey! It seems now that we're mostly OK with this we can plan ahead.

My very high level plan for this is:

• merge this PR

• address Tests: can't use e2e-framework's [After|Before]EachFeature to get conditional setup / teardown run #4847

• reconsider if we want to use kong per feature (current approach mostly due to Tests: can't use e2e-framework's [After|Before]EachFeature to get conditional setup / teardown run #4847) or kong per test (as in Go test)

• migrate all integration tests 1 by 1 to the new (isolated) framework

  • during this migrate decide if the test should be isolated (get its own kong instance and namespace) or use a shared installation of kong (installed in default namespace)

I like this plan, sounds good to me 👍

[mlavacca]

🚀

[pmalek]

@randmonkey and @programmer04 Feel free to respond to the still opened conversations and close them if you feel they have been addressed.

[pmalek]

#4847 can be considered a follow up to this PR.

When this gets merged we're already going to use a version of e2e-framework which will contain the fix.

---

Another follow up could be #4849 which removes the hardcoded FF and would make tests read those from env (as it's currently the case in the standard integration test suite).