Add support for integration testing of cloud features

[jsoriano]

We need to add support for better integration testing of cloud features, mainly to avoid regressions while further changes are introduced.

Some cloud features already have integration testing, but tests are skipped if no configuration is provided, so they are useful for local development, but they don't run in CI. Also there is no way to reproduce scenarios where these tests should pass.

As an initial step, add integration testing functionality to some AWS feature.

• Formalize some general way of defining, deploying and destroying testing scenarios (probably using terraform).
• Add build tags on the test files for each one of the cloud features, and make existing tests fail if the tag is provided but without credentials, so we can detect misconfigurations in CI environments.
• Add job to the Jenkinsfile, that obtains valid credentials, starts an scenario in a cloud provider, runs the tests related to this provider and destroys the scenario.

Additional considerations.

• We should be able to deploy the same scenario several times in the same account without conflicts.
• We should be able to detect and clean leftovers.

Related issues:

• Investigate on localstack to mock AWS for testing #14075 - Use of localstack to mock AWS

[exekias]

cc @kaiyan-sheng @narph

[kaiyan-sheng]

Thanks @jsoriano for starting on this! Should we also consider how frequent these cloud integration tests are ran in CI? Maybe these tests can only run once per day instead of for each PR to save on cost and etc.

[jsoriano]

Thanks @jsoriano for starting on this! Should we also consider how frequent these cloud integration tests are ran in CI? Maybe these tests can only run once per day instead of for each PR to save on cost and etc.

Yes, good point, this is something to have into consideration. There are some options that we could combine:

• Prepare a reduced set of sanity tests that run on all jenkins builds, to early detect serious regressions.
• Take advantage of selective testing (Selective testing in beats and test with latest version of the branch. #9496) so we run certain tests only when certain files are modified.
• Run cloud tests only in master or in daily builds.
• Add some command to run these tests on PRs.