Should run the Integration Tests as part of the continuous integration build

[brainmurphy]

These tests give us greater confidence in the changes we make. There is an issue in that they require a functioning AWS environment to execute.

[slang25]

We should definitely do something around this, cause the integration tests are constantly breaking as a result of this.
I have a branch locally that is working against localstack, which works for most of the tests, but not for cross region tests, or iam policy checks. We could run that on travis with docker without too much effort, although it would be good to have a dedicated aws to run this in properly too.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[martincostello]

I think this is one we shouldn't let go stale 😉

[martincostello]

As discussed, I'm going to pick up looking at this for both real end-to-end tests with real AWS resources, as well as investigating using localstack for inner-loop unit/functional tests.

Once I've got a PoC for it sorted out, I'll raise a PR and we can go through it and decide how to proceed.

[duncanbrown]

Just a random thought - might be worth considering goaws (https://github.com/p4tin/goaws) as an alternative to localstack: I've recently been using it with JustSaying for local development, and haven't had any issues yet (only been doing pretty simple stuff, though). I started using it as it seems much lighter-weight than localstack - docker image is just 12MB, and it runs in just a few MB of memory.

[martincostello]

Cool, thanks @duncanbrown. I'll take a look at that too!

[martincostello]

I've started having looking into this.

[martincostello]

Begun in #381.

[martincostello]

#381 has been merged which adds support for using goaws (thanks @duncanbrown 👍 ) during the Travis CI builds, as well as on Windows with Build.ps1 -EnableIntegrationTests if you have Docker installed and configured for Linux containers.

A future PR will validate the tests work with environment variables for real AWS resources, as well as a way of running the Windows build with integration tests in AppVeyor somehow given the free/OSS plan doesn't support Linux containers.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaynevanasperen]

@martincostello The AppVeyor free/OSS plan does support Linux containers now.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[martincostello]

Windows containers in Docker do not support Linux VMs on the free plan: #522 (comment)

[martincostello]

I've created an internal Jira to track getting us access to two real AWS accounts for doing testing with real AWS resources.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[martincostello]

We still don't have the AWS account, so pushing to a future release so it's not a blocker for v7.

[stale]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[martincostello]

I'm going to close this as "done" on the basis we have integration tests running for Linux in Travis CI.

We can open a new issue and make things even better at some point in the future if we get the time and resource (i.e. the AWS account) to do so.