Add Self-Hosted AWS GPU Runner

[mikemhenry]

No description provided.

[mikemhenry]

Ready for review @RaulPPelaez and @raimis

I only tested the latest version and before we merge this in we will want to decide how often to run the scheduled test and we want to remove the part the triggers the run in this branch.

[mikemhenry]

Also one more thing I want to do before merge is to use variables for the package versions so it is easier to change in the future

[RaulPPelaez]

This is a really cool functionality @mikemhenry, thanks! Its just some minor comments from me.

[mikemhenry]

Okay this is ready for review @raimis and @RaulPPelaez
The only thing left is to decide how frequent you want to run it.
It costs @jchodera $0.526 per hour, and it looks like it runs for ~ 11 minutes so it is pretty cheap. I think running it on every commit on a PR though is wasteful. I recommend running it nightly, merge commits to master, and manually on a branch that can be done on-demand (useful to do before merging).

[peastman]

If we want to keep the cost down, running it only on merge commits to master is probably sufficient. Running it nightly on code that's already been merged and tested doesn't tell us much more.

[RaulPPelaez]

I agree with @peastman, nightly is too much. I say we run this just before committing to master.
Other than that the ci looks great to me, if we want to play it safe get the timeout down to 25 mins.
Thanks @mikemhenry

[mikemhenry]

Okay! This should be good to merge now!

[RaulPPelaez]

This looks great to me. I say lets merge.
BTW, I see this is suffering from #98
Do you have any explanation?
I have not been able to reproduce it. My guess is on some CMake parameter.

[raimis]

Looks good!