End-to-end release test on ci-mgmt

[VenelinMartinov]

@t0yv0 proposed we take #857 one step further and have an end-to-end test by checking for end-user behaviour like Pypi binaries, etc.

We might not want to do this on every commit but it sounds totally doable in general.

It should make iterating on changes to our release process much saner.

[t0yv0]

Something that runs a Pulumi program against a recently published binary in every language would do the trick (with appropriate retry/backoff). As far as I'm aware, the actual capability to publish working programs is not yet tested end to end. Another thing to keep in mind when developing it is that we might want to at some point not only test this to qualify code changes, but also verify actual releases.

[t0yv0]

Lost an AWS release to see untested changes https://github.com/pulumi/pulumi-aws/actions/runs/8454321289

Sadness

[danielrbradley]

Is this solved by #977 once we've also completed #1035 ?

[VenelinMartinov]

Almost, I think this issue was about verifying ci-mgmt changes don't break release processes, so we'd also want to set up a job on xyz to make a release on ci-mgmt PRs and verify they are valid.

[danielrbradley]

Fixed by #1059