Add GCloud module for Google Cloud Datastore, Firestore, PubSub, and Spanner emulators

[eddumelendez]

• Add support to perform additional commands in the container
• Add docs
• Add gcloud module

[eddumelendez]

It is pending to polish the pubsub test 🙈

[bsideup]

does it really need to be an image? Can't we just use the command for it, since the image is not cached anyways?

[eddumelendez]

At the beginning I was not able to run it using command but that led me to know more about testcontainers. Now, I did it but had to set up a startupTimeout.

[rnorth]

Are we sure about this? 😄
It looks like these setup steps will require a non-trivial amount of time to execute - perhaps even the majority of the startup time.

Would it be worth trying to get more attention on GoogleCloudPlatform/cloud-sdk-docker#202 so that we can avoid having these setup steps entirely?

[bsideup]

Actually, I missed that we install Java and other things... Yeah, I believe we need to wait for these images :(

[bsideup]

/cc @jamesward @saturnism

[saturnism]

quick ping on this to see if there are any more work needed, happy to help

[eddumelendez]

@saturnism currently, one of the things is the issue GoogleCloudPlatform/cloud-sdk-docker#202 in order to get images for the rest of the emulators. Also, I would appreciate your feedback in the tests. Thanks!

[saturnism]

Will it be possible for the first iteration to use the non-alpine image that has emulators pre-installed? e.g., google/cloud-sdk:292.0.0 (and being able to specify version would be great!

[eddumelendez]

for the first version, it make sense for me to use that image with components pre-installed. if everyone agree I can work on the changes.

[eddumelendez]

@saturnism I was working on the changes but still using image google/cloud-sdk:304.0.0-debian_component_based we will need to install jdk in order to run emulators 😢

[saturnism]

@eddumelendez can we use gcr.io/google.com/cloudsdktool/cloud-sdk:305.0.0? it's a larger image but should have the emulators.

however, it seems like firestore and spanner ones are not installed in this.
GoogleCloudPlatform/cloud-sdk-docker#217

[saturnism]

added GoogleCloudPlatform/cloud-sdk-docker#218

[eddumelendez]

@saturnism currently, I am using google/cloud-sdk:305.0.0 and it is working for datastore and pubsub. Regarding to spanner, in the cloud sdk use docker so would be a DinD approach. But, since the image in already available gcr.io/cloud-spanner-emulator/emulator:1.0.0 I am using it directly, or the PR you already sent will not use docker?. I will wait for firestore to be added so I will run my tests again and update the PR.

Have the feeling that we are close 😃

[saturnism]

a quick update, the pr to have additional emulators has been merged, and it's likely it'll be released tomorrow under the 306 version tag. will keep you updated! i'm excited to see a closure on this soon :)

[eddumelendez]

saw the merge notification. So excited too! I will be updating the PR tomorrow night and then ask for review

[eddumelendez]

very glad to let you know that now that all components are installed in the image provided we can run everything successfully

[saturnism]

this looks great! i feel this is almost ready to go?

[walles]

It would be nice to have a(nother) test that:

1. Publishes a message with some trivial contents
2. Pulls that message from a subscription to that topic
3. Verifies that the trivial contents was received

[eddumelendez]

@walles thanks for the suggestions. I have added one more commit introducing that change. 👍🏽

[walles]

Beautiful! 🥇

[rnorth]

Thanks! I'll try and have another look at this over the weekend.

For now there's a docs build failure, apparently due to an incorrect codeinclude path:

12:58:58 AM: FileNotFoundError: [Errno 2] No such file or directory: '/opt/build/repo/docs/modules/../../../modules/gcloud/src/test/java/org/testcontainers/containers/DatastoreEmulatorContainerTest.java'

If you want a quicker feedback loop to see/check the generated docs locally, please have a look at: https://www.testcontainers.org/contributing_docs/

[eddumelendez]

@rnorth thanks! fixed the docs

[eddumelendez]

@bsideup branch rebase with master and proper changes applied :)

[rnorth]

I wonder, maybe these should be public constants so that they can be used by tests? If we did that, we could do so for the other container classes in this PR.

WDYT @bsideup, @kiview?

[bsideup]

WDYT about not exposing the constants, but getGrpcPort/getHttpPort methods?

[saturnism]

looking at some of the other modules, private static final seems like the norm/convention atm. would it make sense to follow up w/ an issue for the broader change to expose these across modules?

[rnorth]

I've been persuaded by @bsideup - many of our other modules have some mechanism for getting the mapped ports, address or URL for the running container. It seems like we should do this here, instead of exposing constants, as it's fundamentally more useful for the user.

I'll take the action to do this, as we've asked @eddumelendez to do too much already. I'll raise a quick PR tonight, and will merge this PR now.

[rnorth]

A few pretty trivial comments from me; otherwise I think we should merge soon!

[rnorth]

LGTM but I'll let @bsideup comment too.
Thanks for the PR and the very quick responses @eddumelendez!

[rnorth]

Merging now - thanks once again for the contribution @eddumelendez!