Azure Cosmos DB: Introduce CosmosDBEmulatorContainer

[okohub]

Azure Cosmos DB is a fully managed NoSQL database service for modern app development. I want to introduce it as a test container. Azure Cosmos DB has its own emulator and docker image provided by Microsoft.

The critical point in implementing this container is mandatory SSL connection. Azure Java SDK (https://github.com/Azure/azure-sdk-for-java) has no option to override SSL check, so I added some configurable KeyStore codes to prepare SSL context properly instead of overriding it via reflection or something else.

Please feel free to guide me about contributing this emulator at its best.

Default docker image - Checked

Module dependencies - Checked

API - Checked

Documentation (don't know what can I do extra?)

[okohub]

Fixes #4283

[okohub]

Kindly ping :) @bsideup @kiview @rnorth

[kiview]

Hi @okohub,

thanks a lot for this module, this will be a great addition and it is very much appreciated!
I have some general remarks, afterwards, it would be good, if someone with more CosmosDB background looks at it as well.

[okohub]

@kiview @rnorth ready to another review :) thanks for your help.

[kiview]

Thanks for the quick turnover @okohub, this looks much better, great work!
I will check in detail tomorrow.

[okohub]

@kiview @bsideup done with latest reviews.

[kiview]

LGTM now, thanks a lot.

Maybe @bsideup can have a final look as well.

[okohub]

@kiview @bsideup

ok so, why not have both :)

I added a KeyStoreUtils as before I did, with ssl disabled okhttp. Now we can also build key store from downloaded pem file. I kept both buildKeyStore methods to give flexibility to developer.

We can also describe both methods in documentations by the way.

[rnorth]

I was going to suggest adding a quick bit of code to make sure we cleanup our system properties within our tests, as I have the fear that one day some tests within our wider test suite could break after the trust store modifications.

When trying out a quick patch locally, I encountered something very surprising: the tests now fail.

ERROR 🐳 [mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:2.11.13-c5c131e3] - Log output from the failed container:
Error: The evaluation period has expired.
./palrun: ERROR: PAL initialization failed. Error: 104

This obviously worked previously, and doesn't happen when using the latest tag of the image, so I can only imagine that the pinned tagged images have some kind of expiry built in.

For the first time it seems that we can't pin an image to a specific version, which is quite alarming from a reproduceability perspective. Clearly the real CosmosDB service is itself evolving, but not being able to pin the version used in testing will be a headache.

Is there any possibility that I'm wrong? Can we somehow pin the image and not have it expire?

[okohub]

@bsideup I added Junit TemporaryFolder rule, I didn't know that thank you.

@rnorth I checked latest tags here:

https://mcr.microsoft.com/v2/cosmosdb/linux/azure-cosmos-emulator/tags/list

and re-read documents here

https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator

but I didn't see a way to workaround expire problem. Looks like all emulators have 164 days to expire after they built. May be we should ask it to @JonathanGiles

I was going to suggest adding a quick bit of code to make sure we cleanup our system properties within our tests, as I have the fear that one day some tests within our wider test suite could break after the trust store modifications.

When trying out a quick patch locally, I encountered something very surprising: the tests now fail.

ERROR 🐳 [mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:2.11.13-c5c131e3] - Log output from the failed container:
Error: The evaluation period has expired.
./palrun: ERROR: PAL initialization failed. Error: 104

This obviously worked previously, and doesn't happen when using the latest tag of the image, so I can only imagine that the pinned tagged images have some kind of expiry built in.

For the first time it seems that we can't pin an image to a specific version, which is quite alarming from a reproduceability perspective. Clearly the real CosmosDB service is itself evolving, but not being able to pin the version used in testing will be a headache.

Is there any possibility that I'm wrong? Can we somehow pin the image and not have it expire?

[rnorth]

It's a shame that we have to use a :latest image for our tests, but I don't think this should be a blocker. LGTM, thanks for your hard work!

[rnorth]

Ohoh, this is unfortunate:

Execution failed for task ':azure:japicmp'.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
> Could not resolve all files for configuration ':azure:baseline'.

   > Could not find org.testcontainers:azure:1.16.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
     Searched in the following locations:     

We need to work out a way to prevent the japicmp checks from failing on new modules...

[bsideup]

@rnorth on a second thought (after the review)... Does it mean our users will have to use :latest as well? 🥲

[rnorth]

We can see that the azure:test task passed and then azure:japicmp then failed for entirely predictable reasons.

I was going to say: with this in mind I'm inclined to merge and then revisit japicmp's resolution of new modules if/when necessary.

However, I realise that this will also break master branch until the next release, so we probably don't have the option to defer this 😬

[rnorth]

@rnorth on a second thought (after the review)... Does it mean our users will have to use :latest as well? 🥲

@bsideup yes it does indeed 😬

The slight silver lining compared to modules for 'self hosted' services is that the real prod cloud service is ever-evolving. However, being unable to pin versions and have repeatable tests does add quite a bit of displeasure.

[bsideup]

Hey @JonathanGiles! Is there absolutely nothing that can be done to stop CosmosDB Docker images from "expiring"?
-SNAPSHOT style test dependencies tend to make test suites very unreliable, and we wanted to make sure that our Azure support is solid 🦾 Thanks in advance!

[okohub]

@rnorth after reading again your review, this sentence drew my attention

I was going to suggest adding a quick bit of code to make sure we cleanup our system properties within our tests, as I have the fear that one day some tests within our wider test suite could break after the trust store modifications.

May be we can cover test method with try/finally and set parameters to default state.

Something like that:

String defaultTrustStore = null;
String defaultTrustStorePassword = null;
String defaultTrustStoreType = null;
try {
defaultTrustStore = getprop
defaultTrustStorePassword = getprop
defaultTrustStoreType = getprop

//do some serious thing

} finally {
 if (defaultTrustStore != null) {
    setprop defaultTrustStore
  }
  //and others
}

[rnorth]

@okohub thanks for remembering!

I'll add a suggested change, which I think reduces the amount of noise inside the test method.

[rnorth]

Actually please could you have a look at this commit and use that?

[okohub]

@rnorth Of course! I will give it a try, btw I want to squash these commit flood to single one. Is it ok?

[rnorth]

@okohub Sure, if you want! We will squash before merging anyway, so you only need to squash commits if it helps you.

[okohub]

@rnorth thanks! Also I added your prop suggestion and rebased into one commit.

[rnorth]

@rnorth thanks! Also I added your prop suggestion and rebased into one commit.

Thanks @okohub !

[kiview]

Merged, thanks a lot @okohub for your patience and your commitment to getting this PR merged, much appreciated 🙇

[RikEnde]

hi @okohub do you have an example of a working project using this container with the latest version of the image?

[kiview]

Check the code example in our docs, which are executed as part of our CI:
https://www.testcontainers.org/modules/azure/#cosmosdb