-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Please add support for AWS SDK v2 #1442
Comments
I've used Testcontainers for testing code that uses AWS SDKv2, and I'd agree that LocalStackContainer's coupling to the v1 SDK makes it a rough fit. The AWS SDK coupling in general is, I think, fairly important to retain as it removes some pain points that would be exposed if we just provided a thin wrapper around Localstack. I think that we probably need a separate module for AWS SDKv2 compatibility in this case. Let me think and discuss with @bsideup and @kiview... |
@rnorth - it might be possible to make two subclasses of a Perhaps some of the issue here is that the local stack container doesn't really need to do a HUGE amount of AWS stuff itself. We only really need the URL to pass to our SDK client library. If @bsideup and @kiview and yourself can propose how this should be done, I don't mind hacking a PR together. It would be useful in my current project. |
Thanks @ashleyfrieze - sorry for the slow reply.
You're quite right; this container only faintly interacts with the AWS SDK, but just enough to cause us a compatibility issue. We since learned that this is a bit of a mistaken pattern to use for our modules, and now recommend that new modules only expose JDK types in their public APIs. Too late for this module, though 😬. Thinking about whether this should be within the same module: Pros:
Cons:
In the long run, IMHO it would be good to have a But that wouldn't stop us from taking the first step in that direction: creation of Thoughts? (Aside: it's amusing that this is a prime example of 'naming things' being the hardest part of the problem 😄) |
Would you mind explaining your strategy to use Testcontainers (currently coupled to SDK V1) and code written with AWS SDK V2? @ashleyfrieze, @bsideup, @kiview, and @rnorth I am craving for SDK V2 support in Testcontainers. Any news? |
@oscarvarto we have something like this (in Kotlin): import software.amazon.awssdk.awscore.client.builder.AwsClientBuilder
...
fun <T: AwsClientBuilder<*,*>> T.localStack(
container: LocalStackContainer,
service: LocalStackContainer.Service
): T {
endpointOverride(URI(container.getEndpointConfiguration(service).serviceEndpoint))
credentialsProvider(container.defaultCredentialsProvider?.let { v1Provider ->
val v1Credentials = v1Provider.credentials
AwsCredentialsProvider {
object : AwsCredentials {
override fun accessKeyId(): String = v1Credentials.awsAccessKeyId
override fun secretAccessKey(): String = v1Credentials.awsSecretKey
}
}
})
return this
} And then build SDK v2 clients that connect to a val client = DynamoDbClient.builder()
.localStack(localStackContainer, DYNAMODB))
.build() |
Scala version of the @xenomachina code: import java.net.URI
import org.testcontainers.containers.localstack.LocalStackContainer
import software.amazon.awssdk.auth.credentials.{AwsCredentials, AwsCredentialsProvider}
import software.amazon.awssdk.awscore.client.builder.AwsClientBuilder
import software.amazon.awssdk.regions.Region
object AwsClientBuilderOps {
implicit final class AwsClientBuilderOps[T <: AwsClientBuilder[_, _]](val builder: T) extends AnyVal {
def localStack(container: LocalStackContainer, service: LocalStackContainer.Service): T = {
builder.endpointOverride(new URI(container.getEndpointConfiguration(service).getServiceEndpoint))
builder.region(Region.US_EAST_1)
builder.credentialsProvider {
val v1Credentials = container.getDefaultCredentialsProvider.getCredentials
new AwsCredentialsProvider {
override def resolveCredentials(): AwsCredentials = new AwsCredentials {
override def accessKeyId(): String = v1Credentials.getAWSAccessKeyId
override def secretAccessKey(): String = v1Credentials.getAWSSecretKey
}
}
}
builder
}
}
} I added the Cheers |
And now a Java version of the temporary solution: @Bean
public S3Client s3Client(LocalStackContainer localStackContainer) throws URISyntaxException {
return S3Client
.builder()
.endpointOverride(new URI(localStackContainer.getEndpointConfiguration(LocalStackContainer.Service.S3).getServiceEndpoint()))
.credentialsProvider(new CrossAwsCredentialsProvider(localStackContainer.getDefaultCredentialsProvider()))
.build();
}
private class CrossAwsCredentialsProvider implements AwsCredentialsProvider {
private final AWSCredentials credentials;
public CrossAwsCredentialsProvider(AWSCredentialsProvider provider) {
this.credentials = provider.getCredentials();
}
@Override
public AwsCredentials resolveCredentials() {
return AwsBasicCredentials.create(credentials.getAWSAccessKeyId(), credentials.getAWSSecretKey());
}
} |
When will the fix committed by @rnorth be released? I am using v1.14.3. |
Is this issue solved because I am on 15 and it seems to still be an issue. |
I'am also experiencing issues with version 1.14.3, when using AWS SDK v2:
The reason for that error is, that the localstack testconainers module internally still uses the AWS SDK v1. While that would be no problem, the issue is, that it declares it as compileOnly dependency (see build.gradle) A quick fix for this issue is to also provide the AWS SDK in test scop.
I do however no understand, why this dependecy is set to compileOnly. At least some documentation would be very helpful. |
Hey so I wasn't happy with just adding the v1 dependency, even in test, it felt too hacky for me, so I put together a import org.rnorth.ducttape.Preconditions
import org.testcontainers.DockerClientFactory
import org.testcontainers.containers.GenericContainer
import org.testcontainers.containers.wait.strategy.Wait
import org.testcontainers.utility.DockerImageName
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider
import java.net.InetAddress
import java.net.URI
class LocalstackContainerV2(
dockerImageName: DockerImageName,
// can be ignored, personally I need to leverage the localstack pro offerings
val apiKey: String? = null
) : GenericContainer<LocalstackContainerV2>(dockerImageName) {
companion object {
private const val PORT = 4566
const val ACCESS_KEY = "accessKey"
const val SECRET_KEY = "secretKey"
const val REGION = "us-east-1"
}
// TODO override to prevent version < 0.11.5?
private val defaultImageName = DockerImageName.parse("localstack/localstack")
private var services = emptyList<Service>()
init {
dockerImageName.assertCompatibleWith(defaultImageName)
withFileSystemBind(DockerClientFactory.instance().remoteDockerUnixSocketPath, "/var/run/docker.sock")
waitingFor(Wait.forLogMessage(".*Ready\\.\n", 1))
}
override fun configure() {
super.configure()
setPreconditions()
configureEnv()
exposePorts()
}
fun withServices(vararg services: Service): LocalstackContainerV2 = apply {
this.services = this.services.plus(services)
}
fun getEndpointOverride(): URI {
val hostAddress = InetAddress.getByName(host).hostAddress
return URI("http://$hostAddress:${this.firstMappedPort}")
}
fun getDefaultCredentialsProvider(): AwsCredentialsProvider = StaticCredentialsProvider.create(
AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY)
)
private fun setPreconditions() {
Preconditions.check("services list must not be empty", services.isNotEmpty())
}
private fun configureEnv() {
withEnv("SERVICES", services.joinToString(separator = ",") { it.serviceName })
if (apiKey != null) {
withEnv("LOCALSTACK_API_KEY", apiKey)
}
}
private fun exposePorts() {
addExposedPort(PORT)
}
enum class Service(val serviceName: String) {
S3("s3"),
ECR("ecr")
// etc... more services go here
}
} Then an abstract class instantiating the container can be something like abstract class LocalstackIntegrationTest {
companion object {
private val region = Region.of(LocalstackContainerV2.REGION)
private val services = arrayOf(LocalstackContainerV2.Service.S3, LocalstackContainerV2.Service.ECR)
private val LOCALSTACK_IMAGE = DockerImageName.parse("localstack/localstack:0.12.2")
lateinit var s3Consumer: S3Consumer
lateinit var ecrConsumer: EcrConsumer
val localstack = LocalstackContainerV2(LOCALSTACK_IMAGE).apply {
withServices(*services)
}
}
init {
localstack.start()
}
@BeforeAll
fun setup() {
val s3client = setupS3Client()
val ecrClient = setupEcrClient()
s3Consumer = S3Consumer(s3client)
ecrConsumer = EcrConsumer(ecrClient)
}
private fun setupS3Client() = S3Client.builder()
.region(region)
.endpointOverride(localstack.getEndpointOverride())
.credentialsProvider(localstack.getDefaultCredentialsProvider())
.build()
private fun setupEcrClient() = EcrClient.builder()
.region(region)
.endpointOverride(localstack.getEndpointOverride())
.credentialsProvider(localstack.getDefaultCredentialsProvider())
.build()
} |
@rnorth - perhaps the long-term solution is to make the SDK v1 class (keeping its current name) a subclass of an SDK independent LocalstackTestContainer - say The existing class can become a subclass of the above, adding in the SDK v1 dependencies. It would then be a case of either producing documentation showing how to use API v2, or producing an APIv2 specific subclass if necessary. I suspect the first of these would be better. If there's interest in this, I could look at submitting a PR along these lines...? |
IMHO this issue should not be closed, as the code of #3557 was not merged at all. Could we reopen this issue, as support for aws sdk v2 is still lacking ? |
@rnorth Is it fine to reopen it? |
Hi, just an update on this topic. Methods using aws sdk v1 have been deprecated As mentioned in the checklist for a new module
That said, the issue can remain closed. |
LocalStackContainer
uses the AWS SDK for Java v1.The AWS SDK for Java v2 API has renamed everything. In particular, all of the packages have changed, and so the
AWSCredentialsProvider
andAwsClientBuilder.EndpointConfiguration
returned byLocalStackContainer
are not usable as-is in the AWS SDK for Java v2. It is possible to adapt them with some work (and we can share our wrapper if you like, though it is written in Kotlin) however, it would be nice if clients using one version of the AWS SDK for Java did not need to depend on artifacts from the other version, even transitively.I imagine you'll want to retain support for the v1 SDK at least until AWS deprecates it.
The text was updated successfully, but these errors were encountered: