Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DEVPLAT-1850] Add SQS support to PubSub #127

Merged
merged 6 commits into from
Nov 8, 2024
Merged

[DEVPLAT-1850] Add SQS support to PubSub #127

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
367889a
feat: Upgrade aws-sdk-go-v2 to latest version
Neurostep Nov 5, 2024
56515a5
chore(PubSub): Move pool to its own package
Neurostep Nov 1, 2024
0afe635
feat(PubSub): Add SQS configuration
Neurostep Nov 1, 2024
fda81ed
feat(PubSub): Add SQS PubSub wrappers
Neurostep Nov 1, 2024
a9f9f67
feat(PubSub): Add go-kit SQS transport functions
Neurostep Nov 1, 2024
720e682
docs: Add PubSub SQS section to README file
Neurostep Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ SDK, the Go version.
- [Usage of ORM](#usage-of-orm)
- [PubSub](#pubsub)
- [Kafka specific configuration](#kafka-specific-configuration)
- [SQS specific configuration](#sqs-specific-configuration)
- [Cache](#cache)
- [Redis](#redis-specific-configuration)
- [Experimentation](#experimentation)
Expand Down Expand Up @@ -849,6 +850,23 @@ common: &common
subscriber:
topic: "test-topic"
group_id: ""
sqs:
subscriber:
laynax marked this conversation as resolved.
Show resolved Hide resolved
# Set by APP_PUBSUB_SQS_SUBSCRIBER_ENABLED env variable
enabled: true
# Set by APP_PUBSUB_SQS_SUBSCRIBER_QUEUE_URL env variable
queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/test-queue"
# Set by APP_PUBSUB_SQS_SUBSCRIBER_MAX_MESSAGES env variable
max_messages: 10
# Set by APP_PUBSUB_SQS_SUBSCRIBER_WORKERS env variable
workers: 1
# Set by APP_PUBSUB_SQS_SUBSCRIBER_WAIT_TIME env variable
wait_time: "10s"
publisher:
# Set by APP_PUBSUB_SQS_PUBLISHER_ENABLED env variable
enabled: true
# Set by APP_PUBSUB_SQS_PUBLISHER_QUEUE_URL env variable
queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/test-queue"
```

#### Kafka specific configuration
Expand Down Expand Up @@ -920,6 +938,25 @@ To authenticate the requests to Kafka, Go SDK provides a configuration set for T
| Assumable role | This role will be used to establish connection to AWS MSK ignoring the static credentials | `role` | `APP_PUBSUB_KAFKA_SASL_AWS_MSK_IAM_ROLE` | string | AWS ARN string |
| Session name | Will be passed to AWS STS when assuming the role | `session_name` | `APP_PUBSUB_KAFKA_SASL_AWS_MSK_IAM_SESSION_NAME` | string | application |

#### SQS specific configuration

Under the hood, the SQS integration relies on the provided SQS client. To learn more about the AWS services initialization, check the [AWS service configuration](#aws-service-configuration).

**Subscriber**:

| Setting | Description | YAML variable | Environment variable (ENV) | Type | Possible Values |
|-------------------|-------------------------------------------------------------------------------------------------|-----------------------|---------------------------------------------------|---------|-----------------|
| Queue URL | URL of the SQS queue | `queue_url` | `APP_PUBSUB_SQS_SUBSCRIBER_QUEUE_URL` | string | queue URL |
| Max messages | Maximum number of messages to retrieve from the queue per polling iteration | `max_messages` | `APP_PUBSUB_SQS_SUBSCRIBER_MAX_MESSAGES` | number | 1,2... |
| Number of workers | Number of workers processing incoming messages | `workers` | `APP_PUBSUB_SQS_SUBSCRIBER_WORKERS` | number | 1,2... |
| Wait time | The maximum amount of time in time.Duration to wait for messages to be available for retrieval. | `wait_time` | `APP_PUBSUB_SQS_SUBSCRIBER_WAIT_TIME` | string | 10s |

**Publisher**:

| Setting | Description | YAML variable | Environment variable (ENV) | Type | Possible Values |
|-----------|-----------------------|-----------------------|---------------------------------------------------|---------|-----------------|
| Queue URL | URL of the SQS queue | `queue_url` | `APP_PUBSUB_SQS_PUBLISHER_QUEUE_URL` | string | queue URL |

### Cache

`go-sdk` provides a convenient way to create an application Cache configuration.
Expand Down
53 changes: 27 additions & 26 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,15 @@ require (
github.com/DATA-DOG/go-txdb v0.1.9
github.com/DataDog/datadog-go v4.8.2+incompatible
github.com/aws/aws-sdk-go v1.44.327
github.com/aws/aws-sdk-go-v2 v1.26.1
github.com/aws/aws-sdk-go-v2/config v1.27.11
github.com/aws/aws-sdk-go-v2/credentials v1.17.11
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1
github.com/aws/aws-sdk-go-v2/service/sagemakerruntime v1.27.4
github.com/aws/aws-sdk-go-v2/service/sfn v1.19.4
github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4
github.com/aws/aws-sdk-go-v2/service/sts v1.28.6
github.com/aws/aws-sdk-go-v2 v1.32.3
github.com/aws/aws-sdk-go-v2/config v1.28.1
github.com/aws/aws-sdk-go-v2/credentials v1.17.42
github.com/aws/aws-sdk-go-v2/service/s3 v1.66.2
github.com/aws/aws-sdk-go-v2/service/sagemakerruntime v1.32.3
github.com/aws/aws-sdk-go-v2/service/sfn v1.33.3
github.com/aws/aws-sdk-go-v2/service/sqs v1.36.3
github.com/aws/aws-sdk-go-v2/service/sts v1.32.3
github.com/aws/smithy-go v1.22.0
github.com/getsentry/sentry-go v0.12.0
github.com/go-kit/kit v0.9.0
github.com/go-sql-driver/mysql v1.7.1
Expand Down Expand Up @@ -48,24 +49,24 @@ require (
github.com/DataDog/gostackparse v0.7.0 // indirect
github.com/DataDog/sketches-go v1.4.2 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5 // indirect
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.21.4 // indirect
github.com/aws/aws-sdk-go-v2/service/eventbridge v1.20.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.7.34 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
github.com/aws/aws-sdk-go-v2/service/kinesis v1.18.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.21.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.20.5 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
github.com/aws/smithy-go v1.20.2 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.6 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.18 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.22 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.22 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.22 // indirect
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.36.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ec2 v1.187.0 // indirect
github.com/aws/aws-sdk-go-v2/service/eventbridge v1.35.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.10.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.3 // indirect
github.com/aws/aws-sdk-go-v2/service/kinesis v1.32.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sns v1.33.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.3 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
Expand Down
Loading
Loading