Skip to content

Latest commit

 

History

History
70 lines (55 loc) · 4.44 KB

sqs.md

File metadata and controls

70 lines (55 loc) · 4.44 KB
Raw

SQS - Simple Queue Service

  • SQS provides managed message queues
  • It is a public AWS service
  • It is fully managed with highly available and highly performant queues by design
  • Queues come in two types: Standard (ordering is best effort) and FIFO (guarantee an order)
  • Messages added to the queue are up to 256 KB in size
  • VisibilityTimeout: when a client receives a message, the message is hidden for a period of time. The VisibilityTimeout is the period which is given to the client in order to process the message
  • If the client processes the message, it has to delete the message from the queue. If the message is not deleted, after the VisibilityTimeout the message will be available for other clients
  • Dead-Letter queues: all unprocessed messages can be sent here after a given number of retries. Allows different types of processing on the messages
  • Queues can be used for scaling by ASGs, Lambdas can be invoked based on messages in the queue

SQS Standard vs FIFO Queues

  • Standard:
    • Messages are delivered at least once
    • The order of the delivery is not guaranteed (best-effort ordering)
    • They don't have performance limitations
  • FIFO:
    • Messages are delivered exactly once
    • The order of the delivery is guaranteed to be the same as the messages were sent
    • Performance is limited: 3000 messages per second with batching or up to 300 messages per second without batching
    • There is also available a high throughput mode for FIFO queues
    • FIFO queues have to have .fifo suffix in order to be valid FIFO queues

SQS Billing

  • With SQS we are billed per request
  • With a 1 request we can receive between 1 and 10 messages up to 64 KB total
  • SQS is less cost effective if we poll more frequently

SQS Polling

  • Polling SQS queues:
    • Short polling (immediate): can return 0 or more messages. It will return immediately if there are no messages on the queue
    • Long polling: we can specify waitTimeSecond value which can be up to 20 seconds to wait form messages when polling. It uses fewer requests than short polling

SQS Encryption and Security

  • SQS supports encryption at rest (KMS) and encryption in transit
  • Access to a queue is based on identity policies and queue policies
  • Queue policies can allow cross-account access

SQS Extended Client Library

  • SQS has a message size limit of 256 KB
  • Extended Client Library can be used when we want to send messages larger than this size
  • It can process large payloads and have the bulk of the payloads stored in S3
  • When we send a message using SendMessage API, the library uploads the content to S3 and stores a link to this content in the message
  • When receiving a message, the library loads the payload from S3 and replaces the link with the payload in the message
  • When deleting a message from a queue, the large S3 payload will also be deleted
  • The Extended Client Library can handle messages up to 2 GB
  • It has an implementation in Java. Equivalent libraries are available for other languages

SQS Delay Queues

  • Delay queues allow us to postpone the delivery of messages in SQS queues
  • For a delay queue we configure a DelaySeconds value. Messages added to the queue will be invisible for the amount of DelaySeconds
  • The default value of DelaySeconds is 0, the max value is 15 minutes. In order for a queue to be delay queue, the value should be set to be greater than 0
  • Message times allows a per-message basis invisibility to be set, overriding the queue setting. Min/max values are the same. Per message setting is not supported for FIFO queues

SQS Dead Letter Queues (DLQ)

  • Dead Letter Queues are designed to help handle reoccurring failures while processing messages from a queue
  • Every time a message is received, the ReceiveCount is incremented. In order to not process the same message over and over again, we define a maxReceiveCount and a DLQ for a queue. After the number of retries is greater than the maxReceiveCount, the message is pushed into the DLQ
  • Having a DLQ, we can define alarms to get notified in case of a failure
  • Messages in DLQ can be further analyzed
  • Every queue (including DLQ) have a retention period for messages. Every message has an mq-timestamp which represents when was the message was added to the queue. In case a message is moved into a DLQ, this timestamp is not altered
  • Generally the retention period for a DLQ should be longer compared to normal queues
  • A single DLQ can be used for multiple sources