What is AWS SQS and Industry use case of AWS SQS

Amazon Simple Queue Service is a pay-per-use web service for storing messages in transit between computers. Developers use SQS to build distributed applications with decoupled components without having to deal with the overhead of creating and maintaining message queues.

• System A — produces messages periodically in huge bursts
• System B — consumes messages constantly, at a slower pace

With this architecture, a queue would allow System A to produce messages as fast as it can, and System B to slowly digest the messages at its own pace.

Queues have played an integral role in software architecture for decades along with core technology concepts like API and ETL With the recent trend towards Microservices, have become more important than ever.

QS is a great way to decouple services, especially when there is a lot of heavy-duty, batch-oriented processing required.

For example, let’s say you have a service where people upload photos from their mobile devices. Once the photos are uploaded your service needs to do a bunch of processing of the photos, e.g. scaling them to different sizes, applying different filters, extracting metadata, etc.

One way to accomplish this would be to post a message to an SQS queue. The message describes work that needs to be performed on the newly uploaded image file. Once the message has been written to SQS, your application can return success to the user because you know that you have the image file and you have scheduled the processing.

In the background, you can have servers reading messages from SQS and performing the work specified in the messages. If one of those servers dies another one will pick up the message and perform the work. SQS guarantees that a message will be delivered eventually so you can be confident that the work will eventually get done.

Standard vs FIFO

One important detail that should be mentioned — there are two types of queues within AWS SQS:

• Standard — higher throughput, with “at least once delivery”, and “best-effort ordering”.
• FIFO (First-In-First-Out) — not as high throughput, but guarantees on “exactly once” processing, and preserving the ordering of messages.

Long story short, if you need things super-fast, can tolerate messages out of order, and possibly sent more than once, Standard queues are the answer. If you need absolute guarantees on order of operations, no duplication of work, and don’t have huge throughput needs, then FIFO queues are the best choice.

We’d better make sure we create our MainLine queue using FIFO! While a “mostly in order” guarantee might suffice in some situations, you’d have a riot on your hands at the DMV if people started getting called out of order. Purses swinging, hair pulling — it wouldn’t be pretty. Let’s add “FifoQueue=true” to the query string to indicate that the queue should be FIFO

Delivery Policies: Messages are guaranteed to be delivered at least once since they are retained in the queue as long as a consumer instructs SQS to delete messages. The possible scenarios for more than one delivery are

1. Messages not getting deleted from SQS before visibility timeout due to consumer crashing or taking longer than visibility timeout to process the message. The visibility timeout should be set to a value longer than the consumer workers normally take to finish processing, plus a bit extra for padding.
2. Concurrent consumers polling visible messages simultaneously.
3. In case of SQS server crashing and coming back up, if the messages that the server going down holds are already processed, they would be deleted from other servers but not from the server that went down resulting in re-delivery once the server comes up again.

As a result of at-least-once delivery, consuming services need to make sure that re-processing of messages does not result in an inconsistent state of the system by handling the messages in an idempotent manner.
Message Acknowledgements: this does not support message acknowledgement; a trade-off that SQS makes for providing high throughput.
Purging Policies: Messages are retained in SQS for a default period of 4 days and a max period of 14 days after which they are automatically deleted from the queue.
Message Size/Format: Can store text messages up to 256kb.
Message filtering/routing Policies: Does not support filtering/routing of messages. Message filtering has to be implemented in the MessageHandler component of the consumer service. Does not support the routing of the message to multiple consuming services. Each message will have to be sent to the SQS queue for every other Service that has to consume the message
Message Ordering: Messages are not guaranteed to be delivered in FIFO order. When a client polls for messages, SQS picks up messages from a random sample subset of servers due to which message ordering cannot be guaranteed.
Batching Policies: Messages can be requested in batches with a max batch size of 10 messages.
Scalability: Highly Scalable in terms of throughput. Latency is in the order of a few to several seconds as the message will have to be replicated before it is available for consumption and message consumption by the consumers is not real-time due to polling interval. Supports unlimited volume of queues/messages.
Availability: Supports High availability in case of server failures through replication of the queue across a cluster of servers. Chooses Availability over Consistency in the event of a network partition.

Benefits of Using SQS →

• Eliminates administrative overhead → SQS is a fully managed service provided by us in which no role of administrator came into play again and again. This is possible because once you create the queue, behind the scene was managed all the backend services need to keep uptime and high availability of this service.
• Reliable delivery messages → SQS can deliver the message of any volume with maximum throughput, without losing messages or requiring other services to be available.
• Keep sensitive data secure → YOu can use SQS provided by AWS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. It also has the facility to integrate with AWS key management (KMS) allows you to centrally manage the keys that protect SQS messages.
• Scale elastically and cost-effectively → It provides no limit on the number of messages per queue so if the messages are more then it scale-out automatically and scale-in. This whole process is managed by AWS engineers.

Some More Features of AWS SQS →

• Visibility timeout →
• SQS doesn’t delete the messages from the queue by itself once the message is received by the consumer because the system is distributed and there is no guarantee it will actually receive the message.
• Consumers should delete the message by themselves from the queue when they receive the message and process it.
• If the consumer downloads the message from the queue and processes the message and forgets to delete it then the message is available in the queue and if it again polls by any other consumer from the same distributed system then it will create multiple spamming and this needs to be prevented.
• SQS handles this type of situation with Visibility Timeout. SQS blocks the visibility of messages for some time when one consumer downloads the message. This visibility time is decided by the user on the basis of the time taken to process and pass the delete command to the queue for the message.
• Dead-letter-queue →
• This is also a great feature provided by SQS to prevent the processing of messages many times if the consumer forgets to pass the delete command once it receives and processes the message.
• If the consumer polls and again and again the same message then we can set delete or transfer the message somewhere so that the main queue became empty and next time if the consumer polls they don’t the same message.
• This concept is developed using Dead-letter-queue. In this, you can make one new queue and in the main queue attach this queue in option “dead-letter-queue” by setting a maximum threshold which means maximum how much time consumers can download the same message.
• Integration with Lambda –>SQS provides a great facility of integration with the lambda service of the aws.

Challenge Solved of BMW using AWS →

The BMW Group is using AWS for its new BMW 7 series cars to give drivers dynamically updated map information. BMW built its new car-as-a-sensor (CARASSO) service in only six months using Amazon Simple Storage Service ( Amazon S3 ), Amazon Simple Queue Service (Amazon SQS), Amazon DynamoDB, Amazon Relational Database Service (Amazon RDS ) and AWS Elastic Beanstalk.

BMW uses SQS for decoupling Receiver and Worker. The receiver keeps on adding messages to the SQS and Worker continuously polls the data and processes it and stores this data into the RDS database and MApData.