5 Reasons SQS Lambda Triggers Are a Big Deal

Amazon recently announced AWS Lambda support for SQS triggers, and it was a big day for me. It went a little something like this:

Morning
Reveled in the announcement of the long-awaited, much anticipated feature. Hallelujah!

Afternoon
Scoured projects looking for workarounds I implemented in the absence of SQS trigger support.

Evening
Refactor, rejoice, repeat.

I find there’s no experience like getting rid of code and replacing it with infrastructure, except maybe simply removing code altogether. I hope you had a similar day.

SQS triggers for AWS Lambda

The introduction of SQS triggers for Lambda is exciting because event-driven systems can now natively use SQS. No more workarounds and reduced functionality. I’m no longer responsible for polling SQS queues and triggering Lambda function when queue activity is detected. The Lambda service does all of this for me now.

Simply put, SQS triggers:

• Trigger a Lambda function when on or when messages have been placed in the queue.
• Leverage existing retry logic and dead letter queues. If the Lambda function does not return success, the message will not be deleted from the queue and will reappear after the visibility timeout has expired.
• Allow built-in support for dead letter queues to be leveraged. After a specified number of failures, the message will automatically be placed in the DLQ for later inspection.

Before SQS triggers, workarounds with similar functionality could be achieved, but they often fell short. Many required additional infrastructure or code to accomplish. For example, Kinesis Data Streams provided seamless delivery of messages to Lambda but did so without including a mechanism for retries. SNS and SQS could be used in a Pub-Sub configuration to achieve similar behavior, at increased cost.

I have first-hand experience with this.

A practical use case

I recently built @AircraftML, a simple AI (artificial intelligence) system that can tell you what aircraft are in a picture. Tweet a picture to it, and it will let you know if the pic contains a Boeing 767, an Airbus A380, or any number of other modern commercial aircraft. Try it out.

When you tweet the image to @AircraftML, it first queues up the tweet for analysis. Then, a backend system takes your tweet off the queue, analyzes the image, and responds to you with the results.

Decoupling ingest from analysis provides all the benefits you’d expect in a distributed system: increased scalability, resilience to failure, high availability, etc.

Before SQS triggers, @AircraftML used a couple of different methods (in different iterations), including those mentioned above, Kinesis Data Streams and SNS/SQS. These methods worked, technically, but weren’t as resilient as I would have liked (failed attempts were just dropped on the floor) and were more complex than I wanted (like using two services instead of just one).

SQS triggers provided the perfect solution, one that I was able to easily use as a drop-in replacement for @AircraftML ingest purposes. I simply configured an SQS trigger to process single messages and set the SQS queue up to use a DLQ to deal with troublesome requests. I haven’t looked back since.

Real benefits of SQS triggers

SQS triggers are a game changer. They allow more robust decoupled, scalable, event-driven serverless architectures, and they also trigger some awesome side effects when used properly. Here are my top five:

SQS triggers allow me to reduce my infrastructure footprint. In many cases, my workarounds included the use of alternate services, sometimes with reduced functionality, sometimes augmented with additional code and services to bridge the functionality gap. Where I was once using multiple services (like Kinesis Data Streams and SQS or SNS and SQS), I can now use just a single SQS queue and achieve the same functionality.

SQS triggers cost less than equivalent workarounds. Replacing multiple services with a single one will reduce cost, both in terms of time and money. You’ll spent less time implementing and debugging costly workarounds and less money on redundant services and features.

SQS triggers make things simpler. Removing infrastructure, services, and code means fewer moving parts. Fewer moving parts means you’ll have an easier time maintaining, debugging, tracing, logging, and monitoring these systems. In my case, SQS triggers acted as a drop-in replacement, so I was able to realize these benefits immediately.

SQS triggers improve fault tolerance. OK, I’ll admit, I took the lazy route previously and didn’t completely match the functionality I can now get with SQS events. Failed execution meant tasks were dropped on the floor with no retry attempts. Lambda support for SQS triggers provides built-in retry functionality, with additional resilience and debuggability obtained using DLQs.

SQS triggers make end users happier. All of these benefits lead to happier users. Most of my systems are personal projects, but still, it’s nice to know that the end-user experience (even if it is just me) will be improved. I can spend less time dealing with infrastructure problems and more time on projects. Yay!

A couple of limitations

There are a couple of limitations that you should be aware of when considering the use of SQS as a Lambda trigger. Consider that:

Visibility timeouts are important. Messages are deleted after the Lambda function completes successfully. As such, the visibility timeout for queues and messages should be long enough that any tasks or jobs associated with the message can be completed in time. If you configure the SQS trigger to batch process messages and the combined time to complete all of the batched messages exceeds the message visibility timeout, some messages may reappear in the queue and trigger another Lambda function before they have been processed.

SQS triggers are not free (but you knew that already). SQS trigger support for Lambda is implemented pretty much like you think it would be. The Lambda service long-polls your SQS queues for you, then triggers your Lambda function when messages appear. SQS API calls made by Lambda on your behalf are charged at the normal rate. Also be aware that Lambda will scale the number of pollers (minimum is 5) up or down based on demand.

FIFO queues are not currently supported. If you’re using SQS FIFO queues, you’re stuck with previous workarounds. FIFO queues can be used as an event target for CloudWatch Events, but not as an event source for Lambda. Sorry.

Wrap-up

With Lambda support for SQS triggers, SQS finally joins the ranks of its poll-based brethren, Amazon Kinesis Data Streams & Amazon DynamoDB Streams, and can call itself a true Lambda event source. SQS is now a full member of the serverless cadre.

I hope everyone is as excited about using it as I am.