🤦 LFMM: Know Your AWS Quotas and Limits

This is the first in what will likely be many posts where you get to LFMM (“Learn From My Mistakes”). These are mistakes I have made in the past or just general production misfortunes I have experienced over the years. Despite your best effort things will go sideways and you’ll have to scramble and recover and do your best to limit the blast radius. Use these posts to avoid a mistake or problem without having to feel the pain and stress of actually going through it.

This edition is about AWS quotas and limits, specifically around SMS but the takeaway can apply to any cloud provider and quota limit as all will have similar feature sets.

If you intend to send SMS messages as part of your product and you are using AWS then, as of this writing, you may be using Amazon SNS where you can set SMS messaging preferences. In those preferences you can set an Account spend limit which is an optional preference that allows you to limit the amount of money you spend each month sending text messages.

Now, you may think, “Just don’t set a limit”. While this is technically possible, the option is meant to protect you from yourself so you don’t break the bank in the case of a bug. In production it’s better to have a clear understanding of what your SMS monthly “load” will be and budget accordingly. In AWS there are two values to understand when it comes to SMS messaging via SNS. The account spend limit quota and the account spend limit preference. The quota is the maximum upper limit you can set the preference to. The spend limit preference is the dollar amount at which text messages will stop being sent by the system. Both are per-region settings.

A quick example. Let’s assume your account has a $100 quota max and the current spend limit is set to $10. Once you have sent $10 worth of messages within a month AWS will not send any new SMS messages and they will be logged with the error No quota left for account. But since your quota max is $100 you can increase the spend limit preference to anything up to $100 and your SMS messages will begin to flow again immediately. If you reach the $100 limit, however, you must submit a quota increase request ticket to AWS support. This can take up to several days to process.

This seems pretty straight forward. You will know, based on usage and text message rates, how much you are likely to spend in a month and can set the preference accordingly. As your company grows you can preemptively request quota increases to give you more headroom.

How It Can Go Sideways

Enter your international team members. You found an incredible tester in Germany. They are fantastic. Then one day you start getting customer complaints that SMS messages are not longer sending. You haven’t onboarded a ton of new customers and the system isn’t sending any more messages than usual. What happened? Remember that fantastic tester. Well, those text messages that cost a fraction of a penny to send within the US cost over 10x as much to send to Germany via SNS. That single tester just blew through your entire spend limit for the month and shut down all outbound SMS messaging 🤯.

Production Takeaways

Here are the key takeaways to apply to your production setup:

• Always make sure the upper spend limit quota has plenty of headroom compared to your spend limit preference. This gives you room to rapidly adjust the spend limit without having to request a quota increase from AWS.
• Keep a careful eye on both your max quota and spend limit preference value to make sure they are sufficient. You can use your monthly billing statement to help with this.
• Create an SMS spending alarm. This is an automated way to be notified when you are approaching your limit.
• Create an alarm based on delivery rate. You will never have 100% but if it decreases rapidly in a short time you know you have a problem that needs immediately attention.
• Be mindful of WHERE you may be sending messages and how that will affect your spend rate and therefore how fast you will reach your spend limit each month.

Do you have any fun stories regarding quotas or limits in AWS? The community would love to hear about it in the comments.

Join the community and get the weekly Production ESP32 newsletter. Concise, actionable content right in your inbox.