ProdESP32 #9: AWS Basic Ingest

If you use the AWS IoT Core MQTT Broker you need to be aware of this very simple trick which could lead to significant savings on your monthly bill.

In AWS IoT Core the way you receive and process MQTT messages is with IoT Rules. AWS IoT rule actions allow you to invoke a wide variety of AWS services when a message comes through. A single rule can even dispatch to multiple services simultaneously.

Example Scenario

When setting up a rule you define what topic the rule will apply to. For example, you may create a rule that handles messages sent to topics with the structure sensors/[company]/[building]/[floor]/[sensor_type]. For example, a temperature sensor installed at an Acme Inc location may send a message to sensors/acme/bldg1/floor1/temp. This is a contrived example and, in production, you would likely set up multiple rules instead of a single rule. However, for the sake of what we are talking about here, cost, it doesn’t matter whether you have 50 rules or 1 rule. The focus is on the number of messages coming into the broker from devices.

To have concrete numbers let’s assume, on average, your devices are installed two to a floor in a commercial install. They have sold well and you are now in 3000 buildings averaging 3 floors per building. In total, you have 18000 devices in the field.

18000 devices reporting every minute for a month amounts to approximately 777600000 published messages into the broker. That’s almost 800 million messages a month and that assumes none of the messages are repeated or delivered to other subscribers. This is just the count of messages coming into the AWS IoT Broker rule engine.

The Expensive Way

If all of your devices are publishing to the single contrived topic structure above you are charged the first-billion message rate of $1/1,000,000 per message. Just in published message costs alone you are spending $777 dollars a month! 🤑

You can see this isn’t going to scale very well. Maybe you need to start publishing twice as often. You just doubled your monthly message publishing bill to over $1500. Eighteen thousand devices may seem like a lot but it’s really not that many if your product becomes very popular and is selling well.

At even a modest scale like this, small changes to how you handle MQTT message ingest can result in large swings in cost. Luckily, for message ingest, there is a dead simple trick to save you a ton.

AWS Basic Ingest

Amazon offers a feature in IoT Core called Basic Ingest. Per the docs:

Basic Ingest optimizes data flow by removing the publish/subscribe message broker from the ingestion path.

Implementing Basic Ingest is trivial. All you have to do is use the special Basic Ingest prefix of $aws/rules/[rule_name] on your topic. That’s it! Just by changing the topic to have a special prefix you switch from regularly billed messages to Basic Ingest billed messages. That’s not even the best part though.

The best part is this…all messages published with a Basic Ingest topic structure are FREE!!! That $777/month bill above for message publishing goes to $0. That’s right, you pay absolutely nothing for them.

Revisiting Our Example

Given our example above, a device published to the topic sensors/acme/bldg1/floor1/temp. If the name of the IoT rule for this topic is “SensorManager” you simple change the topic your devices use for publishing to the following:

$aws/rules/SensorManager/sensors/acme/bldg1/floor1/temp

That’s it! Every message from that device will now use Basic Ingest and bypass the broker during ingestion and the message will be sent directly to whatever services you have configured such as Lambda or Kafka. The messages are now free to ingest.

Why Not Always Use It?

At this point you may be asking “Well dang, why would I ever NOT use Basic Ingest?” This is a great question and one you can answer pretty easily by looking at your product requirements. Remember, Basic Ingest completely bypasses the message broker and delivers the message payload directly to your configured services.

But what if the messages need to get to other subscribers. Maybe some of your sensors need to know about data from other sensors. Your new HVAC widget installed in each building needs to get those temperature updates from all of the floors in the building so it subscribes to sensors/acme/bld1/#. If this is the case you can no longer use Basic Ingest for all of your sensors. You MUST go through the broker so it can publish the message out to your HVAC widget subscriber.

You can be creative here though. Considering your costs you may look at another way to get data to your HVAC widget. Perhaps it subscribes to a completely different topic that you publish to much less frequently from a Lambda function. You could aggregate all readings from a building and then, once every 10 minutes, send a single message to the HVAC widget. These are production design considerations and tradeoffs you need to consider.

Just know Basic Ingest is a tool in your toolbox that can have a massive impact on your costs.

Production Pointers

• Use Basic Ingest everywhere possible
• Consider ways to refactor flows to take advantage of Basic Ingest
• Evaluate how often your devices need to report
• Consider ways to consolidate messages to reduce message count

Conclusion

Using Basic Ingest in AWS is part of the Scalable Pillar of Production. While many AWS services are affordable at small scale, the costs can quickly get away from you. This simple change to your topics can result in huge savings.

Join the community and get the weekly Production ESP32 newsletter. Concise, actionable content right in your inbox.