An Introduction to the AWS Serverless Ecosystem

Serverless is still considered to many as a new, emerging technology but the reality is that it isn’t actually that new at all. AWS first released Lambda back in 2014 and S3 even well before then. I personally started to use Lambda for compute in production environments around 2016, probably around the time when Containerisation technologies like Docker and Kubernetes were at their peak in terms of popularity.

What truly separated Lambda apart from Containerisation technologies was that suddenly developers were able to run their code in the cloud on-demand and without the need for a dedicated container or webserver to run it on. Magic!

Since then, and with Serverless computing remaining firmly at its core, the AWS Serverless ecosystem has since grown to encompass dozens of services that include object storage, data stores, API proxies and much more — I’ll touch on those later.

Want to read this story later? Save it in Journal.

So what is Serverless?

TLDR; It enables you, as a developer to shift more responsibility to your Infrastructure-as-a-Service (IaaS) platform allowing you to focus solely on feature development for your product or service, reducing the operational burden of managing a cluster of web servers or containers.

Why use Serverless?

This is where the fun starts and what I find myself talking about most days when advising on the use of serverless to private/public sector organisations in my role as a technology consultant. Serverless architectures offer greater scalability, more flexibility and a quicker time to market at a reduced cost.

The benefits here are focussed on Serverless computing but the benefits listed below apply to all of the serverless products available because no matter how shiny the product label, they’re all just applications that are running on bare metal web servers :)

There are many benefits to using Serverless for your applications’ compute but the 5 non-functional requirements I tend to focus on (in no particular order) are:

• Agility
• Cost
• Security
• Availability
• Elasticity/Scalability

Let’s talk about each one of these points.

Agility

The technology landscape is constantly evolving, and fast and so it’s important that developers are able to build platforms and services that allow their organisations to keep a competitive edge. Take away the burden of operating infrastructure and your engineering team is able to add more value to its users with new feature development.

Cost

Serverless architectures benefit from reduced operational costs. You’re effectively outsourcing your infrastructure operation to AWS and a fully serverless solution completely eliminates all operational costs. Most Serverless products in the AWS ecosystem follow a pay-as-you-go pricing model and so you only pay for the compute that you use rather than having a fleet of EC2 instances operational 24/7.

With Lambda, you’re charged based on the number of requests to your functions and the duration of those functions (the time it takes for your code to execute). If your workload is small enough, you can leverage the free tier which includes 1M free requests per month and 400,000 GB-seconds of compute time per month.

Let’s look at an example for Lambda. If you allocated 512MB memory to a function, executed it 3 million times in one month, and it ran for 1 second each time this would cost you a mere $18.34 which is calculated as:

• The monthly compute price is $0.00001667 per GB/s and the free tier provides 400,000 GB/s
• Total compute (seconds) = 3,000,000 seconds (3M x 1 second)
• Total compute (GB/s) = 3M x 512MB/1024 = 1,500,000 GB/s
• 1.5M GB/s minus 400,000 GB/s from the free tier = 1,100,000 GB/s
• 1.1M x $0.00001667 = $18.34 💰

Security

Having infrastructure in your stack that isn’t under your control scares the sh*t out of risk officers and rightly so.

With serverless computing, the cloud platforms manage the underlying infrastructure, the application platform and the operating system, including any security patches that are needed. The cloud platforms have hundreds of security engineers working on a daily basis to ensure the platforms are the safest they can be. You’re not paying any extra for these security features either, it’s in the cloud provider’s interest to be providing you as the customer with the best service possible.

As the customer of these cloud platforms are only responsible for the security of your application code, access management to the function and the storage and accessibility of sensitive data that your function might process. This model of operation is what is referred to as the shared responsibility model as shown below:

While you won’t be responsible for managing security surrounding the underlying hardware, AWS makes it easy for you to secure your application by applying granular IAM controls that can adopt the principle of least privilege, isolate network traffic using VPC’s, and so on.

Availability

High-availability and fault-tolerance are at the core of the Serverless offerings by cloud providers. Compute capacity is often maintained across multiple availability zones in different regions which helps mitigate your application code against machine failures, or worse, data-centre blackouts.

Elasticity/Scalability

Lambda automatically scales to the precise size of your workload. Every time a request is received to your code on-demand, the platforms quickly locate free capacity with their compute fleet to execute your code and capacity is dynamically allocated by automatically scaling your application horizontally.

With more “traditional” web application infrastructure, support for unexpected traffic spikes would need to be mitigated by you knowing what your upper/lower limits of demand are. Let’s say use the example of a web application running on EC2. As part of your infrastructure configuration, you’ll have to assign an auto-scaling group to your EC2 fleet that’s controlled by hard limits defined by those upper/lower thresholds. This means that any unexpected high workloads have the potential to cause timeouts or blackouts to users trying to access your application through these unusually busy periods.

With Lambda, any elasticity required to accommodate any sudden spikes in service demand is handled by AWS, greatly increasing the reliability and availability of your application. I’m not saying it’s a complete shield but the risk of your service going offline in an event like the above is greatly reduced and with Lambda, you’re also getting the best ratio of cost vs scaling capability.

N.B. Some Serverless services will still require you to tune their capacity (Kinesis Data Streams shards is an example) and the limits of each service vary.

I plan to go deeper into this topic with some hot tips on how to effectively manage concurrency in a future article — stay tuned for that!

What can I use Serverless for?

This is where it gets interesting. “Serverless” isn't just about compute capacity in the cloud to run your application code anymore. When you say the word “Serverless” to most people, Lambda is the first thing that tends to come to people’s minds. Amazon Simple Storage Service, or S3 as it’s most commonly known as is actually AWS’s most popular Serverless service and there are so many more services too.

Some of my favourites Serverless on AWS are:

• API Gateway — Build, maintain and secure API’s at any scale
• DynamoDB — NoSQL database with single-digit millisecond latency at any scale 🤯
• AppSync — Flexible GraphQL API’s
• Kineses — Gone are the days of Kafka headaches, use Kinesis to load and analyze (real-time) streaming data
• Step Functions — I can’t write an article that focuses on Lambda without mentioning Step Functions. While Step Functions aren’t mutually exclusive to Lambda, the two can be paired perfectly to orchestrate the coordination of components in your distributed applications.

You can view the full list of Serverless services that are currently in GA on AWS here and I’ll be writing more articles focussed on these products with configuration tips that will help you get the best bang for your buck when using them.

By opting to use any of these Serverless services you’re leveraging all of the non-functionals that are listed above. Awesome!

Is Serverless always the best solution?

No, and you shouldn’t be using Lambda for your compute just to be with the hipsters using the latest technology. As a Solution Architect, you have to be using your head and selecting the technology that’s the most appropriate to your use-case. There are cases when using Lambda and other tools from the serverless stack makes absolute perfect sense, but also when it doesn’t and so always approach your architecture designs with the view that there’s always an alternative.

Is Serverless only on AWS?

Nope, certainly not. I’ve focussed mainly on AWS in this article because that’s just the cloud platform I use on an almost-daily basis. The agility and constantly-evolving landscape of Serverless products offered by AWS also put them a league ahead of other cloud platforms and that’s not just my opinion. AWS has been named a Leader in the Gartner Magic Quadrant for Cloud Infrastructure & Platform Services (CIPS) for the 10th consecutive year.

Enjoyed this article? Please consider giving it a clap, you can give up to 50!

More from Journal

There are many Black creators doing incredible work in Tech. This collection of resources shines a light on some of us: