What is an Event-Driven Architecture?
An Event-driven architecture (EDA) is a software architecture paradigm promoting the production, detection, consumption of, and reaction to events. It uses events to trigger and communicate between decoupled services and is most commonly used while designing applications built with microservices. Event-driven architectures have three key components, event producers, event routers, and event consumers. For example, an event is a change in state, or an update, like an item being placed in a shopping cart on an e-commerce website.
To know about Event-driven architecture using AWS serverless, let’s look at a real-time use case (or a business problem) and how it can be resolved by designing an Event-Driven Architecture.
Scenario 1:
A group of college students wanted to share their videos online using a website that can be accessed by users that have private access to the website. The students don’t want to publish their video content on youtube but want to develop their own video sharing site. Some of the requirements are:
- Authenticated students can upload their musical, sporting or other event videos. (videos can be 4k quality)
- The application should process the video automatically into smaller sizes (For eg, 480p, 720p, 1080p)
- The website should be able to dispaly videos, manage playlists and channels
- The website should be able to store and retrieve data from the database (let’s not go in detail on the data part)
Architectural Considerations
The traditional way to approach this project is by using Monolithic Architecture (composed all in one piece, a tightly coupled architecture).
There are few problems with this Architecture, it is tightly coupled to the upload, processing, and storage components and if one of the components fails then the whole system fails to work. Whereas upload, processing, and storage components are separate tasks. The person uploading the videos should not care about processing or storing the video. The next problem with this architecture is it has to scale together, if the processing components need additional resources it will impact other components as well. Finally, this will have another problem, it will incur billing charges together.
N-Tier layerd architercture
To solve this, Monolithic architecture can be split into Tiered Architecture ( oven referred to as n-tier architecture). The components can be separated out into different tiers or can be hosted on different servers.
Tiered architecture can vertically scale and can manage resources in separate layers as per the needs independently and can be able to utilize horizontal scaling (additional instances can be added) by using the load balancers in each layer node. Even though the n-tier architecture can be physically separated, they are dependent on each other. If the processing server fails then the Upload and the Storage components cannot communicate with each other, which means layered – architecture is still tightly coupled. If the processing tier is slower to respond while processing tasks then it will impact customer experience. Additionally, the processing layer has to be provisioned (even there are no workloads for it to process) all the time (it requires synchronous communication between layers) incurring ongoing charges.
Queue Based Decoupled Architecture
The N-Tier architecture can be evolved by using Queues. The upload component queues the job in FIFO (first in first out) queue and uploads the video file in the storage. Based on the size of the queue the processing component scales up utilizing the auto-scaling group. When the task is complete and the queue is empty then the processor component will scale down. This is a simple yet powerful architecture where most of the tasks are performed asynchronously and all the components are decoupled from each other.
If will look deep into this architecture, there are individual components (or services) that are assigned certain tasks and only doing that task. The upload component is just uploading the video files, storage is storing files, queues are processing queues in order and processors are processing the video files whenever they appear on the queue. These smaller services that work independently but can be used by other services to do individual tasks are called micro-services and this architecture is also referred to as microservices architecture.
Microservices Architecture
A microservice architecture (a variant of the service-oriented architecture (SOA) style), arranges an application as a collection of loosely coupled services.
In this example Upload service (also called a producer) is a microservice, the processing service (also called a Consumer) is the microservice and store and manage services (works for both) is a microservice.
Producer: The services that produce data or messages
Consumer: The services that consume data or messages
The services that produce or consume the data can be identified as events. Queues can be used to communicate events between different services. But the disadvantage of microservices architecture is that it can be complicated because various nodes can be interconnected with each other increasing the complexity of the application.
Event-Driven Architecture
An event-driven architecture is a software architecture promoting the production, detection, consumption, and reaction to the events. An event can be defined as “a significant change in state”.
In an Event-driven architecture, there are Event Generators (or Event Producers) and on the other side, there are Event Consumers who are waiting for the next event (eg, save, upload, order, etc), and both of these don’t use many resources while waiting. To communicate between these event producers and event consumers there is a third component called event routers which contains an Event Bus that is responsible for a constant flow of information.
Mainly event-driven architecture only consumes resources while handling events and thus use of physical resources (servers) is not meaningful. Therefore the use of serverless architecture can be the best use case for most Event-Driven Architecture.
What is a serverless architecture?
As the name suggests, serverless architecture is a way to build and run applications and services without having to manage infrastructure. In AWS, even though the applications run on servers, all the server management is done by AWS. There is no need to provision, scale and maintain servers to run the applications, databases, and storage systems. Some of AWS serverless services are AWS Lambda, CloudWatch Events, AWS API Gateway, Simple Queue Service (SQS), Amazon Event Bridge, Amazon SNS, Step Functions, and so on. There is also a good article by AWS that can be useful for the serverless designs “10 Things Serverless Architects Should Know.
AWS Lambda
AWS Lambda is a serverless (function as a service) (faas) compute that allows running virtually any type of code without provisioning or managing servers. Lambda is an example of an Event-driven invocation (execution) where a code function can be executed and only be charged for the duration of the function is run (maximum Lambda execution time is 15 mins), there is no charge for hosting the function code itself. Lambda is an essential component of Serverless architecture. The main benefits of using Lambda functions are there is no need to manage servers, it can scale continuously, the cost is optimized with millisecond metering, and consistent performance at any scale. AWS Lambda (free tier) provides 1 million free requests per month and 400,000 GB-seconds of compute time per month which is a substantial amount of free service allocation that organizations can save while using serverless computing.
CloudWatch Events
Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources. For example, when an instance is started, stopped, or terminated these CloudWatch Events get triggered. CloudWatch Events can be incorporated in the serverless architecture to identify various system events and to decide the different actions accordingly.
EventBridge
This is the latest product introduced by AWS that is aimed to replace CloudWatch Events. Amazon EventBridge is a serverless event bus that makes it easier to build event-driven applications at scale using events generated from the applications within AWS and third parties, integrated Software-as-a-service (Saas) applications, and AWS services.
Application Programming Interface (API)
An application programming interface (API) is a piece of code hosted in a server to connect or communicate between different applications. It is a software interface, offering a service to other pieces of software. The user interface connects a computer program to a person or a user, whereas API connects one software application to another software application.
In AWS, API Gateway is a fully managed API endpoint service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. API Gateway can be consumed or used as a service and can act as a significant component in Serverless architecture.
Simple Notification Service (SNS)
Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication. It is a durable secure pub-sub messaging service and coordinates the sending and delivery of messages. SNS topics can be subscribed by a publisher or a system including Amazon SQS queues, AWS Lambda functions, and HTTPS endpoints, for parallel processing, and Amazon Kinesis Data Firehose.
AWS Step Functions
AWS Step Functions is a low-code visual workflow service used to orchestrate AWS services, automate business processes, and build serverless applications. Step Functions helps resolve some of the limitations of AWS Lambda or helps overcome some design decisions while working with AWS Lambda.
Lambda is a function as a service (FaaS) that can be invoked as simple small functions which have a maximum execution time of 15 minutes but cannot be used to perform series of tasks or as an application process. Lambda functions are stateless and cannot maintain any form of state. To overcome these limitations of Lambda, step functions use a State Machine which consists of a collection of states that can do work (Task states), determine to which states to transition next (Choice states), stop execution with an error (Fail states), and so on.
Practical Scenario – Implementing Event-driven Architecture Using AWS Serverless
Prepare a serverless system architecture that sends task reminder notifications on different time durations that can be set by the user. The duration of the task can run more than 20 mins at times. The system should be able to send 3 different types of notifications (email only, SMS only, or both) depending on the client’s request.
This architecture hosts a static website in an S3 bucket and it executes a lambda function using AWS API gateway. Then API Gateway requests the State Machine to execute tasks based on the state change, first state is to wait until given time from the user, then choose between the three different types of notifications and execute one or both lambda functions accordingly. The lambda functions will use Simple Email Service SES for email notification and Simple Notification Service (SNS) can be directly called from the state machine for SMS notification.
Architecting in AWS
Step 1 – Create Lambda execution role
Create an IAM execution role for Lambda ( For steps to create an IAM execution role for Lambda check: Know how to restart EC2 instance using Lambda function in AWS ). Download the cloudformation template (LambdaExecutionRole.yaml ), upload it in S3, and Create Stack.
Go to the IAM Management Console and check the Lambda execution role is created successfully.
This role has permissions to access Amazon SES, SNS, and States (The state machines of step functions) services and it logs in Cloudwatch Logs.
Step 2 – Create Lambda Function for Send Email
Go to Lambda Management Console > Create Function > Author from scratch
Function Name: Lambda_for_email_reminder
Runtime: Python 3.9 (or current version)
Change default execution role > Use an existing role > LambdaRole ( select the lambda role created earlier) > Create Function
In the code section, enter the following code
import boto3, os, json
# Verify a email address in SES Management console
FROM_EMAIL_ADDRESS = 'test@gmail.com' #use SES verified email address to send from
# Initiate a boto3 client to send email.
# Boto3 is the AWS software development kit (SDK) for Python
ses = boto3.client('ses')
def lambda_handler(event, context):
# Print event data to logs ..
print("Received event: " + json.dumps(event))
# Publish message directly to email, provided by Email TASK
# Compose email and send
ses.send_email( Source=FROM_EMAIL_ADDRESS,
Destination={ 'ToAddresses': [ event['Input']['email'] ] },
Message={ 'Subject': {'Data': 'Your Nominated Task has been started-'},
'Body': {'Text': {'Data': event['Input']['message']}}
}
)
return 'Success!'Deploy the code > copy the Function ARN in a notepad so that it can be used at later stages
Step 3 – Create a role for State Machine
Use (Cloudformation template for State Machine role to invoke lambda, SNS, and log in cloudwatch) and upload in S3 and create a stack. Check the role in IAM Management Console.
Setp 4 – Create State Machine
Go to Step Functions > State Machines (from the left menu) > Create State Machine > Write your workflow in code
Type > Standard
In Definition write the following code
{
"Comment": "Task reminder using SNS service for Sms and Lambda for sending email via SES",
"StartAt": "Timer",
"States": {
"Timer": {
"Type": "Wait",
"SecondsPath": "$.waitSeconds",
"Next": "ChoiceState"
},
"ChoiceState": {
"Type" : "Choice",
"Choices": [
{
"Variable": "$.preference",
"StringEquals": "email",
"Next": "EmailOnly"
},
{
"Variable": "$.preference",
"StringEquals": "sms",
"Next": "SMSOnly"
},
{
"Variable": "$.preference",
"StringEquals": "both",
"Next": "EmailandSMS"
}
],
"Default": "NoChoice"
},
"EmailOnly": {
"Type" : "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:ap-southeast-2:371665065798:function:Lambda_for_email_reminder",
"Payload": {
"Input.$": "$"
}
},
"Next": "NextState"
},
"SMSOnly": {
"Type" : "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message": {
"Input.$": "$.message"
},
"PhoneNumber.$": "$.phone"
},
"Next": "NextState"
},
"EmailandSMS": {
"Type": "Parallel",
"Branches": [
{
"StartAt": "ParallelEmail",
"States": {
"ParallelEmail": {
"Type" : "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "arn:aws:lambda:ap-southeast-2:371665065798:function:Lambda_for_email_reminder",
"Payload": {
"Input.$": "$"
}
},
"End": true
}
}
},
{
"StartAt": "ParallelSMS",
"States": {
"ParallelSMS": {
"Type" : "Task",
"Resource": "arn:aws:states:::sns:publish",
"Parameters": {
"Message": {
"Input.$": "$.message"
},
"PhoneNumber.$": "$.phone"
},
"End": true
}
}
}
],
"Next": "NextState"
},
"NoChoice": {
"Type": "Fail",
"Error": "DefaultStateError",
"Cause": "No Choice Made!"
},
"NextState": {
"Type": "Pass",
"End": true
}
}
}The visual representation of the State Machine looks like following
Note: “FunctionName”: “arn:aws:lambda:ap-southeast-2:371665065798:function:Lambda_for_email_reminder”, should be your own ARN.
Press Next > Specify Details
Name: TaskReminderStateMachine ( as per your organisation naming standard)
Permissions > Choose existing role > StateMachineRole ( the role created in step 3)
Logging > Log Level > ALL > Create State Machine
(Copy the ARN of the STATE Machine in a notepad for future use)
Step 5 – Create Lambda function for API gateway
Go to Lambda Management Console > Create Function
Function Name > Lambda_for_API (or as per your naming standards)
Runtime > Python 3.9 (or latest)
Change default execution role > Use an existing role > Lambda Role ( the role created in Step 1) > Create Function
Enter following code > Deploy
import boto3, json, os, decimal
## NOTE use the ARN for your State machine below
SM_ARN = 'arn:aws:states:us-east-1:371665065798:stateMachine:TaskReminderStateMachine'
sm = boto3.client('stepfunctions')
def lambda_handler(event, context):
# Print event data to logs ..
print("Received event: " + json.dumps(event))
# Load data coming from APIGateway
data = json.loads(event['body'])
data['waitSeconds'] = int(data['waitSeconds'])
# Check the parameters coming through from API gateway
checks = []
checks.append('waitSeconds' in data)
checks.append(type(data['waitSeconds']) == int)
checks.append('preference' in data)
checks.append('message' in data)
if data.get('preference') == 'sms':
checks.append('phone' in data)
if data.get('preference') == 'email':
checks.append('email' in data)
# if any checks fail, return error to API Gateway to return to client
if False in checks:
response = {
"statusCode": 400,
"headers": {"Access-Control-Allow-Origin":"*"},
"body": json.dumps( { "Status": "Success", "Reason": "Input failed validation" }, cls=DecimalEncoder )
}
# If none, start the state machine execution and inform client of 2XX success :)
else:
sm.start_execution( stateMachineArn=SM_ARN, input=json.dumps(data, cls=DecimalEncoder) )
response = {
"statusCode": 200,
"headers": {"Access-Control-Allow-Origin":"*"},
"body": json.dumps( {"Status": "Success"}, cls=DecimalEncoder )
}
return response
# This is a workaround for: http://bugs.python.org/issue16535
# Solution discussed on this thread https://stackoverflow.com/questions/11942364/typeerror-integer-is-not-json-serializable-when-serializing-json-in-python
# https://stackoverflow.com/questions/1960516/python-json-serialize-a-decimal-object
# Credit goes to the group :)
class DecimalEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, decimal.Decimal):
return int(obj)
return super(DecimalEncoder, self).default(obj)(Copy the ARN of the function in a notepad for future use)
Step 6 : Create API Gateway
Go to API Gateway Console > APIs (from the left menu) > REST API > Build
Create new API > New API
API Name > TaskReminderAPI > Regional > Create API
Actions > Create Resource
Resource Name: TaskReminder (or your naming standard) > Enable API Gateway CORS > Create Resource
Select > /taskreminder > Actions > Create Method > POST (under options) > click on the TICK icon
Use Lambda Proxy Integration > Tick
Lambda Function: Lambda_for_API (lambda function created in step 5)
Use Default Timeout > Tick
This API Gateway should integrate with the Lambda function to invoke it.
Finally, Save > Ok
Deploy The API
Actions > Deploy API
Deployment Stage > New Stage
Stage Name: Production
Stage Description: Production Environment
Deployment Description: Production Environment
Deploy
Copy the Invoke URL (https://5f0424dxoj.execute-api.ap-southeast-2.amazonaws.com/Production) to be used by client applications to invoke the API.
Step 7 – Client Side web application that invokes API Gateway
Create an S3 bucket to host a static website
( check How to host a static website in AWS S3 Bucket? )
Upload the static website that invokes the API gateway (download code). Open the website hosted in S3.
Now, let’s test it.
Go to Step Functions Console > State Machines > TaskReminderStateMachine
There are no executions and no logs generated yet.
Test in the website and enter values
Seconds: 60
Message: This is my first message as
Email: myemailaddress@gmail.com
Mobile Number: My mobile no
State Machine Status:
Check Cloudwatch>Log Groups> TaskReminderStateMachine-Logs
The serverless Event-Driven architecture is up and running.
Remember to clean up the account to avoid any charges.
S3 > delete bucket.
API Gateway > delete TaskReminderAPI
Lambda > Functions > delete Lambda_for_email_reminder & Lambda_for_API
Step Functions> Delete TaskReminderStateMachine
Cloud Formation > Delete State Machine Role and Lambda Role Stack