Cost-effective automation of Start/Stop of EC2 instances for non critical workload outside business hours.

Savings Cost challenge

SaveCostech, a startup dedicated to providing cost-saving solutions to businesses, identified an inefficiency in their infrastructure. Their non-critical workload hosted on Amazon EC2 was running 24x7, leading to unnecessary costs. This project aims to automate the start and stop of the EC2 instance to align with business hours, thereby optimizing operational costs.

MISSION:

As a DevOps Engineer your mission is to design an automation solution that ensures the EC2 instances for the non-critical workload are active only during business hours (8:00 AM to 5:00 PM, Monday to Friday), and remain dormant during off-hours and weekends.

Given that, the solution should leverage AWS services, including but not limited to AWS IAM, Amazon EC2, AWS Lambda, and Amazon Eventbridge ensuring both efficiency and security.

Services

AWS Lambda:

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications, and only pay for what you use.

Amazon Eventbridge(Formerly called Amazon CloudWatch Events)

Amazon EventBridge is a serverless event bus that makes it easy to connect applications using data from various sources, including integrated SaaS applications, custom applications, and AWS services. It can be used to build event-driven architectures, which enable decoupling of services and enhanced scalability.

AWS IAM:

AWS Identity and Access Management (IAM), allows us to specify who or what can access services and resources in AWS, centrally manage fine-grained permissions, and analyze access to refine permissions across AWS.

Amazon EC2:

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud.

Architecture solution

Goal:

1. Setup a scheduler using the Amazon Eventbridge service which will trigger a Lambda function.
2. Lambda function will temporarily assume the defined IAM role. This IAM role will have an IAM policy associated with it. The IAM Policy is what grants the Role permissions to access our EC2 instances.
3. The Lambda function will stop/start our Amazon EC2 instances as per the schedule we define in Eventbridge.

EXECUTION :

I will breakdown this project in 02 parts:

Part-1:

1. Create the IAM Policy and associate it with an IAM Role.
2. Create 02 Lambda functions: one to stop the EC2 instances and another to start the EC2 instance.
3. Test the Lambda functions by triggering them manually, to check if they have the desired effect on our EC2 instances

Part-2:

we will tie everything together, where we will be configuring the Amazon Eventbridge rules and defining the Cron entry that will trigger our Lambda functions to stop/start our EC2 instances.

Pre-requisites :

1. AWS free tier account is eligible
2. AWS IAM account (do not use root account) having:
3. admin privileges
4. access to AWS Management Console
5. Understanding of AWS Identity based policies In case you are not familiar with IAM Policies or need a quick refresher,
6. At least one EC2 instance created in the account (this is required for testing purpose)

Implementation:

Part-1:

Login to the AWS Management Console as IAM admin user

1. Create IAM policy and IAM role

1. 1 From the AWS Management Console navigate to IAM, Policy, Create Policy, select JSON

1. 2 Paste the below policy into the policy editor (delete all the pre-filled content) and click Next

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*",
      "Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "us-east-1"
                }
            }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "*",
      "Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "us-east-1"
                }
            }
    }
  ]
}

1. 3 Give the policy a meaningful name and description. Review the permissions and then click Click Policy

1. 4 From the left navigation panel, select Roles and then click the Create Role button

IAM, Role, Create, select AWS Service and Lambda as use case

Click to Next

From the policy dropdown list, select the policy we created in the previous step and click Next

Give the role a name and description

1. 5 Verify the trusted entity (should be Lambda) and the associated customer managed IAM Policy. Click Create Role

2. Create Lambda Function to stop EC2 instances

2.1 From the AWS Management Console navigate to Lambda and Create function

Click Create Function,Give the function a name and select the desired runtime.

2.2 Click Create Function

2.3 Scroll down a bit to reach the Code section. You will notice a pre-filled Code source section.

Delete the pre-filled contents and paste the below code in its place:

import boto3

region = 'us-east-1'
def lambda_handler(event, context):
    ec2_resource = boto3.resource("ec2", region_name=region)
    # Apply filter to check for EC2 instances in 'running' state
    instances = ec2_resource.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    print("EC2 region is: ", region)
    # Count running instances
    count = len(list(instances))
    print(f"{count} instances running")
    # Stopping EC2 instances and printing their ID and Name        
    for instance in instances:
        instance_name = [tag['Value'] for tag in instance.tags if tag['Key'] == 'Name'][0] if instance.tags else None
        print(f"Instance ID: {instance.id}, Name: {instance_name}")
        instance.stop()
        print(f"{instance.id} (Name: {instance_name}): Stopping!!! .")

2.4 Click Deploy

2.5 In the Configuration tab, select General configuration and click Edit

2.6 Change the Timeout value to 10 seconds and click Save

2.7 Click Test to test if our lambda function is working as expected
Configure the test event.

There is NO EC2 Instances running

Logs:

Test Event Name
1st-test
Response
null
Function Logs
START RequestId: 16151084-31fd-40ab-bec2-f3ba74fdcd85 Version: $LATEST
EC2 region is:  us-east-1
1 instances running
Instance ID: i-0a9bb4c931105ede1, Name: my-test-instance
i-0a9bb4c931105ede1 (Name: my-test-instance): Stopping!!! .
END RequestId: 16151084-31fd-40ab-bec2-f3ba74fdcd85
REPORT RequestId: 16151084-31fd-40ab-bec2-f3ba74fdcd85 Duration: 4147.16 ms Billed Duration: 4148 ms Memory Size: 128 MB Max Memory Used: 90 MB Init Duration: 278.04 ms
Request ID
16151084-31fd-40ab-bec2-f3ba74fdcd85

For this test you need to have at least one EC2 instances running in us-east-1 region:

In this case 01 EC2 is running in my account

Notice the time duration of our Lambda execution — it is 4 seconds. If I left the timeout value as default 3 seconds, My Lambda function would most likely have timed-out before completing the execution.

2.8 Go to the EC2 dashboard in the AWS Management Console (in the same region defined in your Lambda code) and check if the instances really stopped.

It worked!

3. Create the Start EC2 instance Lambda Function

3.1 Repeat steps to create another Lambda function and this time use the below Python code, which will start our EC2 instances.

use this code

import boto3
region = 'us-east-1'
def lambda_handler(event, context):
    ec2_resource = boto3.resource("ec2", region_name=region)
    # Apply filter to check for EC2 instances in 'stopped' state
    instances = ec2_resource.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['stopped']}])
    print("EC2 region is: ", region)
    # Count running instances
    count = len(list(instances))
    print(f"{count} instances in stopped state")
    # Starting EC2 instances and printing their ID and Name        
    for instance in instances:
        instance_name = [tag['Value'] for tag in instance.tags if tag['Key'] == 'Name'][0] if instance.tags else None
        print(f"Instance ID: {instance.id}, Name: {instance_name}")
        instance.start()
        print(f"{instance.id} (Name: {instance_name}): Starting!!! .")

3.2 Test the Lambda

Test Event Name
start_ec2_instance_test

Response
null

Function Logs
START RequestId: 96c06b24-84c5-4749-a866-4cdcd0447621 Version: $LATEST
EC2 region is:  us-east-1
1 instances in stopped state
Instance ID: i-0a9bb4c931105ede1, Name: my-test-instance
i-0a9bb4c931105ede1 (Name: my-test-instance): Starting!!! .
END RequestId: 96c06b24-84c5-4749-a866-4cdcd0447621
REPORT RequestId: 96c06b24-84c5-4749-a866-4cdcd0447621 Duration: 4213.63 ms Billed Duration: 4214 ms Memory Size: 128 MB Max Memory Used: 90 MB Init Duration: 207.85 ms

Request ID
96c06b24-84c5-4749-a866-4cdcd0447621

Review the execution result of the Lambda function.

• 01 EC2 instance was found in running state in the defined region us-east-1
• It was stopped with it’s EC2 instance names ID logged.
• The execution took ~4 seconds.

3.3 Verify if the EC2 instance started in the EC2 dashboard

Part-2:

We will tie everything together, We will be configuring the Amazon Eventbridge rules and defining the Cron entry that will trigger our Lambda functions to stop/start our EC2 instances.

4. configuring Amazon Eventbridge to trigger our Lambdas functions

Using Amazon EventBridge (formerly known as CloudWatch Events) is a popular way to schedule AWS Lambda functions. You can create EventBridge rules that trigger your Lambda function at specific times.

Here’s how you can set it up to start EC2 instances at 8:00 AM and stop them at 5:00 PM, Monday to Friday:

4. 1. Open the Amazon EventBridge Console:

Go to the Amazon EventBridge console in your AWS Management Console.

4.2. Create a New Rule:

• On the left panel, click on Rules.
• Click the Create rule button.

4.3. Configure the Rule for Starting EC2 instances:

• Name: Provide a name for the rule, e.g., StartEC2Instances.
• Description: (Optional) Add a description.
• Define Pattern: Choose Event Source as Schedule.
• Cron expression: Use the following cron expression to represent 8:00 AM UTC, Monday to Friday:

0 8 ? * MON-FRI *

• (Note: This is in UTC, so adjust the time if you want a different timezone.)
• Select Targets:
• Click on Add target.
• Choose Lambda function for the target type.
• For the Function, select the name of your Lambda function that starts the EC2 instances.
• Leave other options as default.
• Finally, click Create.

5. Configure the Rule for Stopping EC2 instances:

• Follow the same process to create another rule.
• Name: Provide a name like StopEC2Instances.
• Cron expression for 5:00 PM UTC, Monday to Friday:

0 17 ? * MON-FRI *

• Select your Lambda function that stops the EC2 instances as the target.
• Then, click Create.

Adjusting for Timezones:

The above cron expressions are in UTC +1. If your desired timezone isn’t UTC+1, adjust the hours in the cron expression accordingly.

Permissions:

Ensure that the EventBridge (or CloudWatch Events) service has permissions to invoke your Lambda function. This is typically done with an automatically created service role when you’re setting up the rule, but it’s something to keep in mind if you encounter permission issues.

Conclusion:

By following the above steps, your EC2 instances will be automatically started at 8:00 AM and stopped at 5:00 PM, from Monday to Friday, as per the defined schedule in EventBridge.

By automating the shutdown and startup of non-critical EC2 instances outside business hours, companies like SaveTech can efficiently reduce AWS costs. This approach, combined with other cost-optimization strategies, can lead to significant savings, especially for startups and small businesses.