Cloud Cost Optimization using AWS Lambda

Cost optimization is crucial for maintaining efficient and affordable cloud infrastructure in the ever-evolving cloud landscape. AWS provides numerous services and tools to help manage and optimize costs. One effective cost-saving measure is identifying and deleting stale Amazon Elastic Block Store (EBS) snapshots. EBS snapshots are point-in-time backups of EBS volumes, and over time, they can accumulate and incur significant storage costs if not managed properly.

In this blog, we'll walk through a practical project to create an AWS Lambda function that identifies EBS snapshots no longer associated with any active EC2 instance and deletes them to save on storage costs. This Lambda function will utilize AWS SDK for Python (Boto3) to interact with AWS services.

Prerequisites

Before we dive into the implementation, ensure you have the following:

An AWS account.
Basic understanding of AWS EC2, EBS, and Lambda.
AWS CLI configured on your local machine (optional for testing locally).
IAM role with appropriate permissions to manage EC2 instances, EBS volumes, and snapshots.

Project Overview

Step 1: Create an EC2 Instance and EBS Volume

First, create an EC2 instance (free-tier specs) with an EBS volume. This volume will be used to create a snapshot for demonstration purposes.

The instance is up and running, now check the EBS volume attached to this instance.

Step 2: Create a Snapshot for the EBS Volume

In the EC2 dashboard, navigate to the Snapshots section and create a snapshot for the newly created EBS volume.

Mention the resource type as volume, specify the newly created volume ID, and add a description for the snapshot.

Click on Create Snapshot.

Step 3: Create a Lambda Function

Next, create a Lambda function to identify and delete stale EBS snapshots. Navigate to AWS Lambda: Go to Services -> Lambda -> Create Function.

Configure the Function:

Function Name: cost-optimization-ebs-snapshot
Runtime: Python

Create the Function with default permissions (we'll update permissions later).

Click on Create Function.

The function is successfully created.

Step 4: Add Python Code to the Lambda Function

Replace the default code with the following Python script, which will identify and delete stale EBS snapshots:

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    # Get all EBS snapshots
    response = ec2.describe_snapshots(OwnerIds=['self'])

    # Get all active EC2 instance IDs
    instances_response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    active_instance_ids = set()

    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])

    # Iterate through each snapshot and delete if it's not attached to any volume or the volume is not attached to a running instance
    for snapshot in response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')

        if not volume_id:
            # Delete the snapshot if it's not attached to any volume
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted EBS snapshot {snapshot_id} as it was not attached to any volume.")
        else:
            # Check if the volume still exists
            try:
                volume_response = ec2.describe_volumes(VolumeIds=[volume_id])
                if not volume_response['Volumes'][0]['Attachments']:
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as it was taken from a volume not attached to any running instance.")
            except ec2.exceptions.ClientError as e:
                if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
                    # The volume associated with the snapshot is not found (it might have been deleted)
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as its associated volume was not found.")

Refer: https://github.com/nishankkoul/AWS-Lambda-Cost-Optimization/

After pasting the code, click on Deploy. Since we will manually trigger the function, click on Test, set the event name as “test” and then click on Save.

Step 5: Configure Lambda Execution Time

Increase the Lambda function's execution time to 10 seconds to ensure it completes its task. Navigate to Configuration -> General Configuration -> Edit -> Save.

Step 6: Update IAM Permissions

Update the Lambda function's IAM role to allow it to describe and delete snapshots, as well as list EC2 instances and volumes:

Go to Configuration -> Permissions.
Click on the role name to open it in a new tab.
Add the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeSnapshots",
                "ec2:DeleteSnapshot",
                "ec2:DescribeVolumes"
            ],
            "Resource": "*"
        }
    ]
}

Step 7: Test the Lambda Function

Manually trigger the Lambda function to test it. Ensure that the snapshot created earlier is not deleted because it is associated with an active EC2 instance and EBS volume.

The function has been executed successfully. If we go to the EC2 dashboard and see the Snapshot, it would still be there since the EC2 instances are active and as per our Python code, only those Snapshots will be deleted, which are not associated with an active volume/instance.

Step 8: Delete the EC2 Instance and Associated Volume

Delete the EC2 instance and verify that its associated volume is also deleted. Trigger the Lambda function again to confirm that the snapshot is now identified as stale and deleted.

After the instance and its associated volume have been deleted, come back to the Lambda dashboard and click on Test again.

The function has been successfully executed and now let’s see if the Snapshot has been deleted.

The Snapshot has been successfully deleted.

Conclusion

Congratulations! You have successfully created an AWS Lambda function to optimize storage costs by identifying and deleting stale EBS snapshots. This automated approach helps maintain efficient cloud infrastructure and reduce unnecessary storage expenses.

Additional Tips

Automation: Schedule the Lambda function to run periodically using AWS CloudWatch Events.
Cost-Effective Storage: Consider moving snapshots to Amazon S3 Glacier for long-term, low-cost storage.
Monitoring and Alerts: Set up monitoring and alerts to track storage usage and optimization results.

By following these steps, you can implement an effective cost optimization strategy for your AWS environment, ensuring you only pay for the resources you actively use.