Seamless Order Automation: Leveraging AWS Lambda, S3, and DynamoDB for Next-Gen Processing

5 min readJun 11, 2024

What did I use to build this environment?

AWS console
AWS cloud9

What was built?

S3 Bucket
Dynamodb Table
Lambda Function
Iam Role
AWS cloud9 environment

Current Scenario

Imagine that SemoFarm Ltd is a leading agricultural company that supplies fresh products to various regions. They receive daily orders in the form of CSV files from different retailers and distributors. These CSV files contain information about the types, quantities, and delivery details of the products requested.

Problem Statement:

SemoFarm Ltd is looking to automate the order processing system. They want to upload the CSV files to Amazon S3 and then process the data to write the order details to a DynamoDB table. This system must be efficient, scalable, and serverless to handle the varying volume of orders.

Your Mission:

Create a solution to build a serverless system using AWS Lambda to load CSV files from S3, process the data, and write the output to a DynamoDB table.

Implementation

Execution flow

1. Set Up Amazon S3 Bucket:

Create an S3 bucket named semofarm to store the CSV files.

S3 Bucket: semofarm

2. Set Up DynamoDB Table:

Create a DynamoDB table named semofarm-orders with appropriate attributes for storing order details.

Define order_id for the partition key (String).

3. Write the AWS Lambda Function:

Write a Lambda function to be triggered by an S3 event when a new CSV file is uploaded.
Read the CSV file from the S3 bucket, parse the data, and transform it into a format suitable for DynamoDB.
Use the AWS SDK to write the processed data to the DynamoDB table.
Ensure proper error handling and logging.

Use Cloud 9 to create the function code

mkdir freshproducts 
cd freshproducts
pip install virtualenv
virtualenv -p python3 venv 
source venv/bin/activate

Create a new file inside the folder freshproducts named lambda_function.py

Python Script for AWS Lambda Function (lambda_function.py):

import boto3
import csv
import os
from io import StringIO

def lambda_handler(event, context):
    s3_client = boto3.client('s3')
    dynamodb_client = boto3.resource('dynamodb')    # Get the S3 object details from the event trigger
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']    # Read the CSV file from S3
    file_obj = s3_client.get_object(Bucket=bucket, Key=key)
    csv_content = file_obj['Body'].read().decode('utf-8')    # Define the DynamoDB table
    table = dynamodb_client.Table('semofarm-orders')    # Read the CSV content
    csv_reader = csv.DictReader(StringIO(csv_content))    # Iterate through the CSV and write to DynamoDB
    for row in csv_reader:
        order_id = row['order_id']
        product = row['product']
        quantity = int(row['quantity'])
        delivery_date = row['delivery_date']        # Write to DynamoDB
        table.put_item(
            Item={
                'order_id': order_id,
                'product': product,
                'quantity': quantity,
                'delivery_date': delivery_date
            }
        )    return {
        'statusCode': 200,
        'body': 'CSV processed successfully!'
    }

Requirements file (requirements.txt)

boto3
StringIO

Create the deployment file for the Lambda function

cd venv/lib/python*/site-packages/
zip -r ../../../../lambda.zip .  # create the lambda.zip files so that we can have all the librairies required to run the functionrun
cd ../../../../                  #go back to the main directory
zip -g lambda.zip lambda_function.py *.txt   # added the lambda_function to the lambda zip file we have created

This set of commands does the following:
Navigate to the site-packages folder within your virtual environment. This is where all the installed packages are located.
Create a zip file named lambda.zip and recursively add all the contents of the site-packages folder to it.
Move back to the root directory of your project.
Update the existing lambda.zip file (with -g flag) by adding the lambda_function.py file and all .txt files from the current directory to the zip.

Now you should have a lambda.zip file that includes all the necessary dependencies from the virtual environment as well as your Lambda function code and any additional text files you want to include. You can then upload this zip file as the deployment package for your AWS Lambda function.

- Create the AWS Lambda resource on AWS console :

name: freshproducts
runtime : python 3.11

- Deploy the Code using the AWS CLI

Upload and publish the code using AWS CLI

aws lambda update-function-code \
  --function-name freshproducts \
  --zip-file fileb://lambda.zip \
  --publish

4. Configure S3 Event Trigger & IAM role:

Set up an event in the S3 bucket to trigger the Lambda function whenever a new CSV file is uploaded.

Make sure the appropriate IAM roles are set for the Lambda function to access the S3 bucket and write to the DynamoDB table.
Go to Permissions and open the role name that lambda function create

5. Deploy and Test:

Upload a sample CSV file to the S3 bucket and verify that the Lambda function is triggered.
Check the DynamoDB table to ensure that the data has been written correctly.

Sample CSV File:

Upload a sample CSV file to the S3 bucket and verify that the Lambda function is triggered.
Check the DynamoDB table to ensure that the data has been written correctly.

Here’s a sample CSV file that could be used to test the system. You can create a file called orders.csv with the following content:

order_id,product,quantity,delivery_date
102 Oranges 43   15/06/2024
103 Apples 22    17/06/2024
104 Onions 50   19/06/2024
105 Potatoes 60   20/06/2024

This CSV file represents four orders, each containing the order ID, product name, quantity, and delivery date.

You can upload this CSV file to the S3 bucket that you’ve configured to trigger the Lambda function and then verify that the data is written to the DynamoDB table.

Evaluation:

Confirm that the system correctly processes uploaded CSV files.
Verify that the processed data is accurately written to the DynamoDB table.
Evaluate the efficiency and scalability of the solution.

Conclusion:

This challenge offers a real-world scenario where AWS Lambda, Amazon S3, and DynamoDB are used to automate a critical business process. By working on this challenge, you will gain hands-on experience in building serverless architectures, processing CSV files, and integrating various AWS services.