How to integrate Drizzle into AWS Serverless Environment

In this blog post, we will integrate Drizzle ORM into an existing AWS serverless environment using AWS Cloud Development Kit (CDK) in TypeScript. Our existing project includes an API Gateway, Lambda function (get-user-lambda), a PostgreSQL RDS database, and Secrets Manager credentials.

Code First

I know you came also for the code. So here is the code first. We will go through the code step by step in the following sections. 🌐 https://github.com/XamHans/aws-serverless-drizzle-cdk-poc

Video Tutorial

In case you enjoy video more than reading, here is my youtube video:

Prerequisites

• AWS CLI installed and configured
• Node.js installed
• AWS CDK installed
• Basic knowledge of AWS services and CDK

Existing Project Setup

Our project structure includes:

• API Gateway: To expose our Lambda function.
• Lambda Function: Handles requests and interacts with the PostgreSQL database.
• PostgreSQL RDS: Our database.
• Secrets Manager: Manages database credentials.
• CDK: Our infrastructure recipes are written in TypeScript.

Here is our one and only stack that contains all the resources:

import * as cdk from 'aws-cdk-lib';
import * as apigateway from 'aws-cdk-lib/aws-apigateway';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import * as rds from 'aws-cdk-lib/aws-rds';
import { Construct } from 'constructs';
import * as dotenv from "dotenv";
import path = require('path');

dotenv.config();

export class AwsDrizzleApiStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Define the Lambda function
    const getUserLambda = new NodejsFunction(this, 'UserGetHandler', {
      runtime: lambda.Runtime.NODEJS_18_X,
      entry: path.join(__dirname, '../lambdas/user-get.ts'),
      handler: 'handler',
      initialPolicy: [
        new iam.PolicyStatement({
          actions: ['secretsmanager:GetSecretValue'],
          resources: [`arn:aws:secretsmanager:${this.region}:${this.account}:secret:${process.env.POSTGRES_DB_SECRET}-*`],
        })
      ],
      environment: {
        POSTGRES_DB_SECRET: process.env.POSTGRES_DB_SECRET ?? 'connectionUrl',
      }
    });

    // Define the API Gateway
    const api = new apigateway.RestApi(this, 'UsersEndpoint', {
      restApiName: 'users-api',
    });

    // Define a resource and method for the API Gateway
    const drizzleResource = api.root.addResource('users');
    drizzleResource.addMethod('GET', new apigateway.LambdaIntegration(getUserLambda));

    // Define the VPC for the RDS instance
    const vpc = new ec2.Vpc(this, 'RdsVpc', {
      maxAzs: 2,
      natGateways: 1,
    });

    if(!process.env.POSTGRES_DB_SECRET) {
      throw new Error('POSTGRES_DB_SECRET must be defined in the environment');
    }

    // Generate a secret for the database
    const credentials = rds.Credentials.fromGeneratedSecret('dbuser', {
      secretName: `${process.env.POSTGRES_DB_SECRET}`,
    });

    // Allow Lambda function to access Secrets Manager
    credentials.secret?.grantRead(getUserLambda);

    const securityGroup = new ec2.SecurityGroup(this, 'security-group-postgres', {
      vpc: vpc,
      allowAllOutbound: true,
    });

    securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(5432), 'allow public access to the RDS instance', true);

    // Define the RDS instance
    const dbInstance = new rds.DatabaseInstance(this, 'RdsInstance', {
      engine: rds.DatabaseInstanceEngine.postgres({
        version: rds.PostgresEngineVersion.VER_14,
      }),
      instanceType: ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE3, ec2.InstanceSize.MICRO),
      vpc: vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
      publiclyAccessible: true,
      credentials: credentials,
      allocatedStorage: 20,
      backupRetention: cdk.Duration.days(0),
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      securityGroups: [securityGroup],
    });

    // Output the database endpoint
    new cdk.CfnOutput(this, 'DbEndpoint', {
      value: dbInstance.dbInstanceEndpointAddress,
    });

    // Output the secret name
    new cdk.CfnOutput(this, 'Secret Name', { value: credentials.secretName ?? '' });
  }
}


Now lets deploy the stack with the following command:

```bash
npx cdk deploy

Integrating Drizzle ORM

Now that your infrastructure is set up, let's integrate Drizzle ORM into our project. Drizzle has exactly 0 dependencies! Drizzle ORM is dialect specific, slim, performant and serverless ready by design.

Step 1: Install Drizzle ORM

npm install drizzle-orm pg
npm install --save-dev drizzle-kit @types/pg

Step 2: Create a Drizzle Configuration File

In the config file we specify where our schema files are located and where the migrations should be stored. Create a new folder called "lib" and in there a new folder called "db". In the db folder, we will place all drizzle related stuff. So create a file called drizzle.config.ts in the db folder with the following content:

export default {
  schema: './lib/db/schema.ts',
  out: './lib/db/migrations',
  driver: 'pg', 
} 

Step 3: Create the npm scripts for drizzle generate and migrate

Add the following scripts to your package.json file:

  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk",
    "drizzle:generate": "npx drizzle-kit generate:pg  --config='./lib/db/drizzle.config.ts' ",
    "drizzle:migrate": "npx ts-node './lib/db/migrator-script.ts'"
  },

With the generate command we create the migrations files based on our schemas. With the migrate command we execute the migrations. This means we apply them to our postgres database, so altering real database tables. So be careful with this command.

Step 4: Creating our drizzle client

Create a new file called client.ts in the lib/db folder with the following content:

import { GetSecretValueCommand, SecretsManagerClient } from '@aws-sdk/client-secrets-manager';
import * as dotenv from "dotenv";
import { drizzle } from 'drizzle-orm/node-postgres';
import { Client } from 'pg';
import * as schema from './schema';

dotenv.config();

export const initDrizzleContext = async () => {

    //0 connection String? But How? need to define another function that is caring of fetching credentials and builing up the connection String
    const connectionString = await fetchDBSecret();
    //1 create client
    const client = new Client({connectionString})

    await client.connect();

    const db = drizzle(client, {
        schema
    })

    return db;
}

export async function fetchDBSecret(): Promise<string> {
if (process.env.DB_CONNECTION_URL) {
    return process.env.DB_CONNECTION_URL as string;
  }

  const client = new SecretsManagerClient({
    region: 'eu-west-1',
  });

  let response;
  try {
    response = await client.send(
      new GetSecretValueCommand({
        SecretId: process.env.POSTGRES_DB_SECRET ?? 'connectionUrl',
        VersionStage: 'AWSCURRENT', // VersionStage defaults to AWSCURRENT if unspecified
      })
    );
  } catch (error) {
    // For a list of exceptions thrown, see
    // https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
    console.log('ERROR fetching secret', error);
  }
  if (!response) throw new Error('Failed to fetch connection_url secret.');
  if (!response?.SecretString) throw new Error('SecretString not found in connection_url secret.');

  const secret = await JSON.parse(response.SecretString);
  const { password, host, port, username } = secret;
  const connectionString = `postgresql://${username}:${password}@${host}:${port}/postgres`;
  // can we set env with connection string so that we dont need to call it again and again?
  process.env.DB_CONNECTION_URL = connectionString;
  console.log('process env ', process.env)
  return connectionString;
}

With the initDrizzleContext function we create a function that our lambda handlers can call to get a drizzle client. The fetchDBSecret function is responsible for fetching the database credentials from Secrets Manager. We use the AWS SDK to fetch the secret and build up the connection string. With the connection string in place we can create a new client and pass it to the drizzle function. The drizzle function is responsible for creating the ORM client in there we pass the postgres client and the schema to the drizzle function. Our "db" variable is now a drizzle client that we return to the lambda handler.

Step 5: Use the Drizzle Client in the Lambda Handler

import { APIGatewayProxyEvent, APIGatewayProxyResult, Context } from 'aws-lambda';
import { initDrizzleContext } from '../lib/db/client';

export const handler = async (event: APIGatewayProxyEvent, context: Context): Promise<APIGatewayProxyResult> => {
  const db = await initDrizzleContext();
  const users = await db.query.users.findMany();

  return {
    statusCode: 200,
    body: JSON.stringify({ users: users }),
  };
};

In the lambda handler we call the initDrizzleContext function to get a drizzle client. With the client we can now query the users table.

Step 6: Create the migrator script

Create a new file called migrator-script.ts in the lib/db folder with the following content:

import { drizzle } from 'drizzle-orm/postgres-js';
import { migrate } from 'drizzle-orm/postgres-js/migrator';


import { fetchDBSecret } from './client';
import * as schema from './schema';
import postgres = require('postgres');
import path = require('path');

async function initConnection() {
  const connectionUrl = await fetchDBSecret();
  const connection = await postgres(connectionUrl);
  return connection;
}

async function runMigration() {
  const connection = await initConnection();
  const db = drizzle(connection, { schema });

  const res = await migrate(db, { migrationsFolder: path.join(__dirname, 'migrations') });
  await connection.end();
}

runMigration().catch(console.error);

The migrator script is responsible for running the migrations. Again we use here the fetchDBSecret function to get the connection string. Same as in the client we create a new postgres client and pass it to the drizzle function. With the drizzle client in place we can call the migrate function and pass the drizzle client and the path where our migrations are stored.

Finally we call the async migrate function and closing the connection.

Its important that the script has access to your AWS environment variables. So make sure that you have the right permissions to access the secrets manager and the database. In my case I got the ACCESS_KEY and SECRET_KEY from the AWS console and pasted them as environment variables in my terminal before running the script.

Step 6: Generate Migrations and Run Migrations

Okay from the code side we did everything. Now we need to generate the migrations and run them.

npm run drizzle:generate    // --> this should create a new migration file in the migrations folder
npm run drizzle:migrate    // --> this should run our migrator-script.ts file and apply the migration to the database

Open up your database and check if the tables are created. There should be a drizzle schema as well next to the postgres. In there drizzle stores all the information about the migrations and other stuff.

Step 7: Deploy the Lambda Function

Now that we have integrated Drizzle ORM into our lambda, we can deploy the Lambda function with the following command:

npx cdk deploy

At the end of our deployment we will have following architecture in place

Step 8: Test it!

Now that everything is in place, we can test our API by sending a GET request to the API Gateway endpoint. You should get an empty array as a response since we haven't added any data to the database yet. But we have successfully integrated Drizzle ORM into our AWS serverless environment! Cheers to you 🎉

Conclusion

Having drizzle in aws serverless context makes the database interactions so much easier. We can now use the power of an ORM in our serverless environment. We can now easily create, read, update and delete data from our database. We can also easily create migrations and run them. This makes the development process so much easier and faster. I also want to point out that I also tried to integrate prisma in the same context but it was a nightmare because of the dependencies. Drizzle is a great alternative to prisma in a serverless context.