Introduction

CloudFormation Datasource is a solution to make your CloudFormation templates more dynamic by allowing you to run API calls at deployment time. It emulates Terraform data sources behavior and allows you to retrieve AWS Organizations Id, the subnet with most free space or a specific KMS Key ARN from its alias.

It allows you to simplify some CloudFormation templates by removing parameters and mappings and rely instead on AWS API to retrieve the correct value at deployment time.

Solution

The solution is based on a generic custom resource backed with an AWS Lambda Function.

The custom resources takes the following inputs:

• AWS Service
• Action
• Parameters (as defined in the API Reference guide of the AWS Service)
• AWS Region
• Output selector (JMESPath expression)

The Lambda function parses the input and performs the API call on behalf of the CloudFormation template. It filters the result based on the output selector and return the value to the CloudFormation template. The value can then be used using the GetAtt intrinsic function.

Optionally, the function can lookup through CloudTrail events, the identity who perform the API call and check using AWS IAM policy simulator if it is allowed to perform the API call.

By default, the solution is deployed with a limited permissions set using the Amazon Managed policy ViewOnlyAccess which grants permissions to view resources and basic metadata across all AWS services. You can change it to fit your needs.

The behavior of the AWS Lambda function really mimicks the AWS CLI behavior when you pass the –query argument.

Deployment instructions

This template has been built with AWS SAM CLI

In order to deploy the template, run the following commands:

1
2
3

git clone https://github.com/j2clerck/cloudformation-datasource.git
sam build
sam deploy --guided

How to use it

Create a custom resource in your Cloudformation template and the service token should be the Arn of the lambda function. The Arn is exported as CFNDATASOURCE, so you can use it simply by entering ServiceToken: !ImportValue CFNDATASOURCE

It requires the name of the Service (‘organizations’, ’ec2’…) and the Action to invoke (‘DescribeOrganization’…), the parameters as a dict object (per the API reference of the service) and the query to filter the result using a JMESPath expression.

The example below shows how to return the OrganizationId

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

Resources:
  rGetOrgId:
    Type: Custom::GetOrgId
    Properties:
      ServiceToken: !ImportValue CFNDATASOURCE
      Action: DescribeOrganization
      Service: organizations
      Region: !Ref AWS::Region
      Parameters: AWS::NoValue
      Query: "Organization.{Id:Id}"

Outputs:
  Output0:
    Description: OrganizationId
    Value: !GetAtt rGetOrgId.Id

Conclusion

I’ve found this solution useful for recurring use-cases when I needed to retrieve the Organization Id to secure an Amazon S3 Bucket, or to select the subnet with the maximum address space available. I’ve documented some examples in the git repository and I hope it can be useful to you too. If you have some good examples you want to share, please submit a pull request.