Summary
Modern, best of breed, continuous integration tools allow developers to define their delivery pipeline as code and store it in the repository alongside the application source code. Doing so allows you to manage your pipeline much like you would your application source code. In this post, we will go over how to accomplish this using AWS CodeCommit, AWS Lambda, AWS CodePipeline, and AWS CloudFormation. This approach will also enable branch-based builds for CodePipeline, a large gap in the CodePipeline feature set currently.Solution Overview
For this solution, we will be utilizing AWS CodeCommit as our git repository, AWS Lambda to handle the commit events, and CloudFormation to provision the CodePipeline. The pipeline template is stored in the git repository and must contain the required parameters and provision a CodePipeline resource.
When a developer makes a commit to the CodeCommit repository, the lambda is triggered from the CodeCommit repository. This lambda fetches the pipeline template and creates a CloudFormation stack from it. Once the stack has finished deploying, the lambda then scans the stack resources for CodePipeline resources and starts the execution.
A new stack will be provisioned for each branch in the repository, so the template must be designed such that multiple stacks can be created from it. This means any resource attributes that need to be unique, should contain the branch name, which is passed to the template.
Pipeline Template
For convenience, the master template creates an S3 bucket to store your pipeline artifacts in and passes it as a parameter. Along with that parameter, the following parameters must be present in your pipeline template (you don’t have to use them):
ApplicationName - The name of the application for which this pipeline builds
ArtifactBucketArn - The ARN of the bucket used to store artifacts
ArtifactBucketName - The name of the bucket used to store artifacts
Branch - The branch this pipeline is running from
CodeCommitRepositoryArn - The ARN of CodeCommit repository
CodeCommitRepositoryName - The name of the CodeCommit repository
CommitId - The current commit id
PipelineFile - The path to the pipeline file for this repository
PipelineName - The name to use for the pipeline
RetentionPolicyInDays - How long to retain builds in days
These parameters can be used to do conditional logic in the pipeline. You can either use CloudFormation semantics, or pass them to CodeBuild or Lambdas as environment variables.
This solution also handles branch deletes as well. Deleting the stack and corresponding pipeline that is associated with the branch. Any application resources that may have been deployed are not managed by this solution. If the pipeline creates resources, a Lambda AWS::CloudFormation::CustomResource should be created in the pipeline to do any cleanup that may be required when it is deleted.
Getting Started
For this quick start, you will need make installed on your machine as well as the AWS CLI. In addition, the shell
environment will need to be authenticated to AWS.
To get started with this solution, you will first need to clone the repository:
git clone https://github.com/webdevwilson/aws-codepipeline-branches.git
Once the repository has been cloned, the next step is to deploy the commit handler lambda. You can do that with the following commands:
cd aws-codepipeline-branches
make bundle deploy
Once you have the code cloned, you can run ‘make bundle deploy’ to deploy the sample application. This will create an S3 bucket and upload a lambda layer that has the most recent version of boto3 library on it. In addition, it will launch a CloudFormation stack that will create the commit handler lambda and the CodeCommit repository.
Next, you will need to push the repository to that CodeCommit repository using the following commands:
git remote add demo <codecommit url from stack outputs>
git push demo master
This should trigger the lambda, creating the CodePipeline that will deploy the sample application. Now you can navigate to the CloudFormation and CodePipeline console pages to view the created stacks and pipelines.
To further play around with it, try creating a feature branch, then deleting it. This should create a new pipeline for that branch.
git push demo master:featurebranch
git push demo -d featurebranch
Sample Application
The sample application is located in examples/serverless of the codepipeline manager repository. This application
deploys a serverless application using AWS SAM. In this directory you will see the following files:
buildspec.yaml - Used by CodeBuild to package the CloudFormation template
deploy.yaml - AWS SAM template that will be used to deploy the application
lambda.py - Contains the application code that responds to the API Gateway requests
pipeline.yaml - This is the file that creates the CodePipeline and relevant lambda undeploy function
Wrapping Up
If you decide that you would like to utilize this workflow for your applications, you can use the template created at
template-out.yaml to create CodeCommit repositories for your applications. Develop a script to automate this.
It is possible to make this solution apply to all codecommit repositories in an account by using CloudWatch Events instead of CodeCommit triggers. For this you would create a single lambda that listens for CodeCommit events and pulls the file from the repository. Then, any CodeCommit repo would automatically get this functionality.
If you would like for certain branches to have different workflows, you can add conditional logic to your CloudFormation template using the branch name or other information. For example, you can add a condition based on the branch name and skip pipeline steps.
