Reusing resources across multiple pipelines with AWS CodePipeline

Summary

One of the limitations of CodePipeline currently is that there is not a good way to use common components across multiple pipelines. You may want to do this so you do not have to duplicate buildspecs or other files to be used in different repositories. In this post, I will show a strategy for reusing artifacts across multiple pipelines.

Repository Build Scripts

Modern “best of breed” CI servers allow you to commit build configuration to your repository. This is nice because your pipeline gets all the features of your SCM system like versioning and change tracking. But what happens when you would like to create “standard” pipelines for developers? It may not be possible for all repositories, but surely we don’t want to create each “docker image push” or “node js library” ad hoc. It would be much easier to publish CloudFormation templates or provide some other method to easily create pipelines for these types of artifacts.

Solution

The following diagram displays a solution to this problem. By adding an intermediary step that will take the source artifact and add files to it, we can reuse buildspec files. These files should be zipped up and stored in an S3 bucket.

Then, the CodeBuild project will be passed the output artifact of the lambda stage instead of the original source artifact. The build stage for modifying the artifact looks like this:

- Name: Build
  Actions:
    - Name: ModifySource
      ActionTypeId:
        Category: Invoke
        Owner: AWS
        Provider: Lambda
        Version: 1
      Configuration:
        FunctionName: ModifySourceArtifactLambda
        UserParameters: '<bucketname>/<myfile>.zip'
      InputArtifacts:
        - Name: Source
      OutputArtifacts:
        - Name: ModifiedSource
      RunOrder: 1

The ModifySourceArtifactLambda downloads a zip file and combines it with the InputArtifact and creates a new artifact. This artifact is then referenced by the CodeBuild project.

    - Name: DockerBuildPush
      ActionTypeId:
        Category: Build
        Owner: AWS
        Provider: CodeBuild
        Version: 1
      Configuration:
        ProjectName: !Ref 'BuildProject'
      InputArtifacts:
        - Name: ModifiedSource
      RunOrder: 2

The CodeBuild project should be configured to use the path to the buildspec as it is present in the zipped bundle.

Summary

The lambda source and instructions for installation are available on github.

The ModifySourceArtifactLambda was written with the ability to use it for more than just the buildspec use case. You can imagine it being useful sharing other resources across projects. Perhaps you could store a linting or test reporting configuration file to be used by the build.

If you find this helpful, give me a shout and let me know how you are using it!

Avatar
Kerry Wilson
Principal Cloud Architect

Coming from a development background, Kerry’s focus is on application development, infrastructure and security automation, and applying agile software development practices to IT operations in the cloud.

Related