The last thing that I missed in the CloudFormation

By Wojciech Gawroński | October 4, 2018

The last thing that I missed in the CloudFormation

We don’t need expressive language, except when we need it

By the end of August AWS released a developer preview of the Cloud Development Kit (aws-cdk in short), which allows you to codify infrastructure code in your language of choice. A real programming language.

You may ask: why is it a big deal?

Imagine that you have to prepare 30 IAM user accounts and S3 buckets with the following permissions:

  1. Each user can have access to the limited set of services from AWS platform, including S3.
  2. Each user can have read access to the one pre-existing bucket and read-write access to their bucket. Moreover, we revoke any access to the other buckets.

Sounds artificial? It is a real use case - I needed to do that for the workshop about Amazon SageMaker that we created for Big Data Moscow 2018 conference.

Convinced? Let’s work this out together.

First point. A piece of cake! We can make it with use of IAM Groups. The second one… huh, that’s not so easy.

If we carefully think about it, we need to associate access to the previously created bucket individually per user, so that requires a custom IAM policy per user. We can work around that with careful tagging and generic IAM policy, but that is way more complicated than we planned initially.

Now you can see the pitfall. Let me clarify - leveraging declarative approach of CloudFormation is a good thing.

However, it lacks constructs that are helping to be in line with DRY principle (Don’t Repeat Yourself). Both YAML and JSON flavors do not support any templating mechanisms engine, so it is difficult to reuse fragments of code. Even for YAML AWS CloudFormation does not support the anchors and aliases, neither hash merges.

How you deal with repetition then? Either by being “pragmatic” - writing helper scripts/copying&pasting - or harnessing other tools like Troposphere or Sceptre.

However, from now on - we have a better option available.

How to keep up with AWS updates?

Click and enter your email to get access to the distilled and carefully selected version of the Amazon Web Services updates in the form of blog posts.

Subscribe me

AWS-CDK

AWS CDK Application Pipeline

Example of the AWS CDK Application pipeline from AWS blog post.

So the CDK acts like a “compiler” for your infrastructure, that we create with the provided SDK.

It contains a set of high-level class libraries, (they call them Constructs), that abstracts AWS cloud resources and help to leverage AWS best practices. Then we combine those constructs into object-oriented application that precisely define your application infrastructure. Underneath it covers all the complex boilerplate logic.

As a next step you build your CDK application, and it compiles into a CloudFormation template. In that sense CloudFormation is the assembly language for AWS cloud infrastructure. Having generated the template, we use it in the AWS CloudFormation service and provision resources to our AWS account.

In other words: the CDK allows you to define your infrastructure stack with code, while the CloudFormation service takes care provisioning of your stacks - as you usually do.

For now, it supports Java, TypeScript, and JavaScript. You can find official documentation here.

Additionally, tools provided in that CDK allows you to manage CloudFormation stacks (create, update and destroy), but also provides exciting feature that allows you to generate a diff (like with git or any SCM tool of choice) between deployed and the developing states.

AWS CDK provides a diff command for stacks that you are managing with it

You cannot imagine how much I miss such thing when working with classical CloudFormation.

Any other interesting improvements. Of course, there is a couple more.

The ones that I like are grantRead and grantReadWrite methods for particular constructs. As you can imagine, they are generating IAM permissions that we can pass to a policy without looking for specific prefixes and names. For example:

let userBucket = new Bucket(this, `ParticipantBucket${index}`, {
  bucketName: `amazon-sagemaker-in-practice-bucket-user-${index}`
});

userBucket.grantReadWrite(participant);

Sounds fantastic! Are there any problems?

Unfortunately, there is a couple of issues. First and foremost: the resulted template looks like a mess. But, what else you can expect from the assembly language:

Resources:
    AmazonSageMakerInPracticeRole1C6797A8:
        Type: 'AWS::IAM::Role'
        Properties:
            AssumeRolePolicyDocument:
                Statement:
                    -
                        Action: 'sts:AssumeRole'
                        Effect: Allow
                        Principal:
                            Service: sagemaker.amazonaws.com
                Version: '2012-10-17'
            ManagedPolicyArns:
                - 'arn:aws:iam::aws:policy/AmazonSageMakerFullAccess'
            RoleName: amazon-sagemaker-in-practice-workshop-role
    AmazonSageMakerInPracticeParticipants2CBF639E:
        Type: 'AWS::IAM::Group'
    AmazonSageMakerInPracticeParticipantsDefaultPolicyBBAB335C:
        Type: 'AWS::IAM::Policy'
        Properties:
            PolicyDocument:
                Statement:
                    -
                        Action:
                            - 's3:GetObject*'
                            - 's3:GetBucket*'
                            - 's3:List*'
                        Effect: Allow
                        Resource:
                            - 'arn:aws:s3:::amazon-sagemaker-in-practice-workshop'
                            -
                                'Fn::Join':
                                    - ""
                                    -
                                        - 'arn:aws:s3:::amazon-sagemaker-in-practice-workshop'
                                        - /
                                        - '*'
                Version: '2012-10-17'
            PolicyName: AmazonSageMakerInPracticeParticipantsDefaultPolicyBBAB335C
            Groups:
                -
                    Ref: AmazonSageMakerInPracticeParticipants2CBF639E
    AmazonSageMakerInPracticeParticipantsPolicy876F774C:
        Type: 'AWS::IAM::Policy'
        Properties:
            PolicyDocument:
                Statement:
                    -
                        Action:
                            - 'sagemaker:*'
                            - 'ecr:*'
                            - 'cloudwatch:*'
                            - 'logs:*'
                            - 's3:GetBucketLocation'
                            - 's3:ListAllMyBuckets'
                            - 'iam:ListRoles'
                            - 'iam:GetRole'
                        Effect: Allow
                        Resource: '*'
                    -
                        Action: 'iam:PassRole'
                        Condition:
                            StringEquals:
                                'iam:PassedToService': sagemaker.amazonaws.com
                        Effect: Allow
                        Resource: '*'
                Version: '2012-10-17'
            PolicyName: AmazonSageMakerInPracticeParticipantsPolicy876F774C
            Groups:
                -
                    Ref: AmazonSageMakerInPracticeParticipants2CBF639E
    User1E278A736:
        Type: 'AWS::IAM::User'
        Properties:
            Groups:
                -
                    Ref: AmazonSageMakerInPracticeParticipants2CBF639E
            LoginProfile:
                Password: A_DUMMY_PASSWORD
            UserName: amazon-sagemaker-in-practice-user-1
    User1DefaultPolicy8C24FEE1:
        Type: 'AWS::IAM::Policy'
        Properties:
            PolicyDocument:
                Statement:
                    -
                        Action:
                            - 's3:GetObject*'
                            - 's3:GetBucket*'
                            - 's3:List*'
                            - 's3:DeleteObject*'
                            - 's3:PutObject*'
                            - 's3:Abort*'
                        Effect: Allow
                        Resource:
                            -
                                'Fn::GetAtt':
                                    - ParticipantBucket1349F69F9
                                    - Arn
                            -
                                'Fn::Join':
                                    - ""
                                    -
                                        -
                                            'Fn::GetAtt':
                                                - ParticipantBucket1349F69F9
                                                - Arn
                                        - /
                                        - '*'
                Version: '2012-10-17'
            PolicyName: User1DefaultPolicy8C24FEE1
            Users:
                -
                    Ref: User1E278A736
    ParticipantBucket1349F69F9:
        Type: 'AWS::S3::Bucket'
        Properties:
            BucketName: amazon-sagemaker-in-practice-bucket-user-1

    # ...

    CDKMetadata:
        Type: 'AWS::CDK::Metadata'
        Properties:
            Modules: '@aws-cdk/...'

Thankfully, you do not need to look to that very often. However, in any case, when you need to debug something, you are dealing with such content.

There is a couple more issues that I collected as a list:

  • There is no easy way to generate DependsOn which can lead to problematic situations with automatic order resolution.
  • No support for StackPolicies so far.
  • You can see that each logical ID contains a not exactly user-friendly hash at the end - that is a deliberate choice explained here that provides many features - diff is just one of those, so it is a tradeoff.
  • Hard or even impossible to access stack properties (e.g., Description).
  • Not precisely fresh and up to date documentation, but that is a developer preview.
  • No support for newest services (e.g., no construct for Amazon SageMaker in AWS CDK even that there is support in AWS CloudFormation for that service).
  • No support for passing parameters when deploying, you can generate a CloudFormation template with parameters, but you cannot deploy it with use of CDK - instead, they are recommending to use contexts. Again, it is the controversial decision that is a tradeoff. They did that because of compile-time guarantees. You can read more about that here.

If you need more examples or general look and feel, I encourage you to look into our implementation from the workshop mentioned above - it is available on our Github. Stars are more than welcome!

Side note: I have chosen TypeScript. 😉

What’s next?

That is not the end of the enhancements added to the CloudFormation - one additional example can be AWS Lambda based macros inside the templates announced at the beginning of September. With such expressiveness and power, we can do much more - if you are considering to build your product on AWS we strongly encourage you to learn both techniques. That investment is worth having because it is going to make your infrastructure as a code more reliable and extensible.

Build Your Competetive Advantage with Continous Delivery

Infrastructure as a Code is a key element of successful continuous delivery. Partner with our experienced and pragmatic builders that help you innovate, move quickly, maintain your infrastructure reliably with having fewer costs related to the operational side and scalability.

Schedule a call with our expert
comments powered by Disqus