Introduction

This week, I changed my blog’s theme from butterfly to cactus. I also managed to convert all AWS resources (except Route53) into a single CloudFormation template (gist). For this post, I want to write down thoughts and problems I encountered.

Circular Dependency

The Problem

The biggest problem I have encounter is the circular dependency problem between KMS and other services.

Because CloudFront distribution cannot access AWS Managed S3 key (you cannot modify key policy of AWS Managed key), you need your own KMS key, named ResourceKey here, to encrypt the S3 Bucket.

CodeBuild need a IAM service role in order to put the output artifact to the S3 bucket. And for the sake of least privilege, ResourceKey‘s key policy will reference service role in order to restrict access to this CodeBuild.

ResourceEncryptionKey
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
ResourceEncryptionKey:
    Type: 'AWS::KMS::Key'
    Properties:
      Description: AWS KMS key for encryting website resources
      Enabled: true
      EnableKeyRotation: true
      PendingWindowInDays: 7
      KeyPolicy:
        Version: 2012-10-17
        Id: key-consolepolicy-3
        Statement:
          - Sid: Enable Root Permissions
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'
            # CodeBuild uses kms to encrypt artifact
          - Sid: AllowCodeBuildPrinciple
            Effect: Allow
            Principal:
              AWS: !GetAtt CodeBuildRole.Arn
            Action:
              - 'kms:Encrypt'
              - 'kms:Decrypt'
              - 'kms:ReEncrypt*'
              - 'kms:GenerateDataKey*'
            Resource: '*'
            # CloudFront uses kms to decrypt S3 content
          - Sid: AllowCloudFrontPrinciple
            Effect: Allow
            Principal:
              Service: cloudfront.amazonaws.com
            Action:
              - 'kms:Decrypt'
              - 'kms:Encrypt'
              - 'kms:GenerateDataKey*'
            Resource: '*'
            # Condition:
            #   StringEquals:
            #     'AWS:SourceArn': !Sub 
            #       - 'arn:aws:cloudfront::${AccountId}:distribution/${DistributionId}'
            #       - AccountId: !Ref AWS::AccountId
            #         DistributionId: !GetAtt BlogDistribution.Id
CodeBuildRole
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
CodeBuildRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Sub "${ProjectName}-codebuild-service-role"
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: codebuild.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Resource: '*'
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
              - Effect: Allow
                Resource: 
                  - !GetAtt BlogArtifactBucket.Arn
                  - !Join ["/", [!GetAtt BlogArtifactBucket.Arn, "*"]]
                Action:
                  - 's3:PutObject'

The AllowCodeBuildPrinciple policy in ResourceKey has the principle refering CodeBuildRole. However, because CodeBuildRole needs to define a S3::PutObject policy. The S3 Bucket, BlogArtifactBucket, must be created in order for the CodeBuildRole’s policy to reference the bucket Arn. But we need ResourceKey in order to create the bucket. Do you see the problem here?

  • ResourceKey requires CodeBuildRole
  • CodeBuildRole requires BlogArtifactBucket
  • BlogArtifactBucket requries ResourceKey

There is a circular dependency between three resources.

The Discovery

I did not realize the solution until I started to work on the BlogArtifactBucket‘s bucket policy. You see, in CloudFormation, the AWS::S3::Bucket resource does not a “Policies” or “PolicyDocument” property to let you embed the policy directly in the Bucket definition. Instead, you need a seperate resource called AWS::S3::BucketPolicy that references to the bucket.

BlogArtifactBucket
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
BlogArtifactBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
        BucketName: !Sub
        - "${ProjectName}-${S3BucketName}-${StackId}"
        - ProjectName: !Ref ProjectName
            S3BucketName: !Ref S3BucketName
            StackId: !Select [2, !Split ["/", !Ref AWS::StackId]]
        BucketEncryption:
        ServerSideEncryptionConfiguration:
            - ServerSideEncryptionByDefault:
                SSEAlgorithm: 'aws:kms'
                KMSMasterKeyID: !GetAtt ResourceEncryptionKey.Arn
            BucketKeyEnabled: true
        VersioningConfiguration:
        Status: Suspended
        WebsiteConfiguration:
            IndexDocument: index.html
            ErrorDocument: error.html

BlogArtifactBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
        Bucket: !Ref BlogArtifactBucket
        PolicyDocument:
        Version: 2012-10-17
        Id: PolicyForCloudFrontPrivateContent
        Statement:
        - Sid: AllowCloudFrontServicePrincipal
            Effect: Allow
            Principal:
            Service: cloudfront.amazonaws.com
            Action: s3:GetObject
            Resource: !Join ["/", [!GetAtt BlogArtifactBucket.Arn, "*"]]
            Condition:
            StringEquals:
                AWS:SourceArn: 
                - !Sub
                    - arn:aws:cloudfront::${AccountId}:distribution/${DistributionId}
                    - AccountId: !Ref AWS::AccountId
                    DistributionId: !GetAtt BlogDistribution.Id

The outcome this, in my observation of CloudFormation stack creation, is that BlogArtifactBucket is created at first; then the CloudFront distribution; at the end BlogArtifactBucketPolicy is attched to BlogArtifactBucket.

The Solution

I want to use the same method to solve the circular problem above. After a quick digging, I found out that CloudFormation has resources AWS::IAM::Role and AWS::IAM::RolePolicy. The AWS::IAM::RolePolicy has a property called “RoleName” that can attach the policy to specific role. So wala, the solution is solved. Now, the stack creation order becomes

  • ResourceKey requires CodeBuildRole
  • BlogArtifactBucket requries ResourceKey
  • CodeBuildS3PutObjectGrantPolicy requries CodeBuildRole

and there is no circulation now.

new ColdeBuildRole
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Role policy granting PutObject to CodeBuild
CodeBuildS3PutObjectGrantPolicy:
    Type: AWS::IAM::RolePolicy
    DependsOn: CodeBuildRole
    Properties:
        PolicyName: CodeBuildS3PutObjectGrantPolicy
        RoleName: !Sub "${ProjectName}-codebuild-service-role"
        PolicyDocument:
        Version: 2012-10-17
        Statement:
            - Effect: Allow
            Resource: 
                - !GetAtt BlogArtifactBucket.Arn
                - !Join ["/", [!GetAtt BlogArtifactBucket.Arn, "*"]]
            Action:
                - 's3:PutObject'

  # CodeBuild service role
CodeBuildRole:
    Type: 'AWS::IAM::Role'
    Properties:
        RoleName: !Sub 
        - "${ProjectName}-codebuild-service-role-${StackId}"
        - ProjectName: !Ref ProjectName
            StackId: !Select [2, !Split ["/", !Ref AWS::StackId]]
        AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
            - Effect: Allow
            Principal:
                Service: codebuild.amazonaws.com
            Action:
                - 'sts:AssumeRole'
        Policies:
        - PolicyName: root
            PolicyDocument:
            Version: 2012-10-17
            Statement:
                - Effect: Allow
                Resource: '*'
                Action:
                    - 'logs:CreateLogGroup'
                    - 'logs:CreateLogStream'
                    - 'logs:PutLogEvents'

A trick

This is something when I am almost done with the CloudFormation template. I realize that you can use aws cli to describe a existing resources. Since I did not start from a blank, I can use for instance aws cloudfront get-distribution --id <DistributionId> to get the current setting of my CloudFront distribution. That’s a neat trick to save time from deciding which configuration value to use.

Another Circular Dependency

The Problem

This section was written the other day. In my original architecture, my S3 Bucket, BlogArtifactBucket, will invoke a Lambda function, CloudFrontInvalidationLambda, every time CodeBuild put the artifact into the bucket. As name suggests, the function will create a request to invalidate current BlogDistribution in order for the index page to be updated immediately.

Moving forward to CloudFormation, this creates yet another circular dependency problem:

  • BlogArtifactBucket requires CloudFrontInvalidationLambda (reference for EventConfiguration )
  • CloudFrontInvalidationLambda requires BlogDistribution (reference for using DistributionId as request argument)
  • BlogDistribution requires BlogArtifactBucket (reference for origin)

I cannot find a proper way to implement this nicely because unlick Role and RolePolicy discussed above, AWS::S3::Bucket’s NotificationConfiguration property cannot be configured in a seperate resource. Nor that AWS::Lambda::Function have a seperate resource to achieve our goal.

The Discovery

After some heavy digging, I found out that AWS::CloudFormation::CustomResource can be used to invoke Lambda functions during stack creation. Bsed on the documentation, I can invoke a Lambda function to add a lambda NotificationConfiguration to S3 bucket for any new object created.

The solution

Simply define another lambda resource that was granted a RolePolicy to s3:PutBucketNotification. Then define a AWS::CloudFormation::CustomResource with ServiceToken refering to the new lambda function and supply the invalidation lambda function’s arn and the target s3 bucket as argument. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
CloudFrontInvalidationOnS3PutObjectEventConfigurerOnBuildLambdaExecutionRole:
  Type: AWS::IAM::Role
  Properties:
    RoleName: !Sub 
      - "${ProjectName}-${Region}-lambda-configurer-role-${StackId}"
      - ProjectName: !Ref ProjectName
        Region: !Ref AWS::Region
        StackId: !Select [0, !Split ["-", !Select [2, !Split ["/", !Ref AWS::StackId]]]]
    Path: !Sub "/${ProjectName}/lambda/"
    AssumeRolePolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action:
            - 'sts:AssumeRole'
    ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole  # AWSLambdaBasicExecutionRole

# CloudFrontInvalidationLambda need "cloudfront:CreateInvalidation"
S3PutBucketNotificationGrantPolicy:
  Type: AWS::IAM::RolePolicy
  Properties:
    PolicyName: S3PutBucketNotificationGrantPolicy
    RoleName: !Ref CloudFrontInvalidationOnS3PutObjectEventConfigurerOnBuildLambdaExecutionRole
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Resource: !GetAtt BlogArtifactBucket.Arn
          Action:
            - s3:PutBucketNotification

# Lmabda function to put a PutBucketNotificationConfigurationCommand through aws javascript sdk
CloudFrontInvalidationOnS3PutObjectEventConfigurerOnBuildLambda:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: !Sub 
      - ${ProjectName}-${Region}-lambda-configurer-${StackId}
      - ProjectName: !Ref ProjectName
        Region: !Ref AWS::Region
        StackId: !Select [0, !Split ["-", !Select [2, !Split ["/", !Ref AWS::StackId]]]]
    Description: Lambda function to add CloudFrontInvalidationLambda to BlogArtifactBucket's LambdaConfiguration
    Architectures:
      - x86_64
    Runtime: nodejs20.x
    Handler: index.handler
    Timeout: 5
    Role: !GetAtt CloudFrontInvalidationOnS3PutObjectEventConfigurerOnBuildLambdaExecutionRole.Arn
    # LoggingConfig:
    #   LogGroup: /aws/lambda/debug-custom-resource
    Code:
      ZipFile: >
        const { S3Client, PutBucketNotificationConfigurationCommand } = require("@aws-sdk/client-s3");
        var response = require('cfn-response');

        exports.handler = async function(event, context) {
          const client = new S3Client();

          let { SourceBucket, TargetLambdaFunctionArn } = event.ResourceProperties;
          const input = {
            "Bucket": SourceBucket,
            "NotificationConfiguration": {
              "LambdaFunctionConfigurations": [
                {
                  "Events": [
                    "s3:ObjectCreated:Put"
                  ],
                  "LambdaFunctionArn": TargetLambdaFunctionArn
                }
              ]
            }
          };
          const command = new PutBucketNotificationConfigurationCommand(input);
          await client.send(command);

          var responseData = {Status: response.SUCCESS};
          response.send(event, context, response.SUCCESS, responseData);
        };

CloudFrontInvalidationOnS3PutObjectEventConfigurerOnBuild:
  Type: AWS::CloudFormation::CustomResource
  Properties:
    ServiceToken: !GetAtt CloudFrontInvalidationOnS3PutObjectEventConfigurerOnBuildLambda.Arn
    SourceBucket: !Ref BlogArtifactBucket
    TargetLambdaFunctionArn: !GetAtt CloudFrontInvalidationLambda.Arn

Conclusion

Comparing to Terraform, I definitely think that CloudFormation’s syntax and language features is little outdated. However, its integration with aws concole/cli is very helpful development friendly. So it is hard to judge what’s best. My next goal is probably to experiment with the CloudFormation CDK and see if developing in Python/Javascript is better than YAML.