Terraform, API Gateway and Cognito

I’d like to control API Gateway as an HTTP Proxy to an ALB for an ECS Task.

Unfortunately, Terraform’s support of Cognito isn’t quite there.

There are some features missing:

In this context, I need to add a Cognito Authorizer for an existing User Client Pool.

Currently, Terraform only supports making an authorizer for a lambda only. So creating an authorizer for cognito is a manual step. Creating a cognito authorizer is documented but creating it with the AWS console is easy. Just make it of type COGNITO then select the pool you want.

Next you need to attach the authorizer to the aws_api_gateway_method resources desired. Your methods would look similar to this:

resource "aws_api_gateway_method" "api-gateway-method-post" {
  rest_api_id   = "${aws_api_gateway_rest_api.api-gateway.id}"
  resource_id   = "${aws_api_gateway_resource.api-gateway-resource.id}"
  http_method   = "POST"
  authorization = "COGNITO_USER_POOLS"
  authorizer_id = "${var.cognito-authorizer-id}"
}

variable "cognito-authorizer-id" {
  default = "9rvrci"
}

Setting authorization to COGNITO_USER_POOLS isn’t documented but it currently works.

The hard part here is finding the authorizer id. I found this by setting using it manually in the AWS console then running terraform plan to see what terrafrom would change the value of a current method from to empty. I’m sure there are better ways.

Summary

Process for API Gateway with Cognito Authorizer

  • Create API Gateway (minus authorizer) with Terraform
  • Create Cognito User Pool (maybe without Terraform)
  • Create Cognito Authorizer on the API Gateway (without Terraform)
  • Add Cognito Authorizer details to the Terraform configuration then apply

One day soon, Terraform will support all this 🙂

SharpCompress 0.19 – .NET Standard 2.0

New Features:

Bug Fixes:
Opening 7zip archive with invalid win32 date
eliminate spurious rar crc exception
create new memorystream to allow proper resizing as memorystream could be a user provided buffer

NuGet: https://www.nuget.org/packages/sharpcompress/0.19.0

Docker Container with .NET Core 1 and 2 for Cake builds

Cake build container

I love Cake but it doesn’t run on .NET Core 2 yet. Actually, I’m trying to fix that issue: https://github.com/cake-build/cake/pull/1812

Unfortunately, the official Docker images don’t have both runtimes installed. So it should be easy to make an image with both right? Well, it wasn’t as easy as I thought it would but I managed to do it! https://github.com/adamhathcock/cake-build

Uses a Ubuntu 16.04 base image with a .NET Core 1 runtime installed then .NET Core 2 SDK installed. This should be enough to build any .NET Core 2 project using Cake.

CircleCI is my build service of choice and it is Docker all the way!

Sample CircleCI 2.0 Config

version: 2
jobs:
  build:
    docker:
      - image: adamhathcock/cake-build:latest
    steps:
      - checkout
      - run:
          name: Build
          command: ./build.sh

Actually, this config isn’t that interesting 🙂

Using the AWS SDK to login with MFA and Assume Role

A new-ish thing to me is having my IAM account on a centralized AWS account then switching roles to a role in another AWS account. It’s a good way to manage users across a lot of accounts: Cross-Account Access in the AWS Management Console

What is definitely is having to do this programmatically for a script. I’m using C# for this but the guts are the same for any language I’m sure.

Steps:

1) Load main credentials – either hard-coded or from ~/.aws/credentials
2) Get a Session Token from STS
3) Setup MFA callback
4) Use Session Token creds and MFA call back on an AssumeRole credential set then to do work.

Code

//needed info from target account
var targetRoleAccount = "<account id>";
var targetRoleName = "<role name>";
//needed info from main account about my user
var mainAccount = "<account id>";
var mainAccountUser = "<my user name>";
//my user creds
var mainAccountUserAccessToken = "<aws access token>";
var mainAccountUserSecretToken = "<aws secret token>";
//make some ARNs
var roleArn = $"arn:aws:iam::{targetRoleAccount}:role/{targetRoleName}";
var mfaArn = $"arn:aws:iam::{mainAccount}:mfa/{mainAccountUser}";

var basicCreds = new BasicAWSCredentials(mainAccountUserAccessToken, mainAccountUserSecretToken);

var stsClient = new AmazonSecurityTokenServiceClient(basicCreds);
var sessionResponse = await stsClient.GetSessionTokenAsync();

var sessionCreds = new SessionAWSCredentials(sessionResponse.Credentials.AccessKeyId,
    sessionResponse.Credentials.SecretAccessKey, sessionResponse.Credentials.SessionToken);

var options = new AssumeRoleAWSCredentialsOptions()
{
    MfaSerialNumber = mfaArn,
    MfaTokenCodeCallback = () =>
    {
        Console.WriteLine("Enter MFA");
        return Console.ReadLine();
    }
};

var assumeRoleCredentials = new AssumeRoleAWSCredentials(sessionCreds, roleArn, targetRoleName, options);

//time to work!
var client = new AmazonEC2Client(assumeRoleCredentials, RegionEndpoint.EUWest1);

The code roughly follows the steps I listed before. The trick is getting your MFA code in.

Here, I just have a console app so I can just ReadLine() and enter the numbers from my phone which I use for the two-factor code.

Took some figuring out as I didn’t know the AWS termology and had to dig the into the AWS SDK integration tests for STS to get it right. Wasn’t too bad.

SharpCompress 0.18

NuGet
GitHub Release

New
* Breaking change – Remove ArchiveEncoding static class in favor of instance on OptionsArchiveEncoding is now on the Options base class. This now allows for more Encoding class options as well as a custom Func for decoding for more custom options. Being instance based avoids multi-threading issues. See https://github.com/adamhathcock/sharpcompress/blob/master/src/SharpCompress/Common/ArchiveEncoding.cs

Fixes
* LeaveStreamOpen doesn’t work with TarWriter
* If Zip file has normal file header AND a post-descriptor header AND the file is attempted to be skipped by a ZipReader, then the data is attempted to be skipped twice.
* AbstractReader.Skip() does not fully read bytes from non-local streams

.NET Core on Circle CI 2.0 using Docker and Cake

I’ve only just started with Circle 2.0, which just had it’s beta tag removed.

It’s completely Docker based which I adore. I refuse to package code any other way these days.

My goal would was to build on what I previously did on Circle CI but only use an official Microsoft .NET Core SDK docker image. Having to layer extra tools onto another image and manage that is extra work. I abhor extra work.

.circleci/config.yml

Circle 2.0 moves their YAML to a subdirectory which seems to be envogue these days so we can have lots of files for specific services!

version: 2
jobs:
  build:
    working_directory: ~/api
    docker:
      - image: microsoft/dotnet:1.1.2-sdk-jessie
    environment:
      - DOTNET_CLI_TELEMETRY_OPTOUT: 1
      - CAKE_VERSION: 0.19.1
    steps:
      - checkout
      - restore_cache:
          keys:
            - cake-{{ .Environment.CAKE_VERSION }}
      - run: ./build.sh build.cake --target=restore
      - save_cache:
          key: cake-{{ .Environment.CAKE_VERSION }}
          paths:
            - ~/api/tools
      - run: ./build.sh build.cake --target=build
      - run: ./build.sh build.cake --target=test

The hard part with Circle CI 2.0 is that caching is done pretty manually and changes aren’t auto-detected. You have to version cache keys or hashes that act as cache keys. I haven’t mastered it yet.

Ideally, I’d cache the Cake tools directory and my .nuget folder on this running image but I’m not there yet.

The big thing to note is that the image is based on the official SDK image with all the necessary build tools.

Bootstrapping Cake

So it should be easy to do this now as I already have a build.sh to execute Cake right? Nope!

The bash script uses the unzip utility that usually exists. This is needed to extract the nuget package that is downloaded. curl doesn’t exist either, by the way.

Fortunately, the dotnet cli is here. It should easily restore Cake. My new build.sh needs a csproj to restore Cake with. Since the new csproj XML is tiny, this is easy to echo into a file.

#!/usr/bin/env bash
##########################################################################
# This is the Cake bootstrapper script for Linux and OS X.
# This file was downloaded from https://github.com/cake-build/resources
# Feel free to change this file to fit your needs.
##########################################################################

# Define directories.
SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
TOOLS_DIR=$SCRIPT_DIR/tools
TOOLS_PROJ=$TOOLS_DIR/tools.csproj
CAKE_DLL=$TOOLS_DIR/Cake.CoreCLR.$CAKE_VERSION/cake.coreclr/$CAKE_VERSION/Cake.dll


# Make sure the tools folder exist.
if [ ! -d "$TOOLS_DIR" ]; then
  mkdir "$TOOLS_DIR"
fi

###########################################################################
# INSTALL CAKE
###########################################################################

if [ ! -f "$CAKE_DLL" ]; then
    echo "<Project Sdk=\"Microsoft.NET.Sdk\"><PropertyGroup><OutputType>Exe</OutputType><TargetFramework>netcoreapp1.1</TargetFramework></PropertyGroup></Project>" > $TOOLS_PROJ
    dotnet add $TOOLS_PROJ package cake.coreclr -v $CAKE_VERSION --package-directory $TOOLS_DIR/Cake.CoreCLR.$CAKE_VERSION
fi

# Make sure that Cake has been installed.
if [ ! -f "$CAKE_DLL" ]; then
    echo "Could not find Cake.exe at '$CAKE_DLL'."
    exit 1
fi

###########################################################################
# RUN BUILD SCRIPT
###########################################################################

# Start Cake
exec dotnet "$CAKE_DLL" "$@"

Note: I’ve moved the CAKE_VERSION variable out of the script to attempt to use it with CircleCI but it can easily be added back