(Read this article on the blog)

Handling secrets in the cloud

The recommended way to handle secrets in AWS is to put them either into Ssm Parameter Store or into Secrets Manager. Then whatever needs to use these values, give them IAM permissions to read the value.

For example, a Lambda function might need to connect to a MongoDB database. Since it is outside AWS, IAM permissions won’t work, instead, a password is needed. How should the Lambda get this password? Put it into Parameter Store, then use IAM permissions to give the Lambda’s execution role access to the parameter.

This is a security best practice as the value is stored in a place that is protected by IAM permissions. Anybody (or anything) who needs access can be granted via an explicit permission and since every part of the system only references the value it won’t show up accidentally in state files or environment variables.

Generated secrets

The above approach is good for secrets that are generated outside the deployed stack, such as a token to access a third-party system, a database password, or an encryption key that needs to be usable by other systems.

But what about secrets that are used only in this stack?

For example, IVS private channels require a public-private key pair to work. The public part is added to IVS while the private needs to be accessible to the backend to be able to sign tokens for authorized viewers.

Another example is CloudFront Signed URLs. The public key is added to the distribution while the private key is used by the backend for the signature.

Notice that these secrets are different than the MongoDB password we discussed earlier. The exact value of the IVS key pair does not matter, it is only needed to be usable inside the stack. It does not matter if it changes as long as both the IVS and the backend part is changed as well. There is no need for these keys to be stable or usable outside the stack.

Because of this, manually generating these keys and adding them to Ssm Parameter Store before deploying the stack and delete them after seems like an overkill. After all, how an IVS private channel where the backend can control who can view the video different than let’s say a DynamoDB table that the backend can read data from? The only difference is that for IVS a secret must be generated while DynamoDB access is controlled by IAM. An IVS key pair should be part of the stack, generated when all the other resources are deployed and destroyed when the stack is cleaned up.

Solution

The solution is to generate these values dynamically but fully in the cloud. For this, we need a Lambda function that can add a parameter to the SSM Parameter Store and will be called during the deployment process.

Deploying and calling a Lambda during deployment is supported by both CloudFormation and Terraform. In CloudFormation it is based on custom resources which is wrapped with a nicer resource type in the CDK called AwsCustomResource. In Terraform, there is an aws_lambda_invocation resource type. In this article, we’ll focus on a solution based on Terraform.

Creating the function is no different than creating any other functions: it needs the code in an archive file, an execution role with the required permissions attached, and then the function itself:

resource "aws_lambda_function" "generate_value" {
	function_name    = "ssm-generated-value-module-${random_id.id.hex}"
	filename         = data.archive_file.generate_value.output_path
	source_code_hash = data.archive_file.generate_value.output_base64sha256
	timeout = 30
	handler = "index.handler"
	runtime = "nodejs18.x"
	environment {
		variables = {
			SSM_PARAMETER: var.parameter_name,
		}
	}
	role    = aws_iam_role.generate_value_exec.arn
}

To make Terraform call this function during the different stages of the deployment, the aws_lambda_invocation resource is used:

resource "aws_lambda_invocation" "generate_value" {
	function_name = aws_lambda_function.generate_value.function_name

	input = "{}"
	lifecycle_scope = "CRUD"
}

The CRUD lifecycle scope means it will be called when the resource is created, updated, and deleted, allowing it to reliably handle the lifecycle of the SSM parameter.

Terraform module

What we discussed above is the resources that are needed to manage a generated secret in SSM Parameter Store with a custom Lambda function. But this also yields a reusable structure: a value is generated and stored as a parameter then it is deleted when the stack is deleted. The only configurable parts are:

• how to generate the value
• how to delete the value

Abstracting out all the other parts yields a nice structure and it is available as a Terraform module: ssm-generated-value.

We’ll see how to use the module to generate key pair for CloudFront and to generate IAM user credentials.