Adventures in AWS: Understanding the Price List API

The AWS Price List API

In December 2015, AWS announced a new Price List API that exposes access to current prices for several of their products, including EC2 and RDS. If you’re thinking about building some automation to track AWS spend, and you’re used to working with AWS’s generally marvelous APIs and language SDKs, this sounds like great news. At least, that’s what I thought when I sat down last week to build a tool that would track cost savings for stopped EC2 instances in some of my company’s AWS environments.

Let’s just say the Price List API didn’t match my initial expectations. In fact, calling it an “API” is a bit generous; basically it’s a giant JSON or CSV file you can download for various AWS services that contains price and product data indexed by SKU.

What’s a SKU, you ask? The short answer is it’s a “Stock Keeping Unit“; the even shorter answer is that it doesn’t matter, because nowhere else in the AWS API ecosystem are SKUs for individual resources made available, so you’re not going to be cross-referencing them with anything else. But don’t despair; it is in fact possible to automate price lookups inside the AWS Price List data dump. Let me walk you through the approach I used.

Continue reading “Adventures in AWS: Understanding the Price List API”

Adventures in AWS: Understanding the Price List API

Invoke Pester Tests Serverlessly with AWS Lambda and SSM

Pester and CI

If you’re doing Windows scripting in 2016, you’d better be using PowerShell. And if you’re writing PowerShell scripts, you’d better be checking them into source control and covering them with Pester tests.

It turns out that you can do more with Pester than just run tests manually at the console. As part of a continuous integration (CI) process, you may want to invoke Pester tests on a remote server and report the results up through the build chain. Handily, you can export Pester test output in an NUnit XML format that modern CI systems like Jenkins understand.

But what if you’re not using a build server to invoke Pester? What if your CI setup is … dun dun dun … “serverless”?

Continue reading “Invoke Pester Tests Serverlessly with AWS Lambda and SSM”

Invoke Pester Tests Serverlessly with AWS Lambda and SSM

Adventures in AWS: Automating Service Limit Checks

In this series, I explore some of the everyday challenges facing an AWS developer/sysadmin. Today: Can you script service limit checks in Powershell even without service-specific APIs?

THE PROBLEM
AWS sets default limits on the number of resources you can create in a given account. The limits apply to EBS volume storage, EC2 instance reservations and total CloudFormation stacks, among many others. The full list of default limits–at least the ones you can change—is available here. Changing a limit requires submitting a support ticket and can take anywhere from a few minutes to several days (or never), depending on how much additional capacity you’ve requested. You can view your current EC2 service limits inside the AWS Console for your account.

I’m okay with submitting support tickets to change a limit, but usually I don’t find out if I’ve reached that limit until I kick off a CloudFormation template containing multiple resources and the stack rolls back with an error like this:

Continue reading “Adventures in AWS: Automating Service Limit Checks”

Adventures in AWS: Automating Service Limit Checks

Adventures in AWS: How to break the EBS Snapshot feature

In this series, I explore some of the everyday challenges facing an AWS developer/sysadmin. This week: how does the EBS Snapshot API stand up under heavy load?

UPDATE 6/9: Turns out my original fix wasn’t quite enough to overcome the bug 🙂 See the revised solution below.

The Background
If you keep important data on EBS volumes, you’ll want to back them up using EBS’s nice Snapshot feature, which allows you to grab a point-in-time backup of a volume and dump it in S3. It’s an incremental backup, meaning you are only snapping–and being charged for–the blocks changed since the last snap. If you are using EBS for something fast-changing and critical like transaction logs, it’s not a bad idea to snap as often as possible. If you’re snapping many volumes many times a day, you’ll also want to tag them with an identifier or two for sanity’s sake. AWS allows you to place up to ten tags on a resource, so what happens if you have, say, fifty thousand snapshots in an account, each with four to six tags?

Continue reading “Adventures in AWS: How to break the EBS Snapshot feature”

Adventures in AWS: How to break the EBS Snapshot feature

Adventures in AWS: Measuring Ephemeral Storage

In this series, I explore some of the everyday challenges facing an AWS developer/sysadmin. First up: how do you aggregate information about EC2 instance store volumes?

The Problem
Recently, I got an urgent request to estimate the total amount of ephemeral storage attached to hundreds of EC2 instances across several AWS accounts. At first I got that sinking feeling. I don’t have access to billing data. I definitely didn’t have time to remote into each box and look up metadata by hand; security constraints prevented me from using Powershell Remoting, and not all the EC2 instances I was interested in were Windows machines anyway. No, I needed an AWS API solution. But did such a solution exist?

Continue reading “Adventures in AWS: Measuring Ephemeral Storage”

Adventures in AWS: Measuring Ephemeral Storage