Adventures in AWS: Understanding the Price List API

The AWS Price List API

In December 2015, AWS announced a new Price List API that exposes access to current prices for several of their products, including EC2 and RDS. If you’re thinking about building some automation to track AWS spend, and you’re used to working with AWS’s generally marvelous APIs and language SDKs, this sounds like great news. At least, that’s what I thought when I sat down last week to build a tool that would track cost savings for stopped EC2 instances in some of my company’s AWS environments.

Let’s just say the Price List API didn’t match my initial expectations. In fact, calling it an “API” is a bit generous; basically it’s a giant JSON or CSV file you can download for various AWS services that contains price and product data indexed by SKU.

What’s a SKU, you ask? The short answer is it’s a “Stock Keeping Unit“; the even shorter answer is that it doesn’t matter, because nowhere else in the AWS API ecosystem are SKUs for individual resources made available, so you’re not going to be cross-referencing them with anything else. But don’t despair; it is in fact possible to automate price lookups inside the AWS Price List data dump. Let me walk you through the approach I used.

Looking up the Price of a Running EC2 On-Demand Instance

Get the latest EC2 price information

Download the Offer Index file, find the current link to the price file for EC2 and download the data. You’ll get a large ~50MB file that’s broken up into two major sections: product data (descriptions of individual resource categories) and terms (price data). The product and price sections are tied together by SKU, and have a many-to-many relationship; one term can belong to several products, and several products may share a term. The two main offer terms are OnDemand and Reserved. (The Price List does not contain information about Spot Instances or anything purchased through the AWS Marketplace.)

Figure out the cost factors for your instance

Each “product” section in the EC2 data file contains several pieces of data, such as the following entry for SKU ‘DQ578CGN99KG6ECF’:

"DQ578CGN99KG6ECF" : {
 "sku" : "DQ578CGN99KG6ECF",
 "productFamily" : "Compute Instance",
 "attributes" : {
     "servicecode" : "AmazonEC2",
     "location" : "US East (N. Virginia)",
     "locationType" : "AWS Region",
     "instanceType" : "hs1.8xlarge",
     "currentGeneration" : "No",
     "instanceFamily" : "Storage optimized",
     "vcpu" : "17",
     "physicalProcessor" : "Intel Xeon E5-2650",
     "clockSpeed" : "2 GHz",
     "memory" : "117 GiB",
     "storage" : "24 x 2000",
     "networkPerformance" : "10 Gigabit",
     "processorArchitecture" : "64-bit",
     "tenancy" : "Shared",
     "operatingSystem" : "Windows",
     "licenseModel" : "License Included",
     "usagetype" : "BoxUsage:hs1.8xlarge",
     "operation" : "RunInstances:0002",
     "preInstalledSw" : "NA"
    }
 }

That’s a lot of variables! And keep in mind that an EC2 instance does not expose its SKU through an API call, so there’s no easy way to match this product with anything running in your AWS account. Looks like the only way to look up an instance in the product listing is to manually specify the individual attributes.

Fortunately, there are only six factors in the “attributes” list above that actually affect the cost of your instance:

  • Instance type (e.g. ‘t2.micro’)
  • Location (e.g. ‘US East (N. Virginia)’)
  • Tenancy (‘default’, ‘host’ or ‘dedicated’)
  • Platform (‘Windows’, ‘Linux’, ‘RHEL’, ‘Suse’, etc)
  • OS license model (‘bring your own’, ‘no license necessary’ or ‘license included’)
  • AWS-preinstalled software (e.g. SQL Server)

The first three factors – instance type, location and tenancy – are easily accessible through a describe-instance EC2 API call. The Platform factor is semi-accessible – the EC2 API will tell you whether or not the instance is Windows-based, but there’s no way to find out what flavor of Linux you might be running. And as of this writing, there is no programmatic way to determine the license model or AWS-preinstalled software on a running EC2 instance.

Is this a deal-breaker? Maybe. It depends on what you need versus what you know. In my case, I wanted to calculate cost savings across diverse environments with many OnDemand instances of all different types, so I knew if I guessed the cheapest option on any factors I couldn’t determine exactly, I’d wind up with a result that was “at least as good” as the actual cost savings. I ended up with a cost matrix like this:

Instance type
Location
OS
Tenancy
Pre-Installed Software
License models
Windows instances Retrieved via EC2 API Retrieved via EC2 API Retrieved via EC2 API Retrieved via EC2 API Always uses default value “NA” Always uses default value “Bring your own”
Non-Windows instances Retrieved via EC2 API Retrieved via EC2 API Always uses generic value “Linux” Retrieved via EC2 API Always uses default value “NA” Always uses default value “No license required”

Look up the instance price

Once you’ve matched the running instance to a product SKU, you need to look up the price for that SKU. Find the “terms” object in the price file and look for the SKU under “OnDemand”:

"DQ578CGN99KG6ECF" : {
   "DQ578CGN99KG6ECF.JRTCKXETXF" : {
     "offerTermCode" : "JRTCKXETXF",
     "sku" : "DQ578CGN99KG6ECF",
     "effectiveDate" : "2016-08-01T00:00:00Z",
     "priceDimensions" : {
       "DQ578CGN99KG6ECF.JRTCKXETXF.6YS6EN2CT7" : {
         "rateCode" : "DQ578CGN99KG6ECF.JRTCKXETXF.6YS6EN2CT7",
         "description" : "$4.931 per On Demand Windows hs1.8xlarge Instance Hour",
         "beginRange" : "0",
         "endRange" : "Inf",
         "unit" : "Hrs",
         "pricePerUnit" : {
           "USD" : "4.9310000000"
         },
         "appliesTo" : [ ]
       }
     },
     "termAttributes" : { }
   }
 }

What’s going on here? We can see our SKU, but to get down to the pricePerUnit field we have to know a couple of other values:

  • The offer term code, a specific value that describes the term (‘JRTCKXETXF’ for hourly)
  • The rate code, which you can think of as mapping the product and term onto your usage tier. It consists of the product SKU and offer term code, plus another value corresponding to your usage tier, separated by periods.

I couldn’t find a definitive list of offer term codes and usage tier codes outside the API itself, but they seem to be a relatively small number of constants, so you may do well to maintain a list of the ones that map onto your use cases. (AWS breaks down some of the details here.) Once you know these values, you can drill down to find the hourly instance price of $4.931.

Putting it all together

Here’s a snippet of Python to help you read in the offer data and retrieve the instance price for the product shown above:


import json
import requests

url = 'https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.json'
resp = requests.get(url=url)
data = json.loads(resp.text)
hourlyTermCode = 'JRTCKXETXF'
rateCode = '6YS6EN2CT7'

products = data['products']

for sku, properties in products.iteritems():
  if properties['productFamily'] == 'Compute Instance':
    if (properties['attributes']['instanceType'] == 'hs1.8xlarge' and
      properties['attributes']['location'] == 'US East (N. Virginia)' and
      properties['attributes']['operatingSystem'] == 'Windows' and
      properties['attributes']['tenancy'] == 'Shared' and
      properties['attributes']['preInstalledSw'] == 'NA' and
      properties['attributes']['licenseModel'] == 'License Included'):

      price = data['terms']['OnDemand'][sku][sku + '.' + hourlyTermCode]['priceDimensions'][sku + '.' + hourlyTermCode + '.' + rateCode]['pricePerUnit']['USD']

      print price
      return

We haven’t even talked about reserved instances; prices for those get a lot more complicated because there are more offer terms and more possible rate codes. But I hope this post gives you enough information to at least get started digging into the Price List API. I’m glad AWS has shared it with us; they really have exposed some helpful information and it’s a lot better than scraping their website. And knowing them, they’ll continue to iterate and improve on this service over time. I’ll be sure to post updates if and when that happens!

Adventures in AWS: Understanding the Price List API

One thought on “Adventures in AWS: Understanding the Price List API

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s