Serverless Workflows on AWS: My Journey From SWF to Step Functions

AWS Lambda functions can only run for a maximum of five minutes. This must be distinctly understood, or nothing wonderful can come of the story you are about to hear.

The Problem

This past summer, my team and I set out to build an internal software system used for deployment testing on AWS. The application would run a large number of workflow executions in parallel each night and might perform a few one-off executions during the day – maybe six hours total use out of every twenty-four, with only a small fraction of that time spent doing actual compute tasks. Trying to scale, manage and spend money on EC2 instances for that workload didn’t interest us. We wanted to run our whole workflow process end-to-end on AWS Lambda.

And we did. Heaven help us, we did. This is our story.

Continue reading “Serverless Workflows on AWS: My Journey From SWF to Step Functions”

Serverless Workflows on AWS: My Journey From SWF to Step Functions

AI, automation and the merry-go-round of the mind

Is there such a thing as too much automation?

At a family wedding the other weekend, I fell into conversation with a relative who has several decades of experience in the aerospace industry. He bemoaned a growing problem among the younger engineers who work for him. It seems that some of these highly-paid professionals have not developed the ability to look at a finished piece of work and say – “That doesn’t seem right” – because they rely on their advanced computer systems to do the validation. When the computer makes a mistake, they do not have the breadth of experience to realize it.

This point resonated with me for the simple reason that I experience it every day. I’m a professional automator – I automate software processes for a living – and I spend a lot of time inside the Amazon Web Services cloud. AWS handles the compute, storage and networking details for me so I can focus on higher-level tasks, which is both nice and worrisome. Nice because I can get more done in less time, worrisome because I don’t get the opportunity to grapple with the implementation details of server and network virtualization. I understand those things on a theoretical level, but I don’t get to play with them much, and this sometimes hampers my grasp of what’s really going on beneath all the automation.

Continue reading “AI, automation and the merry-go-round of the mind”

AI, automation and the merry-go-round of the mind

Could serverless computing work in a public volunteer cloud?

Serverless computing

What’s that old schoolyard rhyme? “AWS and Azure, sitting in a tree, I – A -A – S,  P – A -Y – G. First come VMs, then containers, then come stateless microservices running on public cloud infrastructure at fractions of a cent per second.” Or something like that.

Anyway, application deployments are getting lighter, backend microservices are getting smaller, and now many development shops are moving toward “serverless architectures” in which dynamic computational tasks are handled using a few cycles on somebody else’s managed server. As of 2016, the public cloud giants (AWS, Google Cloud and Microsoft Azure) all have their own “serverless services” that allow you to buy processing time for cheap. And I do mean cheap – a million AWS Lambda requests per month, each lasting five seconds, will set you back about $10.62.

Developers gravitate toward this approach because it’s scalable, cost-effective and requires little to no infrastructure maintenance. In AWS, you might deploy an application with data stores in RDS or DynamoDB, static web content hosted in S3, an API Gateway directing traffic and Lambda functions running the business rules – look Mom, no servers!

But wait a minute. Is a pay-as-you-go public cloud really the only place to run serverless compute functions? After all, a handful of computer scientists have been running little pieces of code on distributed computers for years, at a price even Lambda will never beat: free.

Continue reading “Could serverless computing work in a public volunteer cloud?”

Could serverless computing work in a public volunteer cloud?

Notes from the Summit: Day 3 Summary

This is the third and final installment of my daily notes from the 2016 Powershell and DevOps Global Summit. Day 1 wrap-up is here, and Day 2 is here.


9 AM (Don Jones): What DevOps Really Looks Like

Don Jones is an effortlessly entertaining speaker who’s not afraid to eviscerate ideas he finds stupid, sort of like (and I mean this in the best possible way) the Donald Trump of Windows IT. He is also a man who thinks clearly about DevOps, a subject usually buried in fuzziness and hype. (I highly recommend his short e-book on DevOps from an ops perspective.)  In this session, he gave a typically animated fireside chat about what a DevOps culture really is: an embrace of the idea – foreign to many ITIL shops – that failure is inevitable and change is good. (He brought down the house with a line about ITIL being IT governance borrowed from the DMV.)

Continue reading “Notes from the Summit: Day 3 Summary”

Notes from the Summit: Day 3 Summary

Notes from the Summit: Day 2 Summary

Being the second installment of my daily notes from the 2016 Global Powershell and DevOps Summit. Day 1 wrap-up is here.


Morning Sessions

9 AM (Neema Saeedi, Windows Server & Services Program Manager): Nano Server and Remote Management

Nano Server is coming, ready or not. And it looks like Microsoft’s new skinny server option is a lot readier than it was at last year’s Powershell Summit in Charlotte. Last year’s big reveal was the fact that Microsoft would offer a web-based remote management console for Nano, including familiar tools like Registry Editor and Task Manager that won’t be available on the headless server itself, as well as a Powershell prompt right in the browser.  That management console is now in preview, and Neema Saeedi from the Nano team spent some time today demonstrating the interface as well as providing updated stats about Nano’s current size and deployment time.

Continue reading “Notes from the Summit: Day 2 Summary”

Notes from the Summit: Day 2 Summary

Notes from the Summit: Day 1 Summary


The Event

The Global Powershell and DevOps Summit strikes you as the sort of conference run by people who have spent a fair share of their professional lives attending terribly-organized technical conferences and swearing to themselves that one day, ONE day they would create an event that lived up to their own expectations. They have certainly exceeded mine. As Don Jones said at one point, 90% of this success comes from choosing the right venue. The Meydenbauer Center is centrally located to Bellevue’s hotels and food, the staff is highly organized, and the catered food is extraordinary. The fact that the event is relatively small, not a 20,000-person cattle call, also really helps, as does the fact that there are no third-party vendors trying to sell you stuff. I would recommend this event to anyone who thinks that technical conferences have to be thinly-disguised, poorly-executed sales pitches. It’s just a good experience.

Continue reading “Notes from the Summit: Day 1 Summary”

Notes from the Summit: Day 1 Summary