Adventures in AWS: A Very Scary Error

In this series, I explore some of the everyday challenges facing an AWS developer/sysadmin. Today: I grapple with a gruesome bug in the AWS Tools for Windows Powershell.

Let’s say you have a Powershell script that makes a call to the S3 API:

Write-S3Object -BucketName mybucket -File C:\path\to\file -Key my/key

This call works great when you run it yourself from the Powershell console or ISE. However, when it runs from a scheduled task on your machine under a service account, let’s suppose that to your surprise, this seemingly straightforward call not only fails to upload your file to S3, but also produces the following gnarly-looking exception:

System.InvalidOperationException: A WebException with status ReceiveFailure was thrown. —> Amazon.Runtime.AmazonServiceException: A WebException with status ReceiveFailure was thrown. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.Management.Automation.PSInvalidOperationException: There is no Runspace available to run scripts in this thread. You can provide one in the DefaultRunspace property of the System.Management.Automation.Runspaces.Runspace type. The script block you attempted to invoke was: $true

Wow. That’s like the Holy Grail of scary runtime errors: it contains vague references to web services, an implication that you have run out of threads, and points to a script block that you never wrote. Hypothetically, what would you do next?

If you said “call Amazon”, congratulations, you’re me, what’s that like?

I got stuck on this issue for awhile because my scheduled task was running several sequential calls to S3 and I thought that maybe the “no Runspace available” errors meant I was exhausting some sort of thread count in the underlying Powershell provider layer. I was entirely wrong about that.

Thanks to the cheerful help of a couple of Amazon support techs, I eventually identified and resolved the issue, but fasten your seatbelts because this one is going to get worse before it gets better.

Notice that the error message says Powershell can’t create a runspace to execute a script block containing the following code: $true. Now where did that $true come from? The Amazon guys suggested it might be coming from a lambda function used to check SSL certs. And they were right.

It turned out that unknown to me, the service account user on the target machine had a server certificate validation callback defined. Click the preceding link if you want the gory details about what that means, but essentially it’s a .NET override function to ignore certificate errors–for example, errors from self-signed certs–while invoking requests in a script. Apparently this callback doesn’t play nice with the S3 Powershell Tools (and possibly other AWS services, though I did not notice issues running EC2 Powershell calls).

I was able to reproduce the error under my own user by executing the following command before running the S3 call:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

I dumped the callback into a variable and poked around in the resulting object, which contained information about the aforementioned lambda function:

$result = [System.Net.ServicePointManager]::ServerCertificateValidationCallback

Update Binder Target
------ ------ ------
System.Func`3[System.Runtime.CompilerService... PSConvertBinder [bool] ver:0 System.Func`3[System.Runtime.CompilerServic...

Hey, look at the first line of output! That’s the $true that showed up in the error message.

In order to eliminate the error for my service account on all target machines, all I had to do was make sure to nullify the callback before making my list of S3 calls:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null

I freely admit to not understanding exactly why this callback kicked in when and where it did, much less how it found its way into the environment of my calling user. The promptness with which the Amazon techs assisted my case suggested to me that the problem has come up before. Feel free to chime in if you have additional thoughts or experience with server certificate validation callbacks in or out of Powershell!

Adventures in AWS: A Very Scary Error

5 thoughts on “Adventures in AWS: A Very Scary Error

  1. Dave says:

    Life saver, had to switch it on to handle SAML authentication with an internally signed certificate to authenticate, and then null it after it had authenticated.

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s