Saturday, March 15, 2014

The purpose of scripting: Using why we script to inform how we script in PowerShell

To properly understand how to write a given script, we have to understand why we are writing it.

That's easy, you say.  I'm writing this script to collect data about my servers.  Or to perform daily maintenance tasks.  Or to build new servers.  Or to automate adding new users to our systems.

But those answers are incomplete.  They don't address the big picture.

I'm writing this script to collect data about my servers so that I don't have to do it manually.  Or to perform daily maintenance tasks more reliably and at a time of day when my staff is asleep.  Or to build new servers quickly, consistently, and cheaply.  Or to automate adding new users to our systems directly empowering my HR staff and freeing up IT support resources for more challenging tasks.

That can be generalized into a standard definition of why we do what we do.

The purpose of scripting is to assist in performing some function by optimizing the resources of the system in which the function is performed.

That sentence is a little dense, but it is important to let it all sink in, because this what we all do professionally, and it is what everything else here flows from.  So let me say that again a little slower.

The purpose of scripting
is to assist
in performing some function
by optimizing the resources
of the system in which the function is performed.

This allows us to analyze our purpose, and to develop a way of thinking about our scripting that will help us make better, easier decisions about our scripts.

The key to understanding what that means and the implications it has on us and our scripts, is the fact that the system is not just the box.  It isn't just the box on our desk or the big box of a datacenter with lots of smaller boxes within.  The system includes the people.  It includes the people for whom the function is performed.  It includes the people responsible for performing the function.  And it includes the people who create and maintain the script.  It includes us.

So when I sometimes say that scripters are lazy, that is just playful shorthand for saying we optimize the resources of the system.

Software developers do this as well, but they balance the requirements differently than scripters, resulting in different design decisions and coding styles.

Here are the four and a half guiding principles supporting the purpose of scripting.

1. The script needs to reliably perform the desired function.
2. The script needs to be easily writeable.
3. The script needs to be easily readable, understandable, and maintainable.
4. The script needs to optimize the resources it uses to run.

These are the requirements you need to balance when designing and writing your script.  You can't do them all perfectly.  You focus on just one at your peril.  You can't balance them the same way on every script; each script is different, each function is different, and each system is different.

But once you understand the principles, and understand that they each only exist to serve the purpose, the reason we are scripting in the first place, it becomes much easier to make the many decisions and trade-offs we make with every script.

1. The script needs to perform the desired function.

The script need to work.  It needs to work well.  It need to work reliably.  That's obvious.

But how well, how thoroughly, and how reliably?  That's different for every script, as the balance with the other requirements is different in every script.

If a script is going to run unattended in the middle of the night on a customer-facing production system, it needs to be well-tested and have lots of built-in error handling.  But if I'm sitting at my desk watching it run, with time to tweak it and run it again if it fails, it doesn't need to be as rigorous.

How much time I spend making the output look pretty varies greatly between output that gets automatically emailed to the CEO and output that I'm personally reading off the screen.

In some cases, the script does not need to perform the entire function.  For an ad hoc report, it might be faster to just have the script give me the raw data, and I can pretty it up more effectively and efficiently in Excel.

2. The script need to be easily writeable.

The whole point of a script is to automate something, to make it less work for me.  If writing the script is a lot of work, that defeats the whole purpose of the script.

Jeffrey Snover and his team and their successors did a wonderful job creating PowerShell with scripters in mind, making it powerful but easy to use, comprehensive but discoverable.  Paradoxically, one of the ways they made it easier for us, was by making it possible to do things the hard way.  This allows us on the ground to decide when and where it is appropriate and necessary to put in the extra work or extra complexity, and where we can do things more simply.

Commands and snippets that are easy to remember, easy to type, forgiving of variations of syntax or runtime environment, and easy to troubleshoot are preferable to those that are not.  Readability is not just about the future reader of the script (as in the next section).  It's also about working with your script while you are writing it.  Effective use of white space can help considerably when you are looking for an errant parenthesis.

Using aliases on the PowerShell command line can be a wonderful time saver.  Writing elegant, dense one-liners is uniquely satisfying.   But when writing scripts, we save more time by not using most aliases, by writing out the commands in more human-friendly language, by adding white space and line continuations that make it easier to work on the script.

Always using the right variable types makes your scripts go faster.  But always using a variable type that is easy to remember, easy to type, and easy to read, makes me go faster.  Which is more valuable and which is the better resource to expend on this project:  200 milliseconds of CPU at runtime or 20 minutes of my time?

In some cases, it is better to spend the 20 minutes shaving 200 milliseconds off of a subroutine.  For software developers, this is usually the case.  For scripters, this is almost never the case.

3. The script needs to be easily readable, understandable, and maintainable.

The system doesn't just include the people writing and using the script.  It also includes the person that has to pull up the script next week or next year and figure out what it does or why it doesn't work anymore or how to adapt it to new requirements or how to borrow chunks of it to adapt for other scripts.

Everything you do while writing your script needs to balance the needs of whoever might someday read or modify your script.

If you are working on the command line or working on an ad hoc script that is going to be deleted as soon as you are done with it, feel free to write it in whatever way is easiest for you to write.  Though I think you will find it easier in the long run to use a single writing style for everything.  My only concessions to these are fewer comments, less consistent capitalization, and simpler variable names.

If you are writing a script that will be read again--even if it's just by you picking up the next day where you left off--you need to write for the reader.

Use comments.  A competent, experienced scripter can look at line of code and figure out what it does.  But that takes time and effort.  It takes less time and effort to add that information to the script while you are writing it.  What is the script assuming about the environment it is running in or its inputs or how it will be used or what the results will be used for?  Add comments explaining all of that.  You will save hours or days of troubleshooting time when the assumptions are no longer valid next year.

Use white space.  Vertical and horizontal.  Use line continuations to break up dense commands.  Make it easy to see the sub clauses of complex statements, to see what goes together in chunks, to see the hierarchy of your code.

Use human-style words and syntax wherever possible.  Use variable names that not only describe their use well, but also help make complete sentences in your commands.  Spell out commands and parameter names instead of using aliases, except for those rare cases where the alias is easier to read and understand than the full command.

Given a choice and all else being equal, use the command or syntax that is more intuitive to the person modifying it later.  You already have a comment stating what you are doing; if it isn't obvious how you are accomplishing that, explain it in the comment.  If you made an unusual choice for a non-obvious reason, explain it in a comment, so that there are not unintended consequences later when someone "fixes" it.  If the need for a section of code or chosen method will become obsolete in the future and can or should be changed when circumstances change, put it in a comment.

If your comments are so numerous or wordy or redundant that they start to interfere with reading and understanding the script, cut them back.  Everything in balance.

(That's the one I'm counting as one and a half if you're wondering.  The three needs covered are too small and overlapping to be discussed or counted separately, but together they are too big to be just one.)

4. The script needs to optimize the resources it uses to run.

Don't use up all of the resources on the computer.  Don't use too much memory, disk or CPU.  Use what you need from what's available, but don't be a pig about it.  Don't use so much resources that you are interfering with other things happening on the server or virtual host or SAN or network.  Rarely is your script the most important thing happening in the system.

But don't be shy about using the resources you have to do the job you need to do.  Moore's Law is forever giving us more power.  Use it.  The whole point of the script is to use those resources instead of more expensive resources.

The server is a resource.  The user is a resource.  You are a resource.  The future is a resource.  Write your script in such a way that you are optimizing your function across all of the resources of the system.

I'll talk about specific examples and scripting styles and decisions in future articles.

Wednesday, December 4, 2013

You want me to stick this PowerShell variable…where?

I was reading an old blog article by Jeffrey Snover and found a hidden gem.  Buried in the middle of the article is a side note with a single paragraph about PowerShell variables that are not stored in memory.  Fascinated, I have been playing and experimenting and researching, and this is what I have figured out so far.

A PowerShell variable name, such as $X, points to a spot in memory where your variable object is stored.  But here's the cool thing.  The pointer doesn't have to point to memory.  It can point to a file.  (It has the potential to point other places as well.  More on that later.)

Strings

Here is the syntax for storing a variable in a file.

${D:\Temp\CoolVariable.txt} = "very blue"

No quotes.  Just a dollar sign with the file path and name in curly braces.

And it works the same for using the variable in a sentence.

$Full = "The sky is " + ${D:\Temp\CoolVariable.txt} + " today."

Numbers and dates

It stores the value of the variable, and only the value, in a text file.  On the surface, this makes it appear that this only works for string variables.  But thanks to the magic of PowerShell's powerful dynamic type conversion, it actually means it works for any simple variable type, any variable that has a single independent value.  So it also can be used for all of the different types of numbers and dates, though you need to be aware that it will come back to you as a string which may or may not dynamically convert back to a number or date depending on the context.

For example

${D:\Temp\CoolVariable.txt} = 27.6
100.3 + ${D:\Temp\CoolVariable.txt}

Will work the way you expect.

But if you reverse the order of these two mismatched variable types, PowerShell will assume you want the result to be a string, and concatenate the two instead of adding them together.  So you need to do the conversion explicitly.

[double]${D:\Temp\CoolVariable.txt} + 100

It does NOT work with Boolean values

It does NOT work with Boolean values.  The problem is false.  That is, the problem is with converting $False to and from a string.

The PowerShell designers made some great decisions about how to convert things into [Boolean] true/false values.   Empty strings, empty arrays, the number zero, $Null, and $False all evaluate as $False when converted to [Boolean] values.  Non-empty strings, populated arrays, non-zero numbers, $True, and more or less every other object evaluate as $True.  This is very nice to have for simplifying syntax of conditionals.

If ( $X )

Is short for "If $X exists and contains a tangible thing". ("Tangible" being a relative term in a computer-modeled world. ; )  I like that functionality.  I use it a lot.

But part of what I said up there is "non-empty strings… evaluate as $True."  Even when the non-empty string is the word "False".

So if we convert $False to a [string], it becomes "False", and when we convert that back to a [Boolean], it becomes $True.  You can demonstrate that like this.

[boolean][string]$False

And of course the same thing happens when you assign $False to a file variable and then try to use it as a [Boolean].

So don't store [Boolean] values in file variables.

Arrays

It does works with arrays.

${D:\Temp\CoolVariable.txt} = Get-ChildItem D:\Stuff | ForEach { $_.FullName }

This stores a list of the full file path and names of the files found in D:\Stuff.

And they come back out as an array of strings.

ForEach ( $FilePath in ${D:\Temp\CoolVariable.txt} ) { $FilePath }

If you stick an array of numbers in there, you can explicitly convert them as you pull them out or as you use them.

ForEach ( $EmployeeNumber in [int[]]${D:\Temp\CoolVariable.txt} ) { $EmployeeNumber }

Or

ForEach ( $EmployeeNumber in ${D:\Temp\CoolVariable.txt} ) { [int]$EmployeeNumber }

Complex objects

It works nicely as a shortcut for complex object types when you only wanted the name of the object anyway.  Whenever you try to assign an object to a file variable, it uses the object's built-in .ToString() method to convert it as it saves it.

For example, if you want to save a list of services that are running,

${D:\Temp\CoolVariable.txt} = Get-Service | Where { $_.Status -eq "Running" }

This will give you a list of the service names, not the display names, because the name is what you get when you run the .ToString() method on a service object.

If you want to save the display names instead, and you are still using PowerShell 2.0, we can do that.

${D:\Temp\CoolVariable.txt} = Get-Service | Where { $_.Status -eq "Running" } | ForEach { $_.DisplayName }

If you are using PowerShell 3.0 or later, we can tighten that up a bit:

${D:\Temp\CoolVariable.txt} = ( Get-Service | Where Status -eq "Running" ).DisplayName

You may have been tempted to try using Select-Object there instead.  If you did, you noticed unexpected results.

When you use Select to choose only certain properties, the objects are converted to type PSCustom.  PSCustom objects act differently when you try to stick them in a file variable.  The object is converted to a hash table, and the hash table is converted to a string, and the string is saved to the file variable.  If you do this with an array of objects, you get an array of hash table strings.

This is almost useful.  This allows you to save all of the properties of a complexes object, or at least those property values that can be easily converted to strings.

But it's not really useful, because when you try to use the variable, what you are going to get is a single string for each object, representing the hash table that represents the custom object, and it does not automatically reconstitute itself.  To use the objects, you have to do something's like this.

${D:\Temp\CoolVariable.txt} = Get-Service | Select DisplayName, Name, Status
$Services = ${D:\Temp\CoolVariable.txt} | ForEach { New-Object PSObject -Property ( ConvertFrom-StringData $_.Substring( 2, $_.Length - 3 ).Replace( "; ", "`n" ) ) }

Which is far more trouble than it's worth.  Use Export-/Import-CSV instead.

$S = Get-Service | Select DisplayName, Name, Status
$S | Export-CSV D:\Temp\Services.csv -NoTypeInformation
$Services = Import-CSV D:\Temp\Services.csv

Variable variable names

One disadvantage is that we are now using paths and file names as a variable name, which means they can't vary.  You don't embed variables within variable names and expect PowerShell to parse them.

This does NOT work:  ${$LogShare\$LogName}

Well, we can make it work by wrapping entire lines in Invoke-Expression, but it gets messy.

Invoke-Expression "`${$LogShare\$LogName} = [array]`${$LogShare\$LogName} + `$LastResultCode"

Not worth the effort.

Shared variables

One way to use a file variable is when you need a variable that can survive a catastrophic failure of the script.  Or any other time you want the variable to persist between subsequent runs of the script. 
Or between runs of the script on different servers.

There might be an issue if two servers tried to access the variable at exactly the same time, but you would have that with any file-based system for sharing the data.  If they never ran at the same time, it would work just fine.

For example…

I have a report that needs to run every night.  A scheduled task kicks off a script to generate the report at 10 PM on server A.  For redundancy, the same script runs at 11 PM on server B.

I don't want server B to tax the database re-running the report if sever A was successful.  But my users have a tendency to move, rename, or otherwise modify the report almost as soon as it's generated, so I can't look to the report file to confirm it's done.  Coincidentally, a file variable is perfect for this.

The script ends with

${\\MainFS\TJC\ReportLastRun.txt} = Get-Date

And starts with

If ( [datetime]${\\MainFS\TJC\ReportLastRun.txt} -gt ( Get-Date ).AddHours( -2 ) ) { Exit }

That way, if the script successfully ran less than two hours ago, it immediately exits without re-running the report.

File variables vs. Get-Content and Set-Content

You may have noticed that working with file variables is similar to using Set-Content and Get-Content.  When you play around with them, you find that they work exactly the same.  This is because behind the scenes, they are the same.  They both work by calling the iContentCmdletProvider interface of the FileSystem provider.

Measuring the performance of the two leads to some interesting results.

This

$ServerList = ${D:\Temp\ServerList.txt}

is actually faster than this

$ServerList = Get-Content D:\Temp\ServerList.txt

Where else can I stick it?

I asked Jeffrey Stover that question, and rather than file a restraining order, he graciously explained that while it had never been a high enough priority for the PowerShell team to extend the functionality to other places, he did make sure that it was possible for anyone to extend it to other things, like the registry or databases.

Then he showed how to do it in MUMPS, which wasn't exactly helpful, but his enthusiasm was inspiring.

I am planning to see if I can use this to create a simpler, more intuitive way to store information in the registry.  I believe it can be accomplished by implementing the iContentCmdletProvider interface on the PowerShell registry provider.  This will be my first foray into such an extreme enhancement to PowerShell, so it will be an adventure.  I'll tell you all about it when it's done.

Pause in PowerShell

So what ever happened to "pause"?

Back in the old batch file days, you could stick in a "pause", and your script would, well, pause.  It would sit there and wait until someone hit the any key.  Most often you would stick it at the end of a batch file, so that the user could see that the script was done and read any output before the little black box disappeared.

It would sometimes be helpful to do that in our little blue boxes as well.  But PowerShell doesn't have a Pause-Script cmdlet.

.Net to the rescue!  I keep telling you that everything in PowerShell is an object.  And when I say "everything", I mean everything.  Even PowerShell itself.  $Host is the built-in variable representing the PowerShell host itself, the thing that is running the script, the little blue box as it were.  And we can access some of the things it can do, such as watching for keyboard input.

The $Host has a UI, $Host.UI.  And the .UI has a UI, $Host.UI.RawUI.  I have no idea why it has two layers.  The .RawUI has some methods.  Pipe it to Get-Member to see them.

$Host.UI.RawUI | Get-Member

.ReadKey() is the one we want.  .ReadKey() tells the RawUI to get the next key press.  Which means it has to sit and wait for someone to press the any key.
Let's try it without parameters.

$Host.UI.RawUI.ReadKey()

Well, that does wait for the any key, but when we press the any key, it tells us more than we ever need to know about the key we pressed.  So let's pipe the output and send it to nowhere.

$Host.UI.RawUI.ReadKey() | Out-Null

Better.  That prevented most of the information from showing up on the screen, but it did delay the key we pressed.  It would be nice if we could make that go away as well.

Maybe if we fed it into a dummy variable.

$X = $Host.UI.RawUI.ReadKey()

Nope.  That just fed the key information we had previously out-nulled into $X.  That can be useful if we want to capture the specific key pressed for some reason.  But the key that was pressed still displayed on the screen.

Maybe we need parameters.  When we ran Get-Member on the RawUI, it showed us that for parameters it wants…  Oh.  It didn't really tell us.  The information was truncated.  We can fool around with the Get-Member command to get it to give up the information, or we can turn to MSDN, which we were going to do in a few paragraphs anyway.

After some Googling and clicking around, we end up at the MSDN article on the "PSHostRawUserInterfaceReadKey method."  And we find that .ReadKey() wants TeadKeyOptions.  What the heck are those?  Well click on it.  Hmm.  That didn't help much.  Apparently the "Windows PowerShell Managed Reference" section isn't formatted an internally linked as logically and intuitively and discoverable as the .Net Framework library.  (Fortunately, most of the time we are searching for information that resides in the .Net library, but not this time.)  so after a little more Googling and clicking, we find the article on "ReadKeyOptions enumeration."
We see there are four options we can include:  Whether to react when the key is pressed or when it is released (you have to specify at least one of the two); whether or not to echo the key press to the screen; and whether or not to intercept Ctrl-c as a key press.

We really don't care whether we respond to the key up or the key down, so just pick one.  We do not want the key to print to the screen, so we want the no echo option.  Don't include the AllowCtrlC option, because we might want to use the pause to stop the script.

We could use the ReadKeyOptions enumerations, using bit-wise or to combine them.

$Host.UI.RawUI.ReadKey( [System.Management.Automation.Host.ReadKeyOptions]::NoEcho `
    -bor [System.Management.Automation.Host.ReadKeyOptions]::IncludeKeyDown )

But that's crazy.

Enumerations just represent integers.  In this case, each enumeration is a power of two, as the ReadKeyOptions parameter is storing each of the options in a single bit within.  We can ask PowerShell what our enumerations represent.

[System.Management.Automation.Host.ReadKeyOptions]::NoEcho `
    -bor [System.Management.Automation.Host.ReadKeyOptions]::IncludeKeyDown

Returns 6.

So for brevity, we could just use:

$Host.UI.RawUI.ReadKey( 6 )

To avoid having to remember 6, flipping all four bits by using 15 actually produces the same desired result.

$Host.UI.RawUI.ReadKey( 15 )

I am tempted to use this one.  It is unlikely that I will ever use .ReadKey() for anything else, so I won't need to remember or understand what the 15 does.  But I don't like it when I don't understand things, so the next time I look at this and can't remember, I'll waste time looking it up.

So I will compromise and use this:

$Host.UI.RawUI.ReadKey( "NoEcho, IncludeKeyDown" )

PowerShell's powerful dynamic type conversion capabilities convert the string to the appropriate enumerations and thence into the integer they represent.  It's easier to read than the full enumerations and more informative than the integers.

.ReadKey(), of course, reads the key, and then gives it to us.  But we didn't really want the key, we just wanted the pause while waiting for it, so we banish it to null.

And that gives us our (almost) simple one line replacement for "pause":

$Host.UI.RawUI.ReadKey( “NoEcho, IncludeKeyDown” ) | Out-Null

I experimented further and found two other one-liners that do the same thing.  I'll spare you the blow-by-blow on discovering these.

[console]::ReadKey($True)

And

cmd /c pause

Except it doesn't work in ISE

If you have been following along and trying this in a PowerShell console, you think I'm brilliant.
But if you have been trying it in PowerShell ISE, you think I'm an idiot, because all three methods have thrown an error every time, no matter what you try.

The problem is, the "console" in the bottom of the ISE is not a real PowerShell console.  It's a simulated PowerShell console.  PowerShell ISE does a very good job of simulating the PowerShell console, but one of the few things that it can't do is give you direct control of the raw console UI, because there isn't one.

So what is my brilliant solution for a snippet that would work equally well in the ISE?  Unfortunately, I don't have one.  (I told you that you would think I was an idiot.)  All of my research thus far has only suggested alternatives to the desired behavior, such as a "Click OK to continue" pop-up box to replace our "Hit any key to continue."

Such solutions have their place, but seem unsatisfactory when we were searching for something specific, something so simple that a batch script could do it with a single word.  If you have any ideas, please comment below.

Thanks to Charles Christianson for asking the question that I ultimately failed to answer.  Thank you, Charles, for making me look like an idiot.