James O'Neill's Blog

January 11, 2011

Why it is better not to use PowerShell Parameter validation

Filed under: Powershell — jamesone111 @ 12:36 pm

I was giving a talk shortly before Christmas and I was giving some advice based on what I had learned writing my PowerShell library for Hyper-V. I  said

  • Don’t force user to use an object as a parameter – convert names to objects in your code
  • Don’t force users to expand arrays – expand in your code
  • Don’t automatically punish users if a parameter is empty

The corollary from this is Don’t be over proscriptive with parameter checking, especially when it comes to types – which it kicked off an interesting debate. This is best explained with real world examples, so lets take a simple case from my Hyper-V world , and all the background you need to know is

  • A server contains zero or more Virtual Machines.
  • Virtual Machines  can be “Running” or “Stopped” (and in other states)
  • Virtual Machines are represented by VM objects, which have a state property to indicate whether they are running or stopped.

With that in mind I want to look at 3 commands:
Get-VM which returns VM Objects and must, as a minimum, accept  parameters of
-server to specify where to look for VMs and –VMName to filter the selection by name 
(in case you don’t know, if there is no other parameter that starts –VM PowerShell will let you abbreviate this as –VM)
Start-VM will change the the state to running
Stop-VM will change the  state to stopped. 
Before implementing the commands one must decide (among other things):

  • What are valid inputs for the -Server and -VMName parameters in Get-VM ?
  • What inputs should start-VM and Stop-VM take.
  • The Output of Get-VM can become the input of Start-VM and Stop-VM. What should happen if no VMs are found on a server ?

It would be a good idea for you to think about how you’d answer these questions before reading on because I’m going to set out my view here. My view is right, of course, but other views are not necessarily wrong.

To me, flexibility is key. Get-VM , in my view, must allow the person typing the command to specify multiple servers easily.  The most obvious example is
Get-VM  -Server ClusterNode1, ClusterNode2
If parameter validation says the server name must be a single string then you force the user to do something like this
"ClusterName1", "ClusterNode2" | foreach-Object {Get-VM –Server $_}

Not only is the first way shorter but it can be done by a user who has no PowerShell background.  In the same way it should be possible to get those VMs whose names indicate they are located in particular cities
Get-VM  -VM "London*" ,"Paris*"
Yes, I have just sneaked in support for Wildcards. Not allowing this means forcing the user into something like 
Get-VM | where-Object {($_.name –like "London*") –or ($_.name –like "Paris*") }

This may mean more work when we implement the Command (which we do once) to save work when it is run (which happens many times). 

What about the case where we run
Get-VM  -VM "London-DC01" -Server ClusterNode1
but London-DC01 is running on ClusterNode2 : Should this command return an error?
My (limited) background in databases says that if the query runs successfully and finds no matching data, “Nothing” is a perfectly valid output, and more desirable than an exception stopping a script. This begins to answer the question of what should the input to Start-VM and Stop-VM be.   

  1. It would be illogical if they did not accept the output of Get-VM, so the following should  be possible
    $myVMs = GET-VM ; Start-VM –VM $MyVMs
    Start-VM  -VM  (Get-VM –VM "London-DC01")
    Get-VM  | Start-VM
    And should not produce an error if the GET-VM command returns no VMs.
  2. Some might think it acceptable to say the -VM parameter of Start-VM and Stop-VM must contain VM objects. But if it is possible to Get VMs by passing VM name(s) and/or server name(s) then many administrators would say that
    Start-VM  -VM  (Get-VM –VM "London-DC01")
    is too like coding, and not enough like the shell command line they would expect which would be
    Start-VM  -VM "London-DC01"

PowerShell parameter declarations can specify how their type and content should be validated.  “Real” programmers who are used to always specifying the type of everything, tend to grasp this and say “We WILL specify a type (and other validation) in every parameter declaration”. In C#, for example, if someone tries to pass your code something of the wrong type, Visual studio will stop them and tell not to be so silly – their code won’t compile so they never see a ugly red runtime error. Making parameter types agree makes a little more work, but their code will be run many times (hopefully) so that’s tolerable.  But a PowerShell user might type a command in the shell once and then it’s gone, that extra work is less tolerable, and if input which seems logical to them violates rules you have set, the first they they will know is a ugly red runtime error:  any programmer should worry when normal user behaviour produces runtime errors (though a lot will just code to avoid the runtime error, not to adapt their rules to the way users expect to work). 

In PowerShell , in practice I’ve found I can only get this flexibility by allowing anything to be passed in and doing the validation, longhand, in the body of the code. In the VM example that means code which says “Is this an array ? I’ll deal with each item”; “Is this a string ? I’ll treat it as a name which I can turn into an object”; “Is it an Object of the Class I want ? Yippee! I can process it !”; “Is it an object of some other class from which I can get an object of the class I want? Turn it into the right object.”; “Was it anything else? If so do I need to stop execution or can I return nothing ?”  Allowing anything into the function body feels wrong, but I’d ask the question “If the language did not allow you to to specify the parameter type, would you expressly write code to throw a runtime error if the parameter passed wasn’t of the expected type ? If so, might it say ‘If you want to use this as an input, then do X’ ?”.  If the answer is yes to both then Your code should do more cope with normal user behaviour  but if it is yes to the first and no to the second then Validating type might be the right way to go.

By way of a second example I came across some code to create a hash from the content of files, and because PowerShell lets you add properties to objects, the code returned file objects with an added hash , so you do

Get_Some_Files | add-hash | something_to_find_Duplicates_using_hashes

But the person who wrote add-hash refused to allow anything but a file object; I couldn’t do $myFile = add-hash "C:\user\James\myFile.stuff" , but worse  dir –recurse  | add-hash produces an error when it hits a directory objects.  
I could insert a where-object command before the add-hash to filter down to the files, but if that is how the command is going to be used on many occasions, wouldn’t it be simpler for it to do that itself ?  If skipping directories silently bothers you, then catch directories, and use write-verbose to say “Ignoring Directory Xyz”, and if  someone is trying to add a hash to something which makes no sense – like a VM object – really bothers you then catch anything that isn’t a filename, file object or directory object and throw a runtime error further down the script.

As I was writing this Shay Levy retweeted a link to the Windows Scripting Guys’ post on Validating parameters what’s interesting is they show a function which checks phone number formats. So lets put in my phone number formatted as the ITU says it should be

test-parameters "+44 (7801) 8 8 10 10"
Test-Parameters : Cannot validate argument on parameter 'phoneNumber'. The argument "+44 (7801) 8 8 10 10" does not match the "\d{3}-\d{3}-\d{4}" pattern. Supply an argument
that matches "\d{3}-\d{3}-\d{4}" and try the command again.
At line:1 char:16
+ test-parameters <<<<  "+44 (7801) 8 8 10 10"
    + CategoryInfo          : InvalidData: (:) [Test-Parameters], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Test-Parameters

What kind of user understands “Supply an argument that matches “\d{3}-\d{3}-\d{4}” and try the command again.”  ?
Even if we know that the number is ALWAYS American, if the ITU says we can put brackets, dashes and spaces into the number to aid readability shouldn’t we allow (425) 555 1234 or 4255551234 and then clean up the number in the function ?

Over-prescriptive (and often plain wrong) validation comes up in plenty of places: I’ve lost count of web sites which tell me “Credit card numbers must be entered without spaces.” (with all that computing power you think they could strip out the spaces, and maybe even identify Visa and Mastercard automatically). And there are the ones who say names can only contain A-Z and a-z, tough luck if yours has a hyphen, apostrophe or accented character. (being an O’Neill this one drives me nuts. So does not checking for apostrophes and throwing a SQL error).  Realistically we’re not going to get rid of it all. Just don’t add to it, OK ?

January 6, 2011

Making DVDs on Windows 7

Filed under: Music and Media,Windows 7 — jamesone111 @ 2:53 pm

Since I acquired my Windows 7 phone and the Zune software the proportion of my music which has been acquired by download has more than doubled, from almost none to hardly any. It’s probably a generational thing but I like to own the artefacts : CDs and DVDs (and before them I liked to own Albums and VHS tapes).  If one just wants to save money it is easy to record a complete series of a show with Windows Media Center, adding disk space as needed, I can defend bootlegging something that can’t be bought – it preserves something without depriving anyone of income.  But I prefer to have the DVDs if they’re available: as the world shifts to downloading content without artefacts I wonder if one will feel obliged to contribute to the income of others.  I have replaced whole series recordings of  “Doctor Who” with the six-DVD box set – since it can be had for as little as £12.99 it’s not worth making bootlegs into DVDs. I was reminded of this recently…

Just before Christmas I recorded BBC4’s showing of  “Macbeth” with Patrick Stewart in the title role; it runs for just over for 2 1/2 hours and produced a 6.2GB .wtv file. Enough people have looked for it that Amazon UK  suggests “Macbeth Patrick Stewart” in the search box but there isn’t even a pre-announced disk, so I decided to copy it off my hard disk.  My stock of blank DVDs are the 4.7GB single layer variety. This file called for a 8.5GB Dual Layer disk – my new Dell laptop has a PLDS DS-8a4s drive which the maker’s site says supports dual layer, The blank disks are hard to find: Ryman was the only place in central Oxford where I could find them – at £12.99 for a pack of five Imation disks. That’s the same price as the set of Six Doctor Who disks, printed boxed and so on, and it isn’t just high street mark-up, Amazon’s price is the same. Imation’s dual layer disks are more than seven times the price of their single layer cousins. .

I’m left asking Why put myself through the frustration of converting Media Center files to DVD movies? I could have transcoded the file to something smaller than 4.7GB. But instead I bought a pack of disks and set off down the path of making a “proper” DVD. And some of lessons might be worth sharing.

Media Center records a couple of minutes preamble and five of over-run, to cope with transmission times being slightly off. Doing a nice job for a “proper” DVD with nice menus and so on demands that the .WTV files are trimmed (there were no commercials in Macbeth – this being the BBC, but you’d want to cut those out too)- the “Burn CD/DVD” option in Media center can’t trim files, and few editors work with .WTV. The choice is either use Windows Live Movie maker or spend time transcoding into another format before editing. I’ve known sound to go out of sync in transcoding and that’s not the only complication…

image

On the left is part of file properties displayed by Windows Explorer: 704×480 x 60 fps is “NTSC format”. Since the recording was made in the UK it should be 576 line “PAL format”, and this is not the only place NTSC turns up where it should not. 
On the right is Windows media player showing the same file really is in “PAL format”. I put PAL and NTSC in quotes because they really define analogue colour encoding, but they double as shorthand for 576i/25 fps and 480i/30 fps respectively. Note the aspect ratio on the right says 16:9 but if you multiply 576 by 16/ 9 the image should be 1024 Pixels wide. This is because widescreen video is stored and transmitted anamorphically  – it is compressed horizontally and expanded during playback.

DVD video is MPEG encoded and stored in files less than 1GB in size with .VOB extensions. Windows Live Movie Maker can read .VOB files and in an ideal world it would create them and take care of DVD menus. It doesn’t. It passes a .WMV file to Windows DVD Maker which re-encodes it. That intermediate file is in NTSC format DVD maker can write PAL format, but there’s no benefit scaling back to 576 lines when video has been scaled down to 480.

To get round this I tried creating 720×576 custom output format in Movie Maker, but there is no anamorphic encoding option: square pixels yield a letterbox picture, 720 wide x 405 tall with 171 lines of black border. I had to create a format of 1024×576 – which DVD maker would convert back to Anamorphic. I found a better way here: Movie Maker’s pre-defined video profiles can be adjusted: I had Media Encoder and its tools installed, so I made a copy of the existing profile with a .BAK extension (if you change the name, Movie Maker can still read the file, and may ignore the modified version). I fired up Windows Media Profile Editor as administrator and changed the settings from NTSC to PAL, 3MBs to 6 and enabled non-square Pixel output. That was one problem solved. I wanted to output the sound track of another recording, but Movie Maker ignores audio-only profiles created in Profile Editor

My recording of Macbeth lasted 157 minutes and Windows Live Movie Maker warned me:

image

DVDs can use different levels of MPEG compression, even within a single disk but Windows DVD Maker’s compression is fixed. It also seems determined that single or dual layer disks always hold 150 minutes of video.  Live Movie Maker spent over an hour outputting a 151 minute version of Macbeth, DVD Maker spent a similar time re-encoding it, copied it the DVD and failed at the end of the burning process. The same thing happened with a shorter edit. Having wasted an afternoon and two disks, I swore at DVD Maker (and myself for trying to make the DVD which the BBC should be selling) and decided to just copy the .WTV file using Windows Explorer. This failed too, with Windows suggesting I check the drive’s firmware; it is available from Dell, but I already had the current version. Roxio Burn, which Dell supplied with the laptop, gave me error  0x80041024 and a fourth coaster. A quick search revealed Roxio had an update to fix this error with other PLDS drives, but I already had that update thanks to Dell’s update system.

I have a Samsung USB drive for the 3 DVD-less machines I have at home, I plugged that in and explorer copied the .WTV file to my fifth and last Dual Layer disk without problems. For the moment I’m blaming the PLDS DS-8a4s for the burning problems.  My first contact with Dell support got a response that the machine doesn’t have a Dual Layer drive – which is odd considering LiteOn say it is Dual Layer and Roxio and Explorer both see the disk as 8.5GB and try to write to it. I have to see where that ends up.

Keeping .WTV format avoids transcoding issues and saves HOURS, and it will  preserve subtitles transmitted with the programme. Along the way I right clicked the file and chose “Convert to .DVR-MS format”, I couldn’t distinguish between the images in the resulting 3.7GB file, and the 6.2GB original and the conversion was fast (and nearly the same time is saved copying a smaller file to DVD) : I could have avoided the whole business with Dual Layer disks. And I’m more convinced than ever that if I want a nice DVD of something the right thing all round is to buy one.

January 3, 2011

Advice, probably wasted …

Filed under: About me,General musings — jamesone111 @ 4:23 pm

Back in 1997 the Chicago Tribune published a column by Mary Schmich. It was entitled
Advice, like youth, probably just wasted on the young

It has a lot of Parallels with Desiderata, and like desiderata it has been mis-attributed. Baz Luhrmann used it for  “Everyone’s free to Wear Sunscreen” , and if you haven’t come across it before you could use the five minutes in many worse ways than watching the youTube version below.

At the start of a new year I’m probably not alone in wanting to share advice – which  Schmich describes as “Fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than it’s worth.”  I think the two texts are as good as anything I could offer of my own, and offer them here as infographics. You can get them from Skydrive.

Blog at WordPress.com.