VMware vSphere design - the book

This is a joint post of three prominent writers in the virtualization community:

Forbes Guthrie, Scott Lowe and Maish Saidel-Keesing.

For the past 6 months we have been working on a joint project. It has actually been kept pretty quiet, and it is now time to make this public.

Previous VMware vSphere books have focused on how to master the technology, deep-diving into certain elements and giving tips & tricks that help you manage your virtual infrastructure.

But we felt there was something missing in all these books. How to design an infrastructure, accounting for all the elements that you need to consider.  For example:

  • What kind of servers should I use?
  • Which storage - NFS/iSCSI/FC?
  • How to scale an vCenter server appropriately?
  • and many more similar questions.

This is how VMware vSphere design was born...

VMware vSphere Design

The three of us collaborated on the book, to not only explain how to configure each element of your infrastructure, but to make you think about all the options available, and how each choice can impact the overall design. It should help you find the right solution for your environment – because no “one size fits all”.

It is the only book focused on designing VMware vSphere implementations. It is written for engineers and architects who plan, install, maintain and optimize vSphere solutions.

The book details the overall design process, server hardware selection, network layout, security considerations, storage infrastructure, virtual machine design, and more. We debate the merits of scaling up servers versus scaling out, ESX versus ESXi hypervisors, vSwitches versus dvSwitches, and FC, FCoE, iSCSI or NFS storage. We show you which tools can be used to monitor, to plan, to manage, to deploy and to secure your vSphere landscape. We run through the design decisions that a typical company might face, and question the choices you come to. The book is packed with real-world proven strategies. VMware vSphere Design examines how the virtualization architecture for your company should ideally look – be it a newly deployed environment or an optimization of the existing infrastructure.

The VMware vSphere design is now available for pre-order on Amazon and will be in the stores around the middle of March 2011.

We would like to thank Jason Boche for acting as the technical editor for the book.

We hope you enjoy reading this book, as much as we enjoyed writing it.

Forbes, Scott and Maish.


Do You Really Need the vMA?

This post was triggered by a very informative conversation I had with William Lam this week. Thanks William!

<Begin Quote>

vMA - vSphere Management Assistant, is a virtual machine that includes prepackaged software such as a Linux distribution, the vSphere command‐line interface, and the vSphere SDK for Perl. Basically it is the missing service console for ESXi. But it's more than that too.

This allows administrators to run scripts or agents that interact with ESX/ESXi and vCenter Server systems without having to explicitly authenticate each time. vMA can also collect ESX/ESXi and vCenter Server logs and store the information for analysis.

</End Quote>

Ever since VMware announced that ESXi will be be the de-facto platform going forward for the hypervisor starting from 4.1 - a large amount of speculation was raised about how to continue forward, and how we will continue to perform what we were doing up until now on the service console - with the console that is gone.

VMware is pushing all of us to move to the vMA to do all this administration.

Now let me share a trail of thought with you. (I would like to stress, these are my thoughts - and are based on my speculation)

Does the vMA have a future? Will there be a need for it down the road? Up until now one of the two limiting factors that were not available in was esxcli and esxtop. Since PowerCLI 4.1.1 this is no longer a limitation.

ESXCLI now available through PowerCLI and ESXTOP Available Through PowerCLI

So my question to you is, what do we need to keep the vMA for? If someone would tell me because of the built-in syslog server which is available - that does not sell me.

The amount of API objects that are currently available for use in PowerCLI - that are not available or exposed through the Perl SDK, has continuously been rising with each new version released.

What is the reason to keep the vMA around in the future? What (if anything) is the necessity for you to have a vMA? Is there anything that you cannot do today with PowerCLI that can only be done with the vMA?

And if that is case - why go down that route and add another VM that has to be managed, has to be patched and will use up resources on your vSphere infrastructure?

I look forward to your comments and your thoughts.


Windows 7/2008 Deployment - KMS and MAK Keys pt. 3

I would close up what we have learned from part 1 and part 2. During my work with the VMware Support on this case, one of the questions I asked, was is it possible to to inject a parameter into the customization process, but the only option was to import a full sysprep file. This would in essence break a good deal of the automation process, such as the rename of the OS to match the deployed VM name. So this was out of the question.

Unfortunately there was not much I could do here. So I rebuilt a new VM. But I had to take some certain precautions. Once your KMS is registered in the DNS and new OS connects to the network and gets an IP, it automatically looks up for a DNS record for the KMS server and will activate - which is exactly what I did not want.

The machine was started disconnected from the external network. I applied a MAK Key to the machine and then connected it to the network. Once a machine is activated with a MAK key, it will not try to contact a KMS server, so I was set.

So just to re-cap

  1. Build your VM
  2. Customize the OS - including your default profile settings
  3. Do not connect it to the network
  4. Install a MAK Key
  5. Connect to the network
  6. Activate the MAK key

The last part of the puzzle was how to change the MAK key to KMS for all the VM's that will be deployed from this template? This was actually the easiest part. In the Customization Spec you can enter a License Key, and yep you guessed it here you insert the MAK key.

Machine is deployed, the Customization Spec injects the xml (with the MAK key) into the VM, the vm is sysprep'ed (without a problem because you have 3 ReArms left) and when it comes back up automatically is activated with the KMS server.

A few things still puzzle me though. All of this worked - flawlessly on vCenter 4.0 - I did not have to go through this whole process with MAK and KMS and USD and BS! So did something change? Was there a change in the way the machines are customized from the previous version?

The deployment and templates were a bit icky after the upgrade.  I could not edit of my previous templates in vCenter - they all had to be re-registered VMware KB………

I also had another issue of not being able to deploy any templates at all (Windows / Linux) with customization until the vCenter Server got a kick in the butt and was restarted.

Small little things that do not add up. I am happy to say that I did manage to educate VMware support a bit on how this works, and in turn I learned a decent amount in the process of how the VMware deployment process works as well. All in all it was an educating experience all around. I was told that the information learned from this case will possibly be used in a VMware KB for all of our benefit.

On to solve the next problem. Hope you enjoyed this series of posts.

Windows 7/2008 Deployment - KMS and MAK Keys pt. 2

Continuing the problem from part 1.

Over the past 8 months this template has been updated with Microsoft security patches (released once a month) and during that time the template OS was activated - while connected to the network.

After the upgrade to 4.1, I noticed that I could no longer deploy a Windows 2008 R2 template with customization. (There were several issues here. In the beginning I could not deploy a template at all - but after a restart of the vCenter server and an SR submitted to VMware on the issue that was solved). But when customizing the the template - either with a manual Customization Spec or with a previous one we had been using for almost a year, the customization would not work. None of it.

Now this can become extremely annoying. Because once you get used to working with deploying a template with a customization spec, to do it manually takes time and can be cumbersome and prone to inconsistencies.The VM name would not be changed, network settings would not work, VM's were not being deployed with their vNIC's connected. There of course is a workaround to the whole problem, to do it manually but as I said before, this is not ideal.

Last week I finally found the reason this was happening, and partly for my own benefit (documentation) and to share the experience, let me explain what was happening.
When a Windows (7 or 2008) OS is activated with a MAK License Key, the OS is automatically set with a ReArm count of 3. This you you can see with a running the following command.

C:\Users\msaidelk>cscript c:\Windows\system32\slmgr.vbs -dlv
Microsoft (R) Windows Script Host Version 5.8
Copyright (C) Microsoft Corporation. All rights reserved.

Software licensing service version: 6.1.7600.16385
Name: Windows(R) 7, Enterprise edition
Description: Windows Operating System - Windows(R) 7, VOLUME_MAK channel
Activation ID: xxxxx-xxxxx-xxx-ad1e-7fe15931a8dd
Application ID: xxxxx-xxxxx-xxx-983e-d6ec3f16059f
Extended PID: xxxxx-xxxxx-xxx-132882-03-1033-7600.0000-2292010
Installation ID: xxxxx-xxxxx-xxx0249086468222046584484281164682806770
Processor Certificate URL:
Machine Certificate URL: http://go.microsoft.com/fwlink/?LinkID=88339
Use License URL: http://go.microsoft.com/fwlink/?LinkID=88341
Product Key Certificate URL: http://go.microsoft.com/fwlink/?LinkID=88340
Partial Product Key: xxxxx
License Status: Licensed
Remaining Windows rearm count: 3
Trusted time: 04/01/2011 08:56:51

An OS which is licensed with a KMS License will look like this

C:\Windows\system32>cscript slmgr.vbs -dlv
Microsoft (R) Windows Script Host Version 5.8
Copyright (C) Microsoft Corporation. All rights reserved.

Software licensing service version: 6.1.7601.17105
Name: Windows Server(R), ServerStandard edition
Description: Windows Operating System - Windows Server(R), VOLUME_KMSCLIENT channel
Activation ID: xxxxx-xxxxx-xxx-97be-d11a0f55633f
Application ID: xxxxx-xxxxx-xxx983e-d6ec3f16059f
Extended PID: xxxxx-xxxxx-xxx-03-1037-7600.0000-0132010
Installation ID: xxxxxxxxxxxxx073476703659683995395732413372005
Partial Product Key: xxxHC
License Status: Licensed
Volume activation expiration: 253260 minute(s) (175 day(s))
Evaluation End Date: 01/12/2011 01:59:59
Remaining Windows rearm count: 1
Trusted time: 16/01/2011 23:27:12

Key Management Service client information
    Client Machine ID (CMID): a41eaf55-64a2-4509-ac85-2118804023f0
    KMS machine name from DNS: ilkms.maishsk.local:1688
    KMS machine extended PID: xxxxx-xxxxx-xxx-021634-03-1033-7600.0000-0132010
    Activation interval: 120 minutes
    Renewal interval: 10080 minutes
    KMS host caching is enabled

Once your OS has been activated through a KMS server - your ReArm count will be set to 1. If you try and change it back to a MAK key thereafter - the Rearm count will still be kept as 1.
So what happened? Somewhere along the way, my template was was activated with a KMS Key - which meant I only had 1 ReArm left.

During the deployment process of a VM with a Customization Spec, a number of files files are injected into the VM by vCenter.

Expanding C:\Windows\TEMP\vmw979D.tmp\guestcustutil.exe
Expanding C:\Windows\TEMP\vmw979D.tmp\imgcust-reboot.exe
Expanding C:\Windows\TEMP\vmw979D.tmp\sysprep\guestcustutil.exe
Expanding C:\Windows\TEMP\vmw979D.tmp\sysprep\sysprep.xml
Expanding C:\Windows\TEMP\vmw979D.tmp\sysprepDecrypter.exe

These files are then moved later on to c:\Sysprep
Moving directory 'sysprep' to 'C:'
vCenter takes the information passed from the the Customization Spec, decrypts the info and creates a new sysprep.xml file which is then in turn called from the customization process
Executing command C:\windows\system32\sysprep\sysprep.exe /quiet /generalize /oobe /reboot /unattend:C:\sysprep\sysprep.xml

Now how do I know all of this? A bit of reverse engineering. The deployment logs are located in C:\Windows\Temp\vmware-imc. The Sysprep logs are located here: C:\Windows\System32\sysprep\Panther

Two log files are created during the Sysprep process - setupacct.log and setuperr.log The first is the activity of the Sysprep and the second reports the errors that occurred.
When deploying a VM with an activated KMS license - I was getting these errors from the setuperr.log

2010-01-14 09:42:42, Error      [0x0f00a4] SYSPRP WinMain: Unable to parse command-line arguments to sysprep; GLE = 0x36b7[gle=0x000036b7]
2010-01-14 09:42:57, Error      [0x0f0043] SYSPRP WinMain:The sysprep dialog box returned FALSE
2010-01-14 09:43:04, Error      [0x0f0060] SYSPRP ParseCommands:Found unsupported command line option '/?'[gle=0x000036b7]
2010-01-14 09:43:04, Error      [0x0f00a4] SYSPRP WinMain: Unable to parse command-line arguments to sysprep; GLE = 0x36b7[gle=0x000036b7]
2011-01-12 11:18:48, Error      [0x0f0082] SYSPRP LaunchDll:Failure occurred while executing 'C:\Windows\System32\slc.dll,SLReArmWindows', returned error code -1073425657
2011-01-12 11:18:48, Error      [0x0f0070] SYSPRP RunExternalDlls:An error occurred while running registry sysprep DLLs, halting sysprep execution. dwRet = -1073425657
2011-01-12 11:18:48, Error      [0x0f00a8] SYSPRP WinMain:Hit failure while processing sysprep generalize internal providers; hr = 0xc004d307

Even though there was 1 left on my ReArm count this was not working. A quick syprep on the machine attested to that fact.

We will now close this series with how the problem was solved in Part 3.


Windows 7/2008 Deployment - KMS and MAK Keys pt. 1

I have been dealing with an issue that has been bugging me for quite a while. It has to do with the deployment of Windows 7 or Windows 2008 R2 VM's and License Activation.

First let's describe the environment and situation. The whole infrastructure is at 4.1, vCenter and all ESX hosts as well. It was upgraded recently from 4.0. 7 months ago I created my templates - with all my customizations. Some of these customizations included different OS settings, menu sizes, toolbars etc. In order to copy all of these settings to the default profile, once upon a time all you had to do was to copy that current user profile to the Default User Profile, and subsequently every user than would logon thereafter would have all those settings defined. Starting with Windows 7 and 2008 (perhaps also Vista - I am not sure, we gave this version a skip) this was not the recommended way to this. There is a detailed Microsoft KB that explains the method - which is to Sysprep the machine and provide a setting in the unattend.xml file which will copy the profile. This can be done manually but that is not recommended. You should use the WAIK. This tool is available from Microsoft.

After installing the software you attach a Operating System Image and create an answer file. In that answer file you can search for the Copy Profile option and set:


My first problem I ran into then was that I was not aware of the fact you can only Sysprep a machine 3 times, thereafter you will not be able to do it any more. This of course led me to a problem of after making changes to the VM and Sysprep'ing again and again and again - I could no longer continue with this Template (Thank you VMware for snapshots!!).

Failure occurred while executing 'C:\Windows\System32\slc.dll,SLReArmWindows', returned error code -1073425657

There is a solution to this issue which can avoided by adding an additional flag to the unattend.xml file <SkipReArm>1</SkipReArm>.

The Activation grace period is typically 30 days. It begins after Windows Setup finishes and the computer boots for the first time. While there is no limit to the number of times that the Sysprep command can run on a computer, there is a limit to the number of times Windows can be rearmed. Typically, a system can be rearmed only three times. Using this setting enables you to run the Sysprep command multiple times without resetting the activation clock.

And now I finally had my template customized, and ready for deployment.

Now over the past 8 months the we have implemented a Microsoft KMS server. Before that let us go into what has changed since the days of XP/2003 in terms of licensing/activation. With Windows 2003/XP organizations were provided with a VL (Volume License) Key. That meant I could put the serial number in the image / template / Sysprep file for each and every machine. Once installed there was no further action needed. Starting with 2008/Windows (remember we skipped Vista) we were provided with 2 different License Keys, a MAK key and a KMS Key.

Frequently Asked Questions About Volume License Keys. A MAK key is one that you add to the OS and that has to be activated with Microsoft. This can be done over the internet or with the VAMT (I will not go into how to provide a proper licensing mechanism for your OS's in your organization). This is not an automatic process and for a small amount of OS's it is quite suitable. Christian Mohn has created a good explanation of how to use this tool. But when you are talking about hundreds and thousands of operating systems, this does not scale well. Also not always do you want to open your firewall to allow each and every computer to activate with a Microsoft Server somewhere "out there". Therefore Microsoft introduced the KMS (Key Management Server). It is a role that you can install on Windows Server 2008 which will act as your activation server for all OS's in the organization. There is a record in the DNS that is created which each new OS will look for by default and if found will activate the OS - automatically. Much better for a bigger environment.

We will continue into part 2 of this series with the issue at hand.

A milestone - 1000 Twitter Followers


When I joined on: Thu 21 Aug 2008 07:23 I would have never thought that anyone would actually have much interest in what I have to say, evidently I was wrong.

On to the next 1000!!




I am a delegate - Gestalt IT Tech Field Day 5

I was honored to receive an email last night from Stephen Foskett offering me a spot on on the next
Tech Field Day in February in San Jose.

I am happy to be able to accept the offer, so I will hopefully be making my way to the US in the beginning of February.

The list of delegates are as follows.

Sean Clark

Bill Hill
Virtual Bill
Gestalt IT

Tom Hollingsworth
The Networking Nerd

Devang Panchigar

Chris Wells

W. Curtis Preston
Backup Central

Maish Saidel-Keesing

Eric Siebert

Greg Stuart

I am really looking forward to this and expect some more blog posts to come.


What do you use for your VMware monitoring?

I would be interested in hearing your opinions on what you use for monitoring your VMware environment. I am conducting a study on what tools you find useful for monitoring and Capacity Management/Planning

If you would be so kind as to answer the poll below and if you could leave a comment below on why you made your choice - it would be highly appreciated.

What do you use for monitoring your VMware environment?

Thanks for the co-operation.


Virtualization Challenges in 2011: Under Blogger Scrutiny

I wanted to give you a heads-up on a webcast that will be held on Wednesday January 12th, 2011
09.00 GMT

You are all cordially invited to join, it should be an interesting talk!

Gary Thornton, Simon Seagrave, Steve Chambers, Richard Garsthagen, and myself for an hour of discussion about Virtualization Challenges in 2011


or on the site directly. Looking forward to see you there!