Automating vSphere With VMware vCenter Orchestrator

I had some time this past week to read Automating vSphere: With VMware vCenter Orchestrator by Cody Bunch.

I stand by my views that vCenter Orchestrator is one of those uncovered gems thatAutomating vSphere with VMware vCenter Orchestrator unfortunately not enough VMware professionals know how to use and do not make enough use of the product and the vast number of options and functionality available with the product (I must confess, I am just guilty as everyone else here).

Orchestrator has been a free product bundled with vCenter ever since the vCenter 4.0 (vCenter Orchestrator 4.0 | 15 April 2009 | Build 4240) and was installed by default with vCenter.

One of my biggest complaints about the product in its earlier version was that the documentation sucked! Really!! There was barely enough info about how to install Orchestrator correctly.

The documentation has improved immensely over the years and Orchestrator is finally being recognized for the useful tool it actually is. What I was still missing was a clear and concise how-to, down-to-earth explanation on Orchestrator and here is where I think Cody has done an extremely good job with this book.

I really liked the real-life scenarios and use-cases that Cody demonstrated. These scenarios are common to most environments and show how certain “annoying” tasks can be simplified in such a way that not only will your boss be amazed – but you will also have the time on your hands to do more important things (like creating even better workflows).

I would highly recommend this book to anyone who is interested in learning more about vCenter Orchestrator.

Should You Patch the vCenter Server Virtual Appliance?

With vCenter 5 came the release of the the vCenter Server Virtual Appliance. This move was applauded by many due to the fact that VMware made the first step in the direction of removing their dependency on Microsoft’s Windows OS and finally making the move to a Linux based platform.

If you would ask me (and this is my personal opinion – not based on concrete info) the days of a Windows based vCenter server are  numbered, if not in the next version – then the one after that – there will only be a Linux Virtual appliance. It makes complete sense.

But here is where I find lies an issue – which could become quite a serious problem.

VMware have announced that they will no longer be using Update Manager for patching the Guest OS – which I think is quite logical. VMware should concentrate on what they do best – and that is the virtualization stack. Shavlik, WSUS, Landesk (just to name a few) do a pretty good job for providing patches to guest operating systems and their underlying applications.

But what about the vCenter Server appliance? Who should be the one to patch that?

I put in a Support Request yesterday with with the following question

Patch Update for vCenter virtual appliance
The virtual appliance is installed on top of OpenSuse 11 SP1. Since it was released almost 2 years ago and since then there have been a number of bug fixes and security patches released.
Who is the responsible to apply these patches? VMware? Me the customer? If you when will these patches be released? If it is me the customer is there a list of approved patches by VMware that will assure I will not break anything?
In addition when will an updated version of the appliance be released so that it matches the one released for Windows?

The answered I received today – was what I expected.

As a follow up from our phone conversation - with regards to your question of patching the Guest OS of the vCenter Appliance - we would advise to not patch the guest OS directly. However VMware would recommend to apply any vCenter Appliance patches as they are released and if there are Guest OS patches required, they will be included in the Appliance patches.

Maybe there were no patches released for the Operating system you might ask?

I went to the Novell Patch Finder to check to see what patches were released for the OS. I selected only the Mandatory patches that were released after August 23 2011 (I will explain why this date in a moment), and lo and behold – yes there were patches – a whole lot of them (210).


To clarify a few things about how I got to my data.

This is the kernel of the vCenter appliance:

localhost:~ # uname –a
Linux localhost.localdom #1 SMP 2011-02-25 13:36:59 +0100
x86_64 x86_64 x86_64 GNU/Linux

The Linux version of the vCenter appliance

localhost:~ # cat /etc/issue
Welcome to SUSE Linux Enterprise Server 11 SP1 for VMware (x86_64) - Kernel \r (\l).

Why did I choose the date of August 23, 2011?
Here is the list of the last 15 RPM’s installed on the vCenter appliance.

localhost:~ # rpm -qa --queryformat '%{installtime} (%{installtime:date}) %{name}\n' | sort -n | tail -15

1314096829 (Tue Aug 23 10:53:49 2011) oracle-instantclient11.2-odbc
1314096829 (Tue Aug 23 10:53:49 2011) vmware-studio-vami-service-services
1314096831 (Tue Aug 23 10:53:51 2011) VMware-sps
1314096834 (Tue Aug 23 10:53:54 2011) VMware-vpxd-agents-eesx
1314096836 (Tue Aug 23 10:53:56 2011) stunnel
1314096836 (Tue Aug 23 10:53:56 2011) syslog-collector
1314096837 (Tue Aug 23 10:53:57 2011) VMware-vpxd-agents-esx3
1314096838 (Tue Aug 23 10:53:58 2011) vmware-open-vm-tools-kmod
1314096839 (Tue Aug 23 10:53:59 2011) vmware-open-vm-tools-common
1314122043 (Tue Aug 23 17:54:03 2011) vmware-studio-vami-login
1314122043 (Tue Aug 23 17:54:03 2011) vmware-studio-vami-service-core
1314122044 (Tue Aug 23 17:54:04 2011) vmware-studio-vami-service-system
1314122045 (Tue Aug 23 17:54:05 2011) vmware-studio-vami-service-network
1314122046 (Tue Aug 23 17:54:06 2011) vmware-studio-appliance-config
1314122047 (Tue Aug 23 17:54:07 2011) vmware-studio-vami-service-update

VMware say that I should not patch the Guest OS. You might say that not all those 210 patches are relevant. Most probably yes – that is true. But there are several that are relevant – very relevant.

Take for example the Linux Kernel.

The current Kernel is

The kernel has been updated 4 times

These are not minor security patches and holes. These are security updates for the kernel.What would your auditor tell you if you had not installed a patch on a Server of yours for over 18 months?

Would you leave your ESX Servers unpatched for 18 months?

Then my question to you all – and to VMware …

Why is it acceptable to not update the central (and some might say – the most important) component of my virtual infrastructure against known security vulnerabilities??

If you pass over the responsibility to me (the admin) to update my Windows vCenter – then you should either do the same for the vCenter Appliance – or VMware should be the one to provide the patches (regularly!!) – but not leave it in a state that leaves it vulnerable.

I do think that providing your customers with virtual appliance means that you should maintain that appliance – including all aspects – your software and the underlying operating system as well.

Did we all think this through when going to the virtual appliance model? Who is the responsible party for the Guest OS ? The vendor? You? And if so what should the patch cycle policy be? Once in 8 months is not a good track record.

Going the way of using Update Manager to patch the Virtual Appliance will probably be the way to go, but that will still have its implications as well.

Oh yes – one more thing.. Same goes for vCloud appliance, Orchestrator, vCOPS, CapacityIQ, ADM, the vFabric family?

I would be very interested in hearing your thoughts on this..


vExpert 2012–Thank You!

I am proud to announce that I have been awarded the VMware vExpert vExpertaward for the year 2012.
The current list has been posted here.

It is nice to receive such an honor but I would rather like to take this opportunity to express my thanks to a few people in appreciation of this honor.

  1. Alex Maier – for being the driving force behind the vExpert Program, for managing the VMware communities and for putting up with our nagging when we have no-one else to complain to. You are an asset to our community and your hard work is highly appreciated!! (even if we do not say it often and loudly enough)
  2. John Troyer – The Godfather of the vExpert program! Thank you John for all that you do for us, VMware is lucky to have a driving force such as yourself who is so passionate about what you do and we as a community are lucky to have you.
    Oh yes, and please keep those cool SWAG trinkets and access to cool stuff coming!! :)
  3. All those who were awarded the vExpert 2012 award. Thank you!! It is an honor to be amongst such a great group of people. I look forward to meeting those of you who I have not yet had the honor to meet, and working together on exciting opportunities over the next year.
  4. To all those who did not get their vExpert renewed this year. I still hold you all in the highest esteem, and do expect and hope that you will continue to contribute to the community.

As in the past years – I have started a Twitter list of 2012 vExperts and will continue to update it.

Congratulations to all the vExperts of 2012!!


What Happens When No Swap Volume is Available?

Have you ever asked yourself that question?

A customer of mine a small number of VM’s on one host that were continuously crashing. He would power them up and within less than 5 minutes – they would get powered off.

In this case – the host had a swap partition configured not with the default setting but rather as a separate shared datastore.

Swap location

So what happened?

The swap_12 datastore was not accessible (the reason – is not relevant at the moment), and then they powered on a number of machines.

Going to the vCenter event logs showed this:


Notice that the error message is not that descriptive. Inside the guest OS there was no additional clues to the cause for the power off of the VM

But if you look closely you will see the last line mentions this -

*** VMware ESX internal monitor error *** vmk: vcpu-0:Unable to read swapped out pgNum(Ox27dca) from swap slot(Oxl000fb8a) for VM(1717757)
Which of course leads to the direction of the VMware Swap file not being available.
You cannot run a VM without an accessible Swap volume… Winking smile