Category: VMware

Mar 09

VMware Certificate Authority overview and using VMCA Root Certificates in a browser

By mike in Security, VMware

With vSphere 6.0 the vCenter Virtual Server Appliance (VCSA), now has a component called the Platform Services Controller (PSC). The PSC handles things like SSO and the License Server and ships with its own Certificate Authority called VMware Certificate Authority (VMCA). In this blog post we’ll quickly go over some of the modes of VMCA operation and how to download and install the VMCA root certificate into your browser.

VMCA overview

VMCA issues certificates for VMware solution users, machine certificates for machines on which services are running, and ESXi host certificates. Host provisioning happens when the ESXi host is added to vCenter Server explicitly or as part of the ESXi host installation.

VMware Endpoint Certificate Store (VECS) serves as a local (client-side) repository for certificates, private keys, and other certificate information that can be stored in a keystore. You can decide not to use VMCA as your certificate authority and certificate signer, but you must use VECS to store all vCenter certificates, keys, and so on. ESXi certificates are stored locally on each host and not in VECS. VECS runs on every embedded deployment, Platform Services Controller node, and management node and holds the keystores that contain the certificates and keys.

With VMCA you can deal with certificates in three different ways. For the purposes of discussion we’ll call them

VMCA Default
VMCA Enterprise
Custom

VMCA Default: VMCA uses a self-signed root certificate. It issues certificates to vCenter, ESXi, etc and manages these certificates. These certificates have a chain of trust that stops at the VMCA root certificate. VMCA is not a general purpose CA and its use is limited to VMware components.

VMCA Enterprise: VMCA is used as a subordinate CA and is issued subordinate CA signing certificate. It can now issue certificates that trust up to the enterprise CA’s root certificate. If you have already issued certs using VMCA Default and replace VMCA’s root cert with a CA signing cert then all certificates issued will be regenerated and pushed out to the components.

Custom: In this scenario VMCA is completely bypassed. This scenario is for those customers that want to issue and/or install their own certificates. You will need to issue a cert for every component, not unlike you do today for 5.5 when using 3rd party certs. And all of those certs (except for host certs) need to be installed into VECS.

In Default and Enterprise modes VMCA certificates can be easily regenerated on demand. In Default and Enterprise modes VMCA certificates can be easily regenerated on demand.

Important: For vSphere 6.0 the procedure for installing these certificates has changed from vSphere 5.x. In order to make this procedure less painful a new Certificate Manager tool is shipped as part of vCenter for Windows and VCSA. It will be located here:

Windows: C:\Program Files\VMware\vCenter Server\vmcad certificate-manager
Linux: /usr/lib/vmware-vmca/bin/certificate-manager

The procedure will be fully documented and will be the topic of a future blog article.

Downloading VMCA’s Root Certificate

Today when you connect to VCSA you get a web page like this:

or this

Ugly, “feels” insecure, gets the security guys all wound up. (and we can’t have that happen!) Let’s get the root certificate from the VCSA and VMCA and install it in the browser so we don’t see these pages anymore.

Get the root certificate

Open up your web browser and go to the VCSA home page. I’ve outlined in red the link you’ll want to click on.

What you’ll get now is a folder in your Downloads folder called “certs”. In that folder are two files. It may also download as a zip file, depending on your browser. You may have to rename the file “download” to “download.zip”.

The file ending in .r0 is the Certificate Revocation List in DER format. You can view the CRL by running

openssl crl –in <filename>.r0 –text –noout

The file ending in .0 is the root CA certificate in PEM format. You can view the CA cert by running

openssl x509 –in <filename>.0 –text –noout

Installing the Root Certificate in the Firefox browser

The root CA is the one we’ll install in our browser. By doing this, the certificate presented by VCSA will chain its root of trust to the imported VMCA root CA certificate.

In Firefox I opened up the certificate list in Advanced settings, selected “Authorities”

I then clicked on Import, selected the .0 file and was presented with this option.

Select “Trust this CA to identify websites” and click OK. Your root CA is now imported and if you open the VCSA web page you’ll find you are no longer presented with the option to verify the certificate. You may need to close and reopen the browser.

The process is similar for other browsers and is well documented for adding the root CA to Windows, Linux and Mac key stores if you prefer to do it that way.

Note: You’ll need to access the VCSA by its FQDN and not its IP address (like I normally do in a lab environment!). Otherwise you’ll get an error like this:

Note that any resource that presents a web page that has its certificate issued by VMCA will now show up as trusted.

For example, host certificates will be valid as well!

Recap

So, to summarize what we’ve learned:

VCSA now has its own certificate authority called VMCA
You can install the root certificate of VMCA in your system or browser
All vSphere components like vCenter, ESXi, solution users, etc can be issued certificates from VMCA if running in Default or Enterprise mode
VMCA can be bypassed if you don’t want to use it, however you’ll need to do more steps to manage your certificates
Regardless of which method, all certificates need to be installed into VECS with the exception of ESXi hosts.
A Certificate Manager tool is provided to help you manage your 3rd party certificate installations

I hope this was helpful. Give it a try in your lab environments and introduce your security people to these new concepts and options. I’ll be curious to hear what they say so send me an email at mfoley at vmware dot com with their feedback!

Thanks for reading,

mike

Mar 09

vSphere 6.0 Hardening Guide – Overview of coming changes

By mike in Security, VMware

The vSphere Hardening Guide provides guidance on how to securely deploy VMware vSphere in a production environment. The vSphere Hardening Guide also serves as a foundation upon which regulatory compliance objectives are built. These organizations map compliance guidelines with vSphere Hardening Guide guidelines.

Hardening Guides are an industry recognized method of implementing stricter security to meet regulatory and local security standards above and beyond frameworks like Common Criteria.

Version 6.0 of the vSphere Hardening Guide is the next step in the evolution of the guide. A goal of the vSphere 6.0 Hardening Guide is to make the guide easier to implement and assess.

The intent of this article is to go over some of the major changes that come with the new 6.0 guide prior to its release. Consider this your “heads up”.

Separation of programmatically configured and testable “settings” from operational guidance.

Within previous guides are a mixture of

Operational guidance – How you use the product
Programmatic Guidance – Set this value to “True”

Operational guidelines present unique challenges.

They can be addressed or mitigated in many ways
They are generally left open to interpretation
In order to implement them they may require cross-functional support across an enterprise
They are usually not something that can be automatically tested and verified. Instead, they rely on an individual signing off that they are done correctly

Consider the example of “Run your vCenter and ESXi management interfaces on a separate management network” as something that meets each of these bullet points.

For the vSphere 6.0 Hardening Guide, the “operational” or “best practices” guidelines have been moved to vSphere Security documentation. This, in no way, lessens their importance! However, each one of these will generally be specific to your environment and as such require conversations with your security professionals and your auditors on implementation details and attestation.

What are left are the things that can be checked via API’s, CLI’s and other tools for attestable values. In some cases, the default values are sufficient and the check is an audit to ensure that the value is correct. In others, it’s a new or different value.

Risk Profiles

Guidelines are grouped in “Risk Profiles”. Risk Profiles indicate the relative increase in security provided by the guidelines. Risk Profiles are grouped as follows.

Risk Profile 1: guidelines that only be implemented in the highest security production environments, e.g. top-secret government or military, extremely sensitive data, etc.

Risk Profile 2: guidelines that should be implemented for more sensitive production environments, e.g. those handling more sensitive data, those subject to stricter compliance rules, etc.

Risk Profile 3: guidelines that should be implemented in all production environments

Risk Profiles in Operational Guidance

In order to create a bridge from the old guide to the new guide + documentation, a separate spreadsheet containing the Operational Guidance, a link to the new location in the vSphere documentation for each guideline and the Risk Profiles values of each of those guidelines will be published.

But why do I need a Hardening Guide? Isn’t it secure by default?

Some customers ask “Then why not ship vSphere at Risk Profile 3 by default?!”. That’s a great question. vSphere, without any changes from the Hardening Guide, already meets Common Criteria standards of security.

By default, security is built into the product. Consider the isolation of virtual machine memory and instructions or the disabling of risky parameters or strong password policies. See the ESXI Security Whitepaper or my session INF2336 from VMworld 2014 for a deeper dive on security. The Hardening Guide builds upon that.

Like many other products in the industry, vSphere “out of the box” is shipped optimized for Proof of Concept and pilot deployments. It is when vSphere is put into a production environment that Hardening Guidelines should be reviewed and applied as necessary.

What other changes have been made to the guide?

Top to Bottom Review

vSphere 6.0 is the first major release in a few years. As such, it was a good time to do a complete review of the Hardening Guide. Guidelines were updated, added or removed based on the incorporation of newer technologies and removal of older technologies. This was done with the great support of the vSphere R&D team and a number of product managers.

New Taxonomy

The Hardening Guide is consumed not only by individuals but also by other software such as scripts, VMware Configuration Manager, vRealize Air Compliance and a number of 3^rd party solutions and organizations.

The primary driver for a new taxonomy was

Ease of production
Ease of use by solutions
Ease of automation

Production of the Hardening Guide has changed from editing the Excel spreadsheet by hand to a database-driven solution that provides numerous advantages such as versioning, collaboration and other benefits.

From a programmatic standpoint I believe the new taxonomy will make the consumption of the Hardening Guide easier by solutions and automation tools.

The previous generation of the guides grouped guidelines by vSphere components. In the Excel spreadsheet this meant separate tabs such as “ESXi”, “vCenter”, “vNetwork”.

Moving forward this changes to a flat taxonomy. Rather than grouping into separate tabs the vSphere 6.0 Hardening Guide will be a single sheet.

Guidelines will now be referenced as “Component.Guideline” For example, what used to be

enable-ad-auth

becomes

ESXi.enable-ad-auth

Unique Names

One of the problems this solves is giving each guideline a unique name. Within the vSphere Hardening Guide there were some guidelines are “the same” (e.g. “configure-NTP”). With this taxonomy, these values become unique. For example:

Before:

configure-ntp

ESXi

After:

ESXi.configure-ntp

As mentioned, the vSphere 6.0 Hardening Guide will now be delivered as a single sheet Excel workbook rather than separate sheets. (That method will also be available for 6.0 but deprecated in future releases) Other methods of delivery such as CSV, PDF and XML are being considered. Your feedback will help define them.

Wrap up

In closing, the biggest goal of the vSphere Hardening Guide is to move towards making security easier to implement and manage. This is a first step of a process. It sets the stage for more things to come. By separating out the Operational Guidance from the Programmatic Guidance and creating a new taxonomy I hope to enable more interesting things moving forward.

A beta of the vSphere 6.0 Hardening Guide will made available in the near future.

Your feedback is valuable to me. You can reach me on Twitter at @vspheresecurity or via email directly at mfoley at vmware.com.

Aug 04

Zero to Windows Domain Controller in 4 reboots!

By mike in PowerCLI, Powershell, VMware, Windows

Hey there,

Yes, it’s time to wow and amaze you with my coding skills! Prepare to be amazed! Or, more appropriately, roll your eyes with my lame ability to hack other peoples code into what I need to get done, quickly. Writing code for me is painful. Oh, like high school physics I can follow what’s going on pretty well, but sitting down and writing? Yea, that’s a challenge.

I needed a Windows Domain Controller for what I’m working on in my work lab. Figuring that I’ll probably need more Domain Controllers for future labs, I thought it would be a good time to automate it. I looked online for a simple PowerShell script that would do what I want and to my amazement, I couldn’t find anything other than pointers to a lot of commands. And, in many cases, those commands were either wrong or broken.

Lots of Steps

Because I work at VMware and this Domain Controller will be running as a VM, I’ll be using VMware VM Templates to create the virtual machines. I built a Windows 2012 template using some simple best practices steps documented here. I wanted to be able to right-click on a template and simply deploy a Domain Controller using GuestOS customization.

Simple, right? Actually, not that difficult. It’s mainly about getting all the steps done in the right order and meeting the requirements of each step.

First, you have deploy the VM with the guest OS customization.
Reboot
Then add the AD bits to your Windows 2012 system so that you can enable the appropriate roles going forward.
You’ll probably want to rename the system.
Reboot
Then install the AD Forest
Reboot
Then add the roles and configure networking/AD/DNS/etc..
Reboot

How do you automate all that?

RunOnce to the Rescue

RunOnce is a Windows registry setting that automatically sets up Windows to run certain tasks when a system starts up or a user logs in. vSphere Guest OS Customization uses this so that when you deploy from a template, you can set the Windows system to run further setup scripts to modify the OS. In this example, you see how I’ve set up my DC-Builder.PS1 script to run via RunOnce.

The command I’m running is:

%SystemRoot%\system32\WindowsPowershell\v.10\Powershell.exe -executionpolicy bypass -file “C:\dc-builder.ps1”

Let’s run that down so everyone gets what’s happening. After deploying from Template and choosing this Guest OS Customization Specification, Windows 2012 will do a sysprep. After that is completed, Windows will reboot and run the RunOnce command above. That sets the ball in motion.

In the command, we’re running the DC-Builder.PS1 script using PowerShell.exe and specifying that it be run with the execution policy of bypass. That’s because the DC-Builder.PS1 script is not signed. Otherwise, it wouldn’t run.

You’ll notice that I’m running DC-Builder.PS1 from C:\. That’s because I built my template with the file copied to that location. Why? Because I didn’t want to have to deal with floppy images or network locations. Oh, sure, you can and then you can just change the location, but for me, I found this was easier. You want to do it differently? By all means, knock yourself out. :)

Now you’re probably asking “Ok, how do you automate all the other reboots?” Well, I iteratively use RunOnce! I just nest the hell out of it!

DC-Builder creates a couple of scripts and sets up the first one to run after it reboots the system the first time. Then that script calls the next one to run. DC-Builder creates the files using the PowerShell “@” operator.

Here’s an abbreviated example. Don’t just cut and paste this.

$InstallForestbat = @"
REM -----------------------------InstallForest.BAT------------------------------
REM Creating a .BAT file to change to bypass, run the commands, then change
REM back to remote
Powershell.exe set-executionpolicy bypass -force
SET FileToDelete="c:\InstallForest-transcript.txt"
IF EXIST %FileToDelete% del /F %FileToDelete%
powershell -noprofile -file c:\InstallForest.ps1 &gt; c:\InstallForest-transcript.txt
REM Powershell.exe set-executionpolicy RemoteSigned
"@
$InstallForestBat |Out-File -FilePath c:\InstallForest.bat -Force -encoding ASCII
$AdminKey = "HKLM:"
$RunOnceKey = $AdminKey + "\Software\Microsoft\Windows\CurrentVersion\RunOnce"
Set-ItemProperty -Path $RunOncekey -Name Foo -Value "C:\InstallForest.bat"

See how I’m using the “-encoding” qualifier? PS1 files don’t need it but .BAT files most definitely do.

DC-Builder creates four files to be run.

InstallForest.BAT – This is set up to run via RunOnce after DC-Builder completes and the system reboots. It runs InstallForest.PS1.
InstallForest.PS1 – This installs the AD Forest, sets up the running of Post-DCCleanup.Bat via RunOnce and reboots the system.
Post-DCCleanup.BAT – It runs Post-DCCleanup.PS1.
Post-DCCleanup.PS1 – This finishes off the rest of the steps that need to be run after installing the AD Forest and reboots the system.

DC-Builder.PS1, InstallForest.PS1 and Post-DCCleanupPS1 also create transaction files in C:\ so that you can check for errors.

Note: You’ll notice that DC-Builder.PS1 sets up the Administrator account to log in automatically. You need this, at least initially, to get all the scripts to run. They need to run under a user context. Because this is primarily for lab environments, I set up my VM’s to log in automatically just to save steps. Yea, the “security guy” said that. Whatever. :)

How do I get started?

Edit DC-Builder.PS1 to set up your IP address, system name and any other custom setting you’d like.
Create your Windows 2012 Template VM.
Copy DC-Builder.PS1 to C:\
Shutdown the Windows 2012 Template VM.
Attach the Windows 2012 ISO image to the VM <This is key!
Select “Convert to Template”
Create a Windows Customization Specification as called out above (or Import the XML)
Deploy a new VM from template using the Customization Specification
The VM will reboot 4 times and you’ll be left with a Windows Domain Controller, ready to roll.

Download the script and guest OS customization XML

~~GuestOS XML: https://dl.dropboxusercontent.com/u/10618705/DC%20Windows%202012.xml~~

~~DC-Builder.ps1 https://dl.dropboxusercontent.com/u/10618705/DC-Builder.ps1~~

These files have now been moved to GitHub. You can get them here:

https://github.com/mikefoley/DC-Builder

Give it a try and let me know what you think and what it’s missing. It could use some more documenting but if you’ve played around with PowerShell before, it should be all pretty straightforward. I’ve also credited where appropriate where I got some of the inspiration for the script and where I’ve blatantly copied code. Many thanks to some helpful hints from Brian @vTagion Graf, especially when dealing with variables embedded when using the “@” operator.