Mike Foley

With vSphere 6.0 the vCenter Virtual Server Appliance (VCSA), now has a component called the Platform Services Controller (PSC). The PSC handles things like SSO and the License Server and ships with its own Certificate Authority called VMware Certificate Authority (VMCA). In this blog post we’ll quickly go over some of the modes of VMCA operation and how to download and install the VMCA root certificate into your browser.

VMCA overview

VMCA issues certificates for VMware solution users, machine certificates for machines on which services are running, and ESXi host certificates. Host provisioning happens when the ESXi host is added to vCenter Server explicitly or as part of the ESXi host installation.

VMware Endpoint Certificate Store (VECS) serves as a local (client-side) repository for certificates, private keys, and other certificate information that can be stored in a keystore. You can decide not to use VMCA as your certificate authority and certificate signer, but you must use VECS to store all vCenter certificates, keys, and so on. ESXi certificates are stored locally on each host and not in VECS. VECS runs on every embedded deployment, Platform Services Controller node, and management node and holds the keystores that contain the certificates and keys.

With VMCA you can deal with certificates in three different ways. For the purposes of discussion we’ll call them

1. VMCA Default
2. VMCA Enterprise
3. Custom

VMCA Default: VMCA uses a self-signed root certificate. It issues certificates to vCenter, ESXi, etc and manages these certificates. These certificates have a chain of trust that stops at the VMCA root certificate. VMCA is not a general purpose CA and its use is limited to VMware components.

VMCA Enterprise: VMCA is used as a subordinate CA and is issued subordinate CA signing certificate. It can now issue certificates that trust up to the enterprise CA’s root certificate. If you have already issued certs using VMCA Default and replace VMCA’s root cert with a CA signing cert then all certificates issued will be regenerated and pushed out to the components.

Custom: In this scenario VMCA is completely bypassed. This scenario is for those customers that want to issue and/or install their own certificates. You will need to issue a cert for every component, not unlike you do today for 5.5 when using 3rd party certs. And all of those certs (except for host certs) need to be installed into VECS.

In Default and Enterprise modes VMCA certificates can be easily regenerated on demand. In Default and Enterprise modes VMCA certificates can be easily regenerated on demand.

Important: For vSphere 6.0 the procedure for installing these certificates has changed from vSphere 5.x. In order to make this procedure less painful a new Certificate Manager tool is shipped as part of vCenter for Windows and VCSA. It will be located here:

Windows: C:\Program Files\VMware\vCenter Server\vmcad certificate-manager
Linux:        /usr/lib/vmware-vmca/bin/certificate-manager

The procedure will be fully documented and will be the topic of a future blog article.

Downloading VMCA’s Root Certificate

Today when you connect to VCSA you get a web page like this:

or this

Ugly, “feels” insecure, gets the security guys all wound up. (and we can’t have that happen!) Let’s get the root certificate from the VCSA and VMCA and install it in the browser so we don’t see these pages anymore.

Get the root certificate

Open up your web browser and go to the VCSA home page. I’ve outlined in red the link you’ll want to click on.

What you’ll get now is a folder in your Downloads folder called “certs”. In that folder are two files. It may also download as a zip file, depending on your browser. You may have to rename the file “download” to “download.zip”.

The file ending in .r0 is the Certificate Revocation List in DER format. You can view the CRL by running

openssl crl –in <filename>.r0 –text –noout

The file ending in .0 is the root CA certificate in PEM format. You can view the CA cert by running

openssl x509 –in <filename>.0 –text –noout

Installing the Root Certificate in the Firefox browser

The root CA is the one we’ll install in our browser. By doing this, the certificate presented by VCSA will chain its root of trust to the imported VMCA root CA certificate.

In Firefox I opened up the certificate list in Advanced settings, selected “Authorities”

I then clicked on Import, selected the .0 file and was presented with this option.

Select “Trust this CA to identify websites” and click OK. Your root CA is now imported and if you open the VCSA web page you’ll find you are no longer presented with the option to verify the certificate. You may need to close and reopen the browser.

The process is similar for other browsers and is well documented for adding the root CA to Windows, Linux and Mac key stores if you prefer to do it that way.

Note: You’ll need to access the VCSA by its FQDN and not its IP address (like I normally do in a lab environment!). Otherwise you’ll get an error like this:

Note that any resource that presents a web page that has its certificate issued by VMCA will now show up as trusted.

For example, host certificates will be valid as well!

Recap

So, to summarize what we’ve learned:

1. VCSA now has its own certificate authority called VMCA
2. You can install the root certificate of VMCA in your system or browser
3. All vSphere components like vCenter, ESXi, solution users, etc can be issued certificates from VMCA if running in Default or Enterprise mode
4. VMCA can be bypassed if you don’t want to use it, however you’ll need to do more steps to manage your certificates
5. Regardless of which method, all certificates need to be installed into VECS with the exception of ESXi hosts.
6. A Certificate Manager tool is provided to help you manage your 3rd party certificate installations

I hope this was helpful. Give it a try in your lab environments and introduce your security people to these new concepts and options. I’ll be curious to hear what they say so send me an email at mfoley at vmware dot com with their feedback!

Thanks for reading,

mike

The vSphere Hardening Guide provides guidance on how to securely deploy VMware vSphere in a production environment. The vSphere Hardening Guide also serves as a foundation upon which regulatory compliance objectives are built. These organizations map compliance guidelines with vSphere Hardening Guide guidelines.

Hardening Guides are an industry recognized method of implementing stricter security to meet regulatory and local security standards above and beyond frameworks like Common Criteria.

Version 6.0 of the vSphere Hardening Guide is the next step in the evolution of the guide. A goal of the vSphere 6.0 Hardening Guide is to make the guide easier to implement and assess.

The intent of this article is to go over some of the major changes that come with the new 6.0 guide prior to its release. Consider this your “heads up”.

Separation of programmatically configured and testable “settings” from operational guidance.

Within previous guides are a mixture of

• Operational guidance – How you use the product
• Programmatic Guidance – Set this value to “True”

Operational guidelines present unique challenges.

1. They can be addressed or mitigated in many ways
2. They are generally left open to interpretation
3. In order to implement them they may require cross-functional support across an enterprise
4. They are usually not something that can be automatically tested and verified. Instead, they rely on an individual signing off that they are done correctly

Consider the example of “Run your vCenter and ESXi management interfaces on a separate management network” as something that meets each of these bullet points.

For the vSphere 6.0 Hardening Guide, the “operational” or “best practices” guidelines have been moved to vSphere Security documentation. This, in no way, lessens their importance! However, each one of these will generally be specific to your environment and as such require conversations with your security professionals and your auditors on implementation details and attestation.

What are left are the things that can be checked via API’s, CLI’s and other tools for attestable values. In some cases, the default values are sufficient and the check is an audit to ensure that the value is correct. In others, it’s a new or different value.

Risk Profiles

Guidelines are grouped in “Risk Profiles”. Risk Profiles indicate the relative increase in security provided by the guidelines. Risk Profiles are grouped as follows.

Risk Profile 1: guidelines that only be implemented in the highest security production environments, e.g. top-secret government or military, extremely sensitive data, etc.

Risk Profile 2: guidelines that should be implemented for more sensitive production environments, e.g. those handling more sensitive data, those subject to stricter compliance rules, etc.

Risk Profile 3: guidelines that should be implemented in all production environments

Risk Profiles in Operational Guidance

In order to create a bridge from the old guide to the new guide + documentation, a separate spreadsheet containing the Operational Guidance, a link to the new location in the vSphere documentation for each guideline and the Risk Profiles values of each of those guidelines will be published.

But why do I need a Hardening Guide? Isn’t it secure by default?

Some customers ask “Then why not ship vSphere at Risk Profile 3 by default?!”. That’s a great question. vSphere, without any changes from the Hardening Guide, already meets Common Criteria standards of security.

By default, security is built into the product. Consider the isolation of virtual machine memory and instructions or the disabling of risky parameters or strong password policies. See the ESXI Security Whitepaper or my session INF2336 from VMworld 2014 for a deeper dive on security. The Hardening Guide builds upon that.

Like many other products in the industry, vSphere “out of the box” is shipped optimized for Proof of Concept and pilot deployments. It is when vSphere is put into a production environment that Hardening Guidelines should be reviewed and applied as necessary.

What other changes have been made to the guide?

Top to Bottom Review

vSphere 6.0 is the first major release in a few years. As such, it was a good time to do a complete review of the Hardening Guide. Guidelines were updated, added or removed based on the incorporation of newer technologies and removal of older technologies. This was done with the great support of the vSphere R&D team and a number of product managers.

New Taxonomy

The Hardening Guide is consumed not only by individuals but also by other software such as scripts, VMware Configuration Manager, vRealize Air Compliance and a number of 3^rd party solutions and organizations.

The primary driver for a new taxonomy was

1. Ease of production
2. Ease of use by solutions
3. Ease of automation

Production of the Hardening Guide has changed from editing the Excel spreadsheet by hand to a database-driven solution that provides numerous advantages such as versioning, collaboration and other benefits.

From a programmatic standpoint I believe the new taxonomy will make the consumption of the Hardening Guide easier by solutions and automation tools.

The previous generation of the guides grouped guidelines by vSphere components. In the Excel spreadsheet this meant separate tabs such as “ESXi”, “vCenter”, “vNetwork”.

Moving forward this changes to a flat taxonomy. Rather than grouping into separate tabs the vSphere 6.0 Hardening Guide will be a single sheet.

Guidelines will now be referenced as “Component.Guideline” For example, what used to be

enable-ad-auth

becomes

ESXi.enable-ad-auth

Unique Names

One of the problems this solves is giving each guideline a unique name. Within the vSphere Hardening Guide there were some guidelines are “the same” (e.g. “configure-NTP”). With this taxonomy, these values become unique. For example:

Before:

After:

As mentioned, the vSphere 6.0 Hardening Guide will now be delivered as a single sheet Excel workbook rather than separate sheets. (That method will also be available for 6.0 but deprecated in future releases) Other methods of delivery such as CSV, PDF and XML are being considered. Your feedback will help define them.

Wrap up

In closing, the biggest goal of the vSphere Hardening Guide is to move towards making security easier to implement and manage. This is a first step of a process. It sets the stage for more things to come. By separating out the Operational Guidance from the Programmatic Guidance and creating a new taxonomy I hope to enable more interesting things moving forward.

A beta of the vSphere 6.0 Hardening Guide will made available in the near future.

Your feedback is valuable to me. You can reach me on Twitter at @vspheresecurity or via email directly at mfoley at vmware.com.