Category: RSA

Oct 11

The missing note from the vShield 5 docs

Every IT guy (or gal) has, at some point in their career called a friend for a lifeline when they’ve gotten stuck. And invariably, every one of us has had that friend look at our problem and say something like “Did you plug it in?”

I had one of those moments today.

You see, I’m rebuilding my lab with vSphere 5, vCloud 1.5 and vShield. I decided to go directly by the documentation (for a change!) as a learning exercise. I ran into a problem with vShield. It drove me batty. I couldn’t get VM’s to either talk to each other or to outside resources like DNS, gateway or DHCP.

Now, this was seriously getting on my nerves, I reviewed everything, I read docs, I read blog articles. I just couldn’t for the life of me find out what was wrong! What did I miss?

It was time to call in my lifeline, Rob Randell from VMware. Rob lives and breathes this product and I’ve worked closely with him on all sorts of security/VMware related stuff. If anyone could figure it out, it would be Rob.

We connected this afternoon over Webex. We stepped thru a few things, looking at settings and such. Then Rob asked me to bring up the vCenter client and asked me “Why are the vShield App VM’s not powered on?”

<facepalm><Homer D’oh!> Yea, I just go bitten by the bug we all run into. The inability to see the obvious. <insert excuse here> My schedule lately is so crazy that I’ve been doing this in fits and starts and not practicing my usually good troubleshooting skills. <\excuse>

After powering on the VM’s, network traffic started flowing and all was right with the world! I talked to Rob and said that there really should be a note in the documentation. Not a note saying “Did you power up the VM’s?” but to set the auto start settings on the ESXi hosts.

As best practices for vShield, I installed, and you should to, the vShield App and Edge VM’s to local storage on the ESXi hosts. But what I failed to do was set the VM’s to auto start on the host and after a reboot, I forgot to power on the VM.

So, click on the host in the vCenter client, click on Configuration and Start/Stop Settings. Ensure the VM is in the auto-start list. I also set the shutdown action to “shutdown” and not “power off”. I also set the power-on time from 120 seconds to 15 seconds to ensure my networking wasn’t out for some period of time after host power-on.

I’ll be sending a pointer to this blog to the vShield product management team in hopes that this one simple documentation note will help you not encounter the techie embarrassment of being asked “Did you plug it in?”

Thanks for reading.

mike

Sep 03

A dinner with infamy


I’m writing this from 39119 feet above the corn fields of Wisconsin & Illinois, having just woken from a long nap brought on by a week in Las Vegas attending VMworld. Like every other attendee and exhibitor, I’m exhausted yet have a smile on my face. This was my third VMworld and like the other two I attended, it was a series of long days, lots of walking, meeting friends old and new and most of all, having fun and learning stuff.

I’ve got a couple of thoughts brewing on more technical blog posts that I’ll get to later. However, this post needs to stand alone and it’s a doozy!

The Wait

This post is about waiting 24+ years for an answer to a question that has weighed on myself and two former co-workers. It’s also about forgiveness for a brash youth who’s paid his debt to society and one of the most interesting, fun and enlightening nights of my life.

The Set Up

I was in the Pallazo, enjoying a drink with Brad Maltz (@bmaltz) and Keith Norbie (@keithnorbie) and Keith’s wife Kim. The iPhones were out and we were all tweeting while Keith’s wife watched us, being quite polite and not rolling her eyes.

I was trying to organize folks to meet up for Sushi and then head over to the Ellis Island Casino for vBeers and Karaoke when I saw a tweet from Christofer Hoff (@beaker) about having dinner with a celebrity. Here’s my response

image

Mind you, it’ wasn’t so much the celebrity bit that piqued my interest, it was steak and vino with a guy for whom food is something to be worshiped. I knew Hoff wasn’t going to the buffet.

He kindly DM’d me “You may attend if you’d like. 8pm” followed by an ask to dress decently. Well, of course, that was making it even more interesting so I said my goodbyes and went back to the hotel and pulled out my last clean, pressed dress shirt and pants out of the closet and proceeded to head off to Caesars Palace.

I arrived to find Hoff and Karl Renneker of VKernel at the bar but no mystery guest. We had a drink and I ask Hoff who’s the celebrity. Now, if you know him, you know that devilish smile he gets when he’s up to something (which is just about all the time, but I digress). He said “It’s a surprise…” and smirked.

Oooookay..  I think the anticipation of my reaction was killing him so a few minutes later, after discussing 3D televisions, back came the smile and he leaned over and said “Want to know the mystery guest?” “YES!” (the anticipation was killing me as well)

“It’s Mitnick”he responded.

<THUD>

I’m sure the look on my face was priceless. Hoff sure seemed to enjoy the reaction.

“Wow!” I said. “Really? Wow! I’ve got a question to ask him that’s been bugging me for 20 something years!”

The Back Story

You see, back in the mid/late 80’s, I was working at Digital Equipment Corp. Only nobody called it that. It was known internally and externally as DEC. You can read all about my time there here. I was working at DEC’s Littleton King Street facility, also known internally as LKG and home to DEC’s Networks and Communications group. I was the systems manager on the largest VAXcluster there, the DELNI:: cluster, supporting over 600 users.

DEC had been experiencing lots of hacking around that time and the buzz internally was that the hacker was getting into everything. We were finding crumbs on our systems so my co-workers and I decided to lay out a trap.

My two buddies in this endeavor are still my good friends to this day. Dean McGorrill was a master coder/hacker who could whip up almost anything in VAX Macro in a frighteningly short time.

Dave Cantor, who I still have dinner with every month, is also known as DaveC, DC, DuraCell and DCL Dave for his unbelievable mastery of the VMS DCL command language. Dave was also know for saying “Sigh!” a lot and because of that Dave, Dean and I became “Sigh Old, Sigh Mid and Sigh Young”.

Me, I was the young buck with a mastery for trying the latest things available, living on the razors edge of tech by putting beta code into production without anyone knowing. I pushed all the boundaries of VAXcluster technology and went on to work in the VMS Development Group where I built one of the most complex VAXclusters on the planet.

So, over one of our many post-work nights at Barnaby’s in Ayer, munching on buffalo wings and beer, we crafted a plan to catch the perp in the act and watch what he was doing. Deano wrote up some Macro to attach to the process of a username we thought was created by our hacker. We could watch everything he typed. (Remember, this was still the days of green screens and modems!) DaveC wrote some DCL to launch the watch.exe code when the hacker logged in and honestly, I can’t remember what part I contributed lo these many years later. hahaha!

So, we now thought we were smart and when the alarm went off, we scrambled into Deano’s cube and hunkered down to watch. We made one mistake though. We left the name of the Macro code that Dean had compiled as “WATCH.EXE”. We immediately realized our mistake but it was too late, we were committed at this point.

We watched as the user did a “SHOW PROCESS”. This is the DCL equivalent to “ps” on Unix. He listed all the processes and noticed Dean’s account logged in running WATCH.EXE. He did a “SHOW PROCESS/FUL MCGORRILL”. We about shat ourselves. We were made. The next thing that happened was epic.

DCL Overview

Before I go there, a quick review of the DCL Command Language. In DCL the system prompt is a “$” sign. A typical command would look like this.

$ SHOW PROCESS

or

$ MAIL

In DCL you can write scripts that can allow you to string together lots of commands. Unfortunately what DCL lacked at the time was a pipe. If you look thru DCL and you’re familiar with PowerShell, you’ll catch some similarities and that’s intentional as my friend Jeffery Snover, the architect of PowerShell and now lead architect of Windows Server at Microsoft, was an ex-DECcie and loved the Verb/Noun syntax.

So, as you write DCL scripts and want to add a comment, you use the “!”. Here’s an example:

!This is a comment

$ SHOW PROCESS

$Exit

Ok, back to our story. I know this is written in Inception style but I’ll wrap it all up to the dinner shortly.

Our hacker has noticed that Dean is running WATCH.EXE and proceeds to type on his terminal

$ !Hi there.

<Crap, we were so totally pwn’d, years before pwn’d became a term!>

$ !What’s the matter, cat got your tongue?

He then went on his merry way looking around the OS. Dean and Dave looked at me and I bolted for the datacenter. I ran at top speed from LKG2 to LKG1, opened the door to the raised floor lab that held some systems, the building networking gear and modem banks and slammed the Big Red Button that took down the lab. (I always wanted to do that!)

Back to dinner

Back at the bar, Hoff is still smirking and I quickly tell my tale, not nearly as detailed as above. We settle into dinner and shortly after Kevin walks in. He shows up wearing a classic geek t-shirt with a broken Rubik’s Cube and a screw driver and the word “Cheater” underneath the cube.

image

We made our introductions, ordered our dinner and enjoyed our wine. Kevin asked me where I work and Hoff snorted. I deflected that for now and told him I used to work at DEC a long time ago. “Really? Did you know….” and he mentioned a number of names, most I recognized and one of them a friend of mine, Derrell Piper. I told him I was a VMS system manager back in the day and Kevin lit up and said that was his favorite operating system.

I told Kevin that I met him once there in Vegas in 1991 at a DECUS convention. “That’s the one where they revoked my pass!” he said. I didn’t tell him that we were strictly forbidden to talk to him back then. :)

I proceeded to discuss some of my history at DEC, how I worked on systems at LKG and eventually moved up to VMS Engineering. I told him I worked for Andy Goldstein and he said that Andy visited him in jail and brought him comic books. Kevin was very grateful and I told him I’d say hi to Andy for him. I mention a few more names and one of those was Dave Cantor. Kevin said “I remember him!”

The Question

That’s when, after another glass of vino, I said “I’ve got this question to ask you….” and proceeded to describe the “Cat got your tongue” scenario. As I went on, Karl and Hoff were silent and Kevin was politely smiling.

I got to the “What’s the matter? Cat got your tongue?” part and Kevin said “That was ME!!” Hoff roared and we all started laughing our asses off.

I waited for that confirmation for almost 25 years and to get it directly from the source was priceless.

I quickly emailed DaveC and told him I had confirmation. He responded within moments with “Yup. I don’t think there was ever any doubt, but I’m glad you got that confirmation.”

Apologies and forgiveness

Throughout the conversation Kevin apologized for any pain he might have caused. I’m a pretty forgiving person. It was almost 25 years ago and he has since paid his dues. It’s all good and I hold no grudge. Life is too short. There are plenty of people who have done things that landed them in jail and they have not turned their lives around and tried to give back to society like he has. So, he’s makes a living but it’s now an honest living. Good on ya mate.

On our way out of Caesars Kevin gave me a copy of his book. I know I should have asked him to sign it, but I don’t know why I didn’t. In hindsight I’m kind of kicking myself I didn’t have him write “What’s the matter, cat got your tongue?” Maybe another time?

As a wrap up to all this, the next morning Hoff Tweets about the night before.

image

It really was one of the best nights a geek like myself could ask for. I can’t thank Hoff enough for allowing me to join them for dinner.

I hope you enjoyed this piece of Geek History. I find I’ve been in the middle of a lot of it and that’s one of the coolest things a geek like myself can hope for.

mike

 

Jul 13

Dr. iSCSI or How I learned to stop worrying and love virtual distributed switches on vSphere V5

Intro

It’s July 12th 2011 around 2:30pm EST as I start to write this. Anything interesting in the world of VMware virtualization go on today? :)

Riiight, VMware vSphere Version 5 has been announced! Along with many other products from VMware. Definitely go to  VMware.com and EMC.com to see all the “awesomesauce” as one crazy Canadian puts it. <\shameless_plug>

As with everyone else’s blog, these words are my own and reflect my state of mind and not that of my employers. (RSA, the Security Division of EMC for those not keeping up)

iSCSI and vDS

I’m going to talk about virtual distributed switches (vDS) and how I set up my lab to get rid of traditional vSwitches. In my current lab of two ESXi hosts and some Iomega storage, I strive to make it look as “customer like” as possible. I don’t do the typical vCenter install that you click,click,click thru letting it install SQL Express and walk away. Nope, I do stuff like installing a full-blown SQL server and setting up vDS!

I wanted to separate out my VMkernel from my iSCSI. I wanted to put them on separate switches. vDS seemed logical for my setup as I could more easily play around with jumbo frames and all sorts of other cool stuff that’s come out with V5 like Netflow and port mirroring. Mind you, I’m probably not going to be dealing with that stuff on my iSCSI network but the option will be there..

Besides, vSwitches, as useful as they are, are kinda passé and I wanted to learn how to configure a vDS. So, I went searching around on the Interwebs and found a great article from Mike Graham at his blog “Mike’s Sysadmin Blog”. It showed how to set up iSCSI traffic to go over vDS. But it had a bunch of workarounds that required you to go into the command line on the ESX/ESXi server to “fix” things to work.

An additional article I highly recommend reading is from my friend Scott Lowe on Jumbo Frames and iSCSI+vDS. Scott’s article is based on vSphere V4 and like Mike Graham’s article on iSCSI and vDS, it lays out the required tweaking at the command line to make things work. In both cases they go into the requirements around modifying the VMkernel to start talking Jumbo Frames. According to my review of the vSphere V5 RC docs, I believe that’s no longer needed.

Well, in vSphere V5, you can do it all via the GUI! (and that means, of course, you can probably build a PowerCLI script that’ll do it!)

Step By Step

Start by creating a new vDS. We’ll call it “iSCSI dvSwitch”

Select the 5.0.0 version

ScreenShot132

Here I’ll set the number of dvUplink ports = 2. This means I’ll use 2 physical adapters PER HOST.

ScreenShot055

Now I’ll select what physical adapters I’ll use. In the example below, I’m using vmnic4 and vmnic5 on each of my Dell R610’s

ScreenShot099

So, now you’ll see the dvUplink’s ready to roll.

ScreenShot103

Just to clarify

dvUplink1 –

Host1/vmnic4

Host2/vmnic4

dvUplink2 –

Host1/vmnic5

Host2/vmnic5

ScreenShot103

The picture above will create default port groups. We’re going to create two of them. You’ll need to adjust the Teaming and Failover as follows.

From the docs:

If you are using iSCSI Multipathing, your VMkernel interface must be configured to have one active
adapter and no standby adapters. See the vSphere Storage documentation.

Setting things up as outlined below will ensure your iSCSI adapter is set up using a compliant portgroup policy.

Portgroup1 –

Active Uplink = dvUplink1

Unused Uplink = dvUplink2

Portgroup2 –

Active Uplink = dvUplink2

Unused Uplink = dvUplink1

ScreenShot105

When that’s done, the switch should look like this:

ScreenShot133

Now, in order to talk iSCSI to the iSCSI host, we need to bind a VMkernel to each port group. On each host, go to Configuration…Networking…vSphere Distributed Switch and down to the iSCSI-dvSwitch. Open “Manage Virtual Adapters…”

ScreenShot137

Create a new Virtual Adapter with a VMkernel.(vmk1)

ScreenShot140

Assign IP address

ScreenShot110

Mapped to Portgroup1

ScreenShot108

Repeat all these steps for your other hosts.

Configuring iSCSI

If this was vSphere V4, you’d have to be doing some command line stuff to get the rest of this to work. Not in vSphere V5!

First step, we need an iSCSI Storage Adapter. For your first host, open Configuration…Storage Adapters and click on Add…

image

Now click the OK button to add the iSCSI Adapter. You see the following dialog box. Click OK

image

Ok, you now have your iSCSI adapter and it’s time to configure it. Click on Properties…

image

The dialog box opens and click on the Network Configuration tab.

image

Click the Add… button

You should now see something like the following:

image

Click on OK and you get this screen. Here’s where you’ll see the compliance portgroup policy check I pointed out earlier!

image

Now it’s time to set up some iSCSI connections. Click on the Dynamic Discovery tab and then on Add… I’ll add two connections to my two Iomega IX4-200R devices that I’ve preconfigured iSCSI disks on.

image

After I’ve added my connections and clicked on Close, I’ll be prompted to rescan for devices. Click on Yes.

image

In my case, you’ll see the two iSCSI disks show up in the Details for the iSCSI Storage Adapter

image

Now let’s go over to Storage and see the new datastores. You MAY have to click on Refresh to get a clean view.

image

Jumbo Frames

Lots has been talked about with regard to Jumbo Frames. Scott Lowe’s article touches on it for V4. Jason Boche has an awesome article on whether they actually help or not from a performance standpoint. I’ll leave that up to you as to whether it works or not.

As a bit of a recap, there’s two places in vSphere that Jumbo Frames need to be enabled.

  1. The VMkernel. Specifically, the NIC attached to the VMkernel being used for iSCSI traffic
  2. The vDS switch itself

Note that for Jumbo Frames to really work, you need to have everything from the vSphere level all the way thru the switches and the disk array supporting Jumbo Frames, otherwise it’ll either not work or performance will suffer.

VMkernel MTU Settings

Thankfully, in V5, this is all now settable via the GUI. Let’s start with the VMkernel. Go to Inventory…Hosts and Clusters…

Click on the host and then the Configuration Tab. Select Networking… and then Virtual Distributed Switch.

image

Now, for the iSCSI vDS, click on Manage Virtual Adapters… You’ll see the VMkernel, click on that and then Edit.

Under the General tab, you’ll see the NIC Settings and the MTU value. Set that to 9000.

Previously, this was a funky set of command line steps, all called out in Scott’s article. MUCH simpler now.

image

UPDATE!

Want to change this from PowerCLI? Well, you CAN!

[sourcecode language=”powershell” padlinenumbers=”true” wrap=”true”]
get-vmhost $host | get-vmhostnetworkadapter -vmkernel -name vmk1 |set-vmhostnetworkadapter -mtu 9000
[/sourcecode]

Virtual Distributed Switch MTU Settings

Now, let’s take a look at the vDS. Open the Inventory…Networking page. Select your vDS and click Edit Settings…

image

You’ll see the MTU value (Default 1500). Set that to 9000.

image

Wrap-up

Well, that’s it! You should now have iSCSI traffic moving across your vDS! Note: I haven’t tested jumbo frames and if they are truly working as advertised yet. I HAVE testing iSCSI over vDS and it works just fine. Just too much going on at the moment. If you can, please post some feedback.

So, with V5 of vSphere, VMware has continued to raise the bar in easy setup and configuration. They’ve now dropped the requirement to step into the command line of ESX(i) and run obtuse commands to get what seems like simple tasks to run. You no longer have to bind and set MTU values at the command line.

Please email me if I’m incorrect and I’ll be glad to fix this posting.

Thanks for reading!

mike