Husband, Dad, Geek & Senior Technical Marketing Architect for vSphere Security
Author's posts
Dec 19
The Palace of Harmonious Virtualization
Introduction
In my job, I get to think a lot about where things are going. I’m hearing day in and day out that security is a major stumbling block to fully virtualizing a datacenter and also for “cloud”.
In the case of the virtualized datacenter, what many call Private Cloud, this stumble usually happens when the security guy is brought in after the ball is already in motion and promptly puts a stop to things “until it’s secure”
A common reaction is to then stop, investigate and build a wall/secure the edge.
The issue I have with this is that a wall/edge solution is no longer good enough. An edge only solution is not going to help you deal with an insider and todays advanced persistent threats[Chuck Hollis’ blog]. It’s only a single layer of Defense in Depth [us-cert.gov]. Today’s threats go far beyond targeting the edge. In the virtual world, they take on a whole new level of importance.
In this article, I’ll endeavor to show you, based on the use of ancient and modern history, that building a wall around your virtual datacenter is only the first line of defense. Knowing what’s going on INSIDE the wall is critical to the protection of your data, your intellectual property and ultimately, your business.
Different cultures, common solutions
I’ve been traveling a lot lately. Just in the month of November I traveled to both China and Israel. In China I presented at the VMware vForum and RSA Conference events in Beijing. These three presentations were mostly on visibility into the virtual environment. While in Israel as the latest member of RSA’s Cross-Product Architecture Team we had meetings with our development teams there during which I got to have a number of great conversations around virtualization and security design.
Both countries were eye-opening experiences on a number of levels. China blew me away with its breath-taking rapid growth and the corresponding growth of my waistline from the excellent food (minus the scorpions on a stick)! Visiting China also helped set off a light bulb moment about defense in depth while walking through the Forbidden City. I mentioned this to my colleague Bob Griffin, one of the industry’s most accomplished security professionals. Out of that came his elegant blog posting [rsa.com] titled The Forbidden City and Defense in Depth. I’ll go into more detail about that in a moment.
Israel was amazing for its geo-political place in the world and shawarma! As I explained to a Jewish friend here in the US, to me, Jerusalem was a mix of bizarre, solemn and thought provoking all wrapped up in lafa bread with a dab of hummus. I loved every minute there and am very thankful to have experienced this wonderful place with my new Israeli friends.
(Have you figured out that I’m a foodie yet? I really need to get back to the gym after these trips….)
Both visits brought insight into history and how learning from history allows us to move forward into the future. You see, both countries have something to teach us about walls, perimeters and defense in depth and I’ll try to relate that to virtualization security and why you, as an IT professional need to involve your security team and why, as a Security Professional you need to jump in and get your hands dirty and not just build a wall.
China – The Great Wall and The Forbidden City
China’s Great Wall was originally designed to keep out the invasions of nomadic peoples like the Mongols and according to Wikipedia; it’s over 5500 miles long. Like any good fortification or wall, it’s designed to keep out people or things you don’t want on the inside. It has controlled openings where only what/who you want to come through are allowed. Not unlike an IT Security firewall, eh?
The Forbidden City was also designed to control access. Not access to a country, but to the Emperor and his entourage and ministers. As you can see by this photo, there are a series of walls and courtyards, all designed to limit access to only those allowed to specific places. Ultimately, ending up within an area just for the Emperor and his closest contacts. It was a fascinating walk through history and was what sparked this whole thinking about defense in depth and specifically how it relates to virtualization. I very much encourage you to read Bob Griffin’s blog post, referenced earlier. He goes into good depth on this topic.
Israel – Walls that separate
Israel also has walls. These are walls surrounding the Old City of Jerusalem with controlled access points such as the Jaffa Gate, a narrow road that takes a 90-degree turn. It’s pretty amazing to see cars navigating this ancient road! The marks on the walls from bumpers scraping says it all!
In addition to ancient walls, there are the new walls separating regions, limiting and controlling access.
Defense in Depth and the Insider Threat
China
Each are examples of Defense In Depth. I was in China for two weeks and went to the Forbidden City over the weekend before Bob arrived. It struck me as I walked through the Forbidden City a few days before Bob and I did that I was clearly walking through a series of control points through thick walls. All designed to protect the Emperor and his family and not unlike the series of walls one must go through to get at data in the enterprise. For example, a VPN, an internal website and a download on that website. Each may have various levels of authentication needed to access information.
But what the walls didn’t account for was the insider threat. Security was deployed to control the checkpoints and was outward-facing to gauge risk to threats from the outside. But what if someone or something within the walls became a threat? Would security be able to react? Or after the fact would they be able to reconstruct what went wrong and adjust going forward?
Israel
In ancient Israel there were also walls and control points into the Old City. In modern day Israel, as I point out above, there are a number of newer walls. The difference is that there is also strong internal security. Intelligence is regularly gathered and acted upon. For example, as was related to me, internal intelligence sources acting on information, regularly change operations such as troop movement and transportation resources in order to mitigate potential threats. So, not only does the country look outward to its security but also inward. And when something does unfortunately happen, there is usually a pretty good amount of forensic information reconstructed to learn from so that it doesn’t happen again.
“Those who cannot remember the past are condemned to repeat it” – G. Santayana
So you are probably asking yourself now how this relates to virtualization and security? Well, every IT and security professional needs to begin to look inward to virtualization and private cloud. It’s not good enough to just build a wall around your virtualized datacenter.
One of the presentations I gave at the RSA Conference in Beijing was about visibility in the Private Cloud. My theory (a generalization, I’ll admit!), based on lots of discussions with customers, partners, sales people and community members, is that IT is pushing the use of virtualization and private cloud and the last people IT involves is Security. As was said by a gentleman at VMworld 2011 in Las Vegas during a security session: “I’m the security guy, I put the NO! in InNOvation”! (For the record, EVERY IT guy in the room agreed with him!)
Typically, when Security is presented with something to secure that they know little about, the first reaction is to build a perimeter or wall around it. But, as I laid out in the examples above and below, that’s just not good enough anymore. Visibility, or “intelligence” into the operations of virtualization is fundamentally important. You must now start to look inward to mitigate threats. You must be looking for the things that stand out and react accordingly. And finally, you need a way to reconstruct what happened when something DOES go bad (and it will!) and work to mitigate that failure in the future.
Examples of the insider threat
Many who have worked in IT have heard stories of a sysadmin gone bad. It’s someone in IT who gets ticked off and has the privileged role to do something destructive. In this case, because everything is virtualized, he has decided to copy the Domain Controller, Web App and Database virtual machines. No longer am I worried about just credit card data going out the door, I as a CIO/CISO have to be concerned with my intellectual property and the fully running application leaving my control!
Another example is one of not understanding the implications of virtualization and security. I encounter many people who haven’t quite connected the dots around protecting and monitoring privileged access to their virtualization environment. The number of ESX and vCenter servers that are unprotected is staggering! Just ask virtualization security expert Edward Haletky [virtualizationpractice.com]! He pointed me to some penetration testing that was specifically looking for vSphere components open on the Internet and the results were scary. Compromising these critical components is the analog equivalent of breaking the lock on your physical datacenter! You DO have a lock on your datacenter door and the racks inside, right? And you DO limit access to those resources?
So as we think about history, old and modern, and how they apply to different scenarios, we must reflect and ask some questions.
· Is it good enough for you to just build a wall?
· Is it good enough to be just looking outward, waiting for that nomadic hacker to cross over the hill?
· Who’s guarding inside the palace and can you trust them?
Plan, design, implement
The importance of good design should not be discounted. As an IT manager, when you are going through the process of the redesign your datacenter you bring in your network and storage teams and plan and design a solution that works for you. It should also be the time you bring in your security guy, get him up to speed and make him/her part of the team.
By working together to build strong controls of identity and access and leverage new capabilities of the virtual infrastructure you can provide the business potentially greater visibility and combine them with tools to manage the risk. Leveraging this kind of intelligence can protect your Palace of Harmonious Virtualization!
Bob said it best when he wrote, “Defense in depth is a valid idea, but the analogy needs to be with adaptive and responsive systems like those of human physiology.” After all, if history is any indication, the biggest threats will be when someone gets inside your infrastructure.
mike
—————————————————
I don’t call myself “a security guy”. It’s not my background. I’m an IT/Infrastructure guy who’s been involved with security at various times of his career. I’ve taken on as a mission during my time at RSA to bring IT and Security together because only together can we all fight these new and emerging threats.
Oct 11
The missing note from the vShield 5 docs
Every IT guy (or gal) has, at some point in their career called a friend for a lifeline when they’ve gotten stuck. And invariably, every one of us has had that friend look at our problem and say something like “Did you plug it in?”
I had one of those moments today.
You see, I’m rebuilding my lab with vSphere 5, vCloud 1.5 and vShield. I decided to go directly by the documentation (for a change!) as a learning exercise. I ran into a problem with vShield. It drove me batty. I couldn’t get VM’s to either talk to each other or to outside resources like DNS, gateway or DHCP.
Now, this was seriously getting on my nerves, I reviewed everything, I read docs, I read blog articles. I just couldn’t for the life of me find out what was wrong! What did I miss?
It was time to call in my lifeline, Rob Randell from VMware. Rob lives and breathes this product and I’ve worked closely with him on all sorts of security/VMware related stuff. If anyone could figure it out, it would be Rob.
We connected this afternoon over Webex. We stepped thru a few things, looking at settings and such. Then Rob asked me to bring up the vCenter client and asked me “Why are the vShield App VM’s not powered on?”
<facepalm><Homer D’oh!> Yea, I just go bitten by the bug we all run into. The inability to see the obvious. <insert excuse here> My schedule lately is so crazy that I’ve been doing this in fits and starts and not practicing my usually good troubleshooting skills. <\excuse>
After powering on the VM’s, network traffic started flowing and all was right with the world! I talked to Rob and said that there really should be a note in the documentation. Not a note saying “Did you power up the VM’s?” but to set the auto start settings on the ESXi hosts.
As best practices for vShield, I installed, and you should to, the vShield App and Edge VM’s to local storage on the ESXi hosts. But what I failed to do was set the VM’s to auto start on the host and after a reboot, I forgot to power on the VM.
So, click on the host in the vCenter client, click on Configuration and Start/Stop Settings. Ensure the VM is in the auto-start list. I also set the shutdown action to “shutdown” and not “power off”. I also set the power-on time from 120 seconds to 15 seconds to ensure my networking wasn’t out for some period of time after host power-on.
I’ll be sending a pointer to this blog to the vShield product management team in hopes that this one simple documentation note will help you not encounter the techie embarrassment of being asked “Did you plug it in?”
Thanks for reading.
mike
Sep 03
A dinner with infamy
I’m writing this from 39119 feet above the corn fields of Wisconsin & Illinois, having just woken from a long nap brought on by a week in Las Vegas attending VMworld. Like every other attendee and exhibitor, I’m exhausted yet have a smile on my face. This was my third VMworld and like the other two I attended, it was a series of long days, lots of walking, meeting friends old and new and most of all, having fun and learning stuff.
I’ve got a couple of thoughts brewing on more technical blog posts that I’ll get to later. However, this post needs to stand alone and it’s a doozy!
The Wait
This post is about waiting 24+ years for an answer to a question that has weighed on myself and two former co-workers. It’s also about forgiveness for a brash youth who’s paid his debt to society and one of the most interesting, fun and enlightening nights of my life.
The Set Up
I was in the Pallazo, enjoying a drink with Brad Maltz (@bmaltz) and Keith Norbie (@keithnorbie) and Keith’s wife Kim. The iPhones were out and we were all tweeting while Keith’s wife watched us, being quite polite and not rolling her eyes.
I was trying to organize folks to meet up for Sushi and then head over to the Ellis Island Casino for vBeers and Karaoke when I saw a tweet from Christofer Hoff (@beaker) about having dinner with a celebrity. Here’s my response
Mind you, it’ wasn’t so much the celebrity bit that piqued my interest, it was steak and vino with a guy for whom food is something to be worshiped. I knew Hoff wasn’t going to the buffet.
He kindly DM’d me “You may attend if you’d like. 8pm” followed by an ask to dress decently. Well, of course, that was making it even more interesting so I said my goodbyes and went back to the hotel and pulled out my last clean, pressed dress shirt and pants out of the closet and proceeded to head off to Caesars Palace.
I arrived to find Hoff and Karl Renneker of VKernel at the bar but no mystery guest. We had a drink and I ask Hoff who’s the celebrity. Now, if you know him, you know that devilish smile he gets when he’s up to something (which is just about all the time, but I digress). He said “It’s a surprise…” and smirked.
Oooookay.. I think the anticipation of my reaction was killing him so a few minutes later, after discussing 3D televisions, back came the smile and he leaned over and said “Want to know the mystery guest?” “YES!” (the anticipation was killing me as well)
“It’s Mitnick”he responded.
<THUD>
I’m sure the look on my face was priceless. Hoff sure seemed to enjoy the reaction.
“Wow!” I said. “Really? Wow! I’ve got a question to ask him that’s been bugging me for 20 something years!”
The Back Story
You see, back in the mid/late 80’s, I was working at Digital Equipment Corp. Only nobody called it that. It was known internally and externally as DEC. You can read all about my time there here. I was working at DEC’s Littleton King Street facility, also known internally as LKG and home to DEC’s Networks and Communications group. I was the systems manager on the largest VAXcluster there, the DELNI:: cluster, supporting over 600 users.
DEC had been experiencing lots of hacking around that time and the buzz internally was that the hacker was getting into everything. We were finding crumbs on our systems so my co-workers and I decided to lay out a trap.
My two buddies in this endeavor are still my good friends to this day. Dean McGorrill was a master coder/hacker who could whip up almost anything in VAX Macro in a frighteningly short time.
Dave Cantor, who I still have dinner with every month, is also known as DaveC, DC, DuraCell and DCL Dave for his unbelievable mastery of the VMS DCL command language. Dave was also know for saying “Sigh!” a lot and because of that Dave, Dean and I became “Sigh Old, Sigh Mid and Sigh Young”.
Me, I was the young buck with a mastery for trying the latest things available, living on the razors edge of tech by putting beta code into production without anyone knowing. I pushed all the boundaries of VAXcluster technology and went on to work in the VMS Development Group where I built one of the most complex VAXclusters on the planet.
So, over one of our many post-work nights at Barnaby’s in Ayer, munching on buffalo wings and beer, we crafted a plan to catch the perp in the act and watch what he was doing. Deano wrote up some Macro to attach to the process of a username we thought was created by our hacker. We could watch everything he typed. (Remember, this was still the days of green screens and modems!) DaveC wrote some DCL to launch the watch.exe code when the hacker logged in and honestly, I can’t remember what part I contributed lo these many years later. hahaha!
So, we now thought we were smart and when the alarm went off, we scrambled into Deano’s cube and hunkered down to watch. We made one mistake though. We left the name of the Macro code that Dean had compiled as “WATCH.EXE”. We immediately realized our mistake but it was too late, we were committed at this point.
We watched as the user did a “SHOW PROCESS”. This is the DCL equivalent to “ps” on Unix. He listed all the processes and noticed Dean’s account logged in running WATCH.EXE. He did a “SHOW PROCESS/FUL MCGORRILL”. We about shat ourselves. We were made. The next thing that happened was epic.
DCL Overview
Before I go there, a quick review of the DCL Command Language. In DCL the system prompt is a “$” sign. A typical command would look like this.
$ SHOW PROCESS
or
In DCL you can write scripts that can allow you to string together lots of commands. Unfortunately what DCL lacked at the time was a pipe. If you look thru DCL and you’re familiar with PowerShell, you’ll catch some similarities and that’s intentional as my friend Jeffery Snover, the architect of PowerShell and now lead architect of Windows Server at Microsoft, was an ex-DECcie and loved the Verb/Noun syntax.
So, as you write DCL scripts and want to add a comment, you use the “!”. Here’s an example:
!This is a comment
$ SHOW PROCESS
$Exit
Ok, back to our story. I know this is written in Inception style but I’ll wrap it all up to the dinner shortly.
Our hacker has noticed that Dean is running WATCH.EXE and proceeds to type on his terminal
$ !Hi there.
<Crap, we were so totally pwn’d, years before pwn’d became a term!>
$ !What’s the matter, cat got your tongue?
He then went on his merry way looking around the OS. Dean and Dave looked at me and I bolted for the datacenter. I ran at top speed from LKG2 to LKG1, opened the door to the raised floor lab that held some systems, the building networking gear and modem banks and slammed the Big Red Button that took down the lab. (I always wanted to do that!)
Back to dinner
Back at the bar, Hoff is still smirking and I quickly tell my tale, not nearly as detailed as above. We settle into dinner and shortly after Kevin walks in. He shows up wearing a classic geek t-shirt with a broken Rubik’s Cube and a screw driver and the word “Cheater” underneath the cube.
We made our introductions, ordered our dinner and enjoyed our wine. Kevin asked me where I work and Hoff snorted. I deflected that for now and told him I used to work at DEC a long time ago. “Really? Did you know….” and he mentioned a number of names, most I recognized and one of them a friend of mine, Derrell Piper. I told him I was a VMS system manager back in the day and Kevin lit up and said that was his favorite operating system.
I told Kevin that I met him once there in Vegas in 1991 at a DECUS convention. “That’s the one where they revoked my pass!” he said. I didn’t tell him that we were strictly forbidden to talk to him back then. :)
I proceeded to discuss some of my history at DEC, how I worked on systems at LKG and eventually moved up to VMS Engineering. I told him I worked for Andy Goldstein and he said that Andy visited him in jail and brought him comic books. Kevin was very grateful and I told him I’d say hi to Andy for him. I mention a few more names and one of those was Dave Cantor. Kevin said “I remember him!”
The Question
That’s when, after another glass of vino, I said “I’ve got this question to ask you….” and proceeded to describe the “Cat got your tongue” scenario. As I went on, Karl and Hoff were silent and Kevin was politely smiling.
I got to the “What’s the matter? Cat got your tongue?” part and Kevin said “That was ME!!” Hoff roared and we all started laughing our asses off.
I waited for that confirmation for almost 25 years and to get it directly from the source was priceless.
I quickly emailed DaveC and told him I had confirmation. He responded within moments with “Yup. I don’t think there was ever any doubt, but I’m glad you got that confirmation.”
Apologies and forgiveness
Throughout the conversation Kevin apologized for any pain he might have caused. I’m a pretty forgiving person. It was almost 25 years ago and he has since paid his dues. It’s all good and I hold no grudge. Life is too short. There are plenty of people who have done things that landed them in jail and they have not turned their lives around and tried to give back to society like he has. So, he’s makes a living but it’s now an honest living. Good on ya mate.
On our way out of Caesars Kevin gave me a copy of his book. I know I should have asked him to sign it, but I don’t know why I didn’t. In hindsight I’m kind of kicking myself I didn’t have him write “What’s the matter, cat got your tongue?” Maybe another time?
As a wrap up to all this, the next morning Hoff Tweets about the night before.
It really was one of the best nights a geek like myself could ask for. I can’t thank Hoff enough for allowing me to join them for dinner.
I hope you enjoyed this piece of Geek History. I find I’ve been in the middle of a lot of it and that’s one of the coolest things a geek like myself can hope for.
mike