Virtualization for Service Providers

Cisco Live, London–Day 1: Calvinball

Have you ever read the Calvin and Hobbes comic strips? I grew up with them, and am a huge fan. One of my favorite recurring things in the series was the concept of “Calvinball”, which was their way of rebelling against the rules they were surrounded by every day. The only rules were that there were no rules, and you couldn’t play it the same way twice.

Well, I’ve come to realize that day one of a major marketing event is much like a real-life game of Calvinball: it never comes off as you expected it to, you never have the same challenges twice and the best way to get through it is to laugh, go with the flow and enjoy it!

VCE is one of the platinum sponsors of Cisco Live in London (and Melbourne, and in the US) and our booth is fantastic. The theater area is expanded, and we regularly had 30-50 people crowded around to hear about topics presented by VCE partners, employees and customers. For the first time we also have two hands-on-lab stations where customers can get a self-paced walkthrough of the new version of our UIM/P provisioning a discovery tool. In addition we have all the usual stations, including one with partner information, one with VDI solutions and more. The event marketing team at VCE has done their usual bang-up job with the logistics.

Of course, day one is barely-organized chaos. With one of our main event coordinators in her hotel bed with the flu, and with the rest of the team cutting their teeth in London, it was up to those of us who have been through the gauntlet before to settle things down pre-show. Since my field enablement team (with the incomparable Jae Ellers and Tom Chatham) have done a number of these, we were able to get things moving. Of course, some things are out of our control…

First presentation of the day, by Colt, one of our great Service Provider customers no less, and the power goes out to the whole booth. The event staff gets things back in order quickly, and then we realize that the power cycle has wiped out the configuration on the Cisco-provided internet switch that is powering the hands-on-labs (copy run start is your friend, people). 20 minutes later the Cisco team gets that resolved, only to find out that there’s some issue upstream from us that’s affecting multiple exhibitors. After about an hour, everything is back in working order and we can finally start settling into a groove…

But of course there are more issues. iPads that haven’t been configured correctly, surveys that are a little tricky to submit, booth presentations that seem to have disappeared, booth personnel that seem to have disappeared, all of the day one things that could be issues, were issues. Hey, it is the first event of the year, maybe we just got everything out of the way early, right?

Once we did get things settled down, the rest of the day ended up being great! So many customers, so many people wanting to talk technology, strategy, positioning, convergence… The traffic stream through the booth was overwhelming at times, even with the team we had there to handle it, and by 2:00pm or so I was reminded why the marketing events are so important in our industry. No where else could we reach so many people in such a short amount of time.

Two final things about day one before I get back to work. First, Amy Lewis is awesome, and the entire Cisco Social Media presence is impressive. The amount of energy she puts into everything is amazing, and while there are many cool things they are doing, the Cisco Services pinball machine is my favorite. Of course I also have the high score, so I’m biased. Aaron Delp talked a big game, but not even his vaunted foot kick could push him into the top spot…

Finally, most of you have probably see the Tweet and picture by now, but we had one of the NetApp FlexPod engineers come over from the booth next door and ask to run through the UIM hands on lab, which of course I was fine by me. I’m not sure of the gentleman’s name, but over the course of the next hour we had a great conversation about the converged infrastructure market and management and orchestration stacks and he even made some suggestions for how we could improve the lab itself which I’ll have in place before the next event. Sure, it was funny to have someone from NetApp asking to go through the lab, but it was a cool move by a smart engineer, and I think everyone came away better and more educated for it. He was definitely a credit to his employer, and if there is anyone else who wants to come over and walk through UIM, please just let me know!

Day 2 is about to start (thank goodness for long cab rides…) so I’ll sign off here. If you are at the event, please swing by the booth and say hi!

VMworld 2011: Recap Day 2

And the hits just keep on coming from VMworld.

Day two kicked off with Steve Herrod’s keynote, and he introduced a couple new VMware projects.  Appblast, which allows you to start a Windows App on any HTML 5 client, ThinApp Factory, which scans your local device and lets you ThinApp the applications that can work with Horizon (and the demoed Horizon mobile) and Project Octopus which looks to be a VMware integrated “dropbox-like” service for the enterprise.  All of the demos were well done, and you can see VMware pushing ahead in their “post-PC” agenda with each one.

There were also a number of “technology previews” that were done during the day that really interested me.

1. Expanded “Project Lightning” Demo: At EMC World, we were introduced to this project, with the idea that sometimes it’s beneficial to get the VM workload as close to the compute resources as possible.  There are a couple startups out there that are doing more or less the same thing, and so this part of the concept was pretty straightforward: put a PCIe-based flash card in the servers, “enable” it in vSphere, and then vMotion the workload to the card.  Simple.  The other side of that coin, when it’s beneficial to move the workload closer to the storage was demoed yesterday, showing how you can move a workload closer to the storage in order to alleviate I/O or latency bottlenecks.
   
   I want to be clear when I say this, because I’m not sure that a lot of people understand the implications: We saw vSphere running on an Isilon node.  We saw the complete merging of the hypervisor and the storage array, and since almost every EMC storage array also runs on Intel processors, it’s not hard to start connecting the dots there.  I may be wrong, but I’m not sure that there’s any other enterprise storage vendor out there who’s tried this and shown it publically.  Think about the implications there…
2. Our first ever look at “Avamar vCloud Protector” in public.  This really makes my Service Provider brain go a little crazy, so expect to see a blog post strictly about this preview.  Finally, a completely integrated, vCD-enabled, customer facing portal that allows for data protection classes-of-service, self-service backups AND restores using the only product on the market that can do change block tracking (CBT) at the native vSphere block size for both backups and restores.  Very exciting for all of the service providers who are struggling to make a legacy backup platform work in a cloud services environment!
3. vShield5 and RSA integration being shown, demoed and talked about.  This is another place where the work between EMC and VMware is really paying dividends for the customers.  Part of the reason I’m excited about this is going to be how it plays into the new VSPP licensing model.  Since service providers are getting the full  version of vShield Edge for free as part of the program, they have a very, very short financial runway to travel in order to add the RSA DLP functionality.  This is something that Cisco needs to be paying very close attention to, since their version of the vShield Edge appliance (the virtual ASA firewall) is still a good ways out on the roadmap.  If VMware can seed the SPs with vShield Edge and then start rolling both compliance and multi-site functionality into it, Cisco may lose the war before they ever join the battle.  I don’t care how much more functionality you have, service providers typically won’t say no to free provided the product meets the needs of the customer. 

The night ended with the EMC Customer Appreciation Party at the Surrender nightclub over at the Wynn hotel.  Holy cow, what an event.  My hat is off to Chad Sakac and his team for putting together an incredible way to say “thank you” to customers and partners.

Two days down.  Two more to go.  Red Bull to the rescue!

VMworld 2011: Recap Day 1

The first day of VMware is always a tone-setter for me.  Sunday is a way to ease in, get your bearings, try and sync up your body clock and get ready, but Monday really determines what kind of conference it’s going to be.  If today is any indication, we have a great week in front of us.

Unfortunately, I don’t have a full conference pass, and I don’t have access to any of the individual sessions.  This is actually a huge, huge issue for me, and one that I’m working with my employer on to make sure gets fixed for Copenhagen and future conferences.  Without the sessions I feel like I’m not getting close to the technical content I could otherwise.  If only there were a Scott Lowe in every session live-blogging… :-)

The morning was filled with customer meetings, blog posts (pssst, I published a big one on the new VSPP program features and what it means for service providers here) and conference calls.  After a great lunch with Jason Nash and Dan Weiss from Varrow I headed down to the VCE booth to get my vCloud on Vblock theater presentation loaded and make sure there weren’t any issues.  The VCE booth looks great, as usual, and the Vblock 300FX was purring along nicely.  We’ve added some additional demo stations and a station dedicated just to partners, so come by and check it out. 

Speaking of the booth, it’s awesome to see the interest from people in the Vblock.  Giving away iPads in the booth definitely helps bring people in, but even outside that, the number of customers I talked to about the Vblock and its ecosystem was awesome.  Customers big and small, existing EMC or NetApp customers, customers looking for infrastructure or just for an app platform, the one constant is that everyone wants to take the complexity out of it.  The FastPath VDI product announcement (which went out a little earlier than planned, oops) was definitely a hot topic!  Come to the booth today and you’ll be able to see the demos and pick up some collateral.

One of the things that is different being a vendor instead of a partner is that there's not a lot of mystery left in the keynote content. The general session from Paul Maritz was packed, well orchestrated and Paul always does a good job of delivering the message.  The things that stood out for me were these:

1. The service provider space is becoming not only more important tactically to VMware as a place to deliver products and services, it’s becoming strategic as well, with vCloud Director being the hub around which the VMware development wheel is turning.  Considering as little as 18 months ago that hub was the vSphere hypervisor, that’s a huge strategic change.  The VSPP license changes, the focus on delivering services through the vCD framework (watch how many demos for new products and technologies use the vCD UI this week) all highlight this change.  Needless to say, I couldn’t be happier.
2. The introduction of vFabric Data Director is huge.  Think about both the importance today (really, self-service, vSphere/vFabric optimized, multi-tenant VMware databases as a service?) as well as the strategic importance over time.  This is VMware taking a first bold step into the big-boy pool of database services and it’s not going to take long for the other people who think they OWN that pool to notice.  Also, think about the amount of strategic work that went into that one announcement.  Chad Sakac put up a great blog post here, and the answer to his question at the end is yes, we would have thought he was high.  How much coordinated acquisition went on between EMC and VMware at make “Project Aurora” happen?  GemFire. Gemstone, Greenplum, SQLFire, how many millions of dollars were bet on this one path? 

After the solutions floor closed, the fun continued.  The EMC vSpecialist party at the Moon nightclub at the Palms was great, the #CXIparty at the Cosmo was fun, the VMwareTips.com TweetUp at the Vesper was awesome.  I watched Tee Glasgow break a limo, I watched Chris Horn give movie watching recommendations to John Avery, and I worked hard to quit while I was ahead, so all in all it was a great night!

Day two promises more fun.  I’ll be down in the booth around 9am (maybe a LITTLE late.:-) so stop by and say hi!  Also, don’t forget the VCE Tweetup tonight at the Helix in the Palazzo!

VSPP for vSphere 5: An Offer that Service Providers Can't Refuse

With the announcement of vSphere 5, and more importantly it seems, with the change in the way that enterprise licenses are sold and managed, there has been a lot of talk (here and elsewhere) about how the cost of virtualizing, especially in a dense fashion got more expensive.  I won’t bore you with a rehash here, because all my readers are brilliant and can use a calculator, but there’s one thread that’s been lost in all of the noise that I want to dig into here: what about service providers?

Since the service providers use a different licensing model from the enterprise-based ELA and socket-licensing model, it was expected that there would be changes for them as well.  Speculation ranged all over the board as to what the changes would be, but after getting my briefing from VMware I’m happy to announce that it’s better than I could have imagined.  Seriously.  Sound interesting?  Put simply, VMware is making the overall value proposition so enticing to Service Providers that it’s going to be hard to ignore. The barrier to entry for creating a business model has always been first, the cost of the hardware and second, the cost of the licensing.  Both barriers are falling rapidly, and the new VSPP details show that clearly.

(Note: All of these scenarios use the “new” vSphere5 licensing rolled out on 8/3/2011 with the increased vRAM entitlements.)

As some background, I have a long and (sometimes) glorious history with the VSPP program.  I worked for a VMware vCloud partner for years, and got to see the good, bad and ugly of the evolution of that program, along with the joys and heartaches of building a business around it.  All of you reading this who work for service providers are probably nodding your head right now… With VCE, I’ve seen the ups and downs from the other side of the table as well.  The goal of the program, much like the Microsoft SPLA, was always laudable: create a way for SPs to “lease” licenses on-demand and provide those to their customers.  With each change to the program, VMware tried (I’ll give them credit) to identify more and more with their SP customers, and starting in late 2010 it seems like the feedback they were getting from customers started to sink in. 

For those unfamiliar with the VMware Service Provider program, Mathew Lodge has posted a very good overview of the program and the recent changes on the VMware vCloud Blog.  VSPP is one of the few VMware products that isn’t actually licensed directly to the customer.  VMware uses a series of “aggregators” to resell the program, the customer signs a contract with these aggregators, and the contract includes both a “point” cost and a point commitment level that the service provider promises to pay.  The “retail” price of a point is $1, and the higher the commitment (generally) the lower the point cost.  By default, each bundle has a corresponding point value.  The point values of the bundles has stayed the same with the move to vSphere 5, with the Standard bundle having a value of 5 points and the Premier sitting at 7.  Both include vSphere, vCloud Director, vCenter Chargeback, vCenter Server and Production SnS.  The Premier bundle also includes the full version of vShield Edge as well, which adds VPN and load balancing.

In the most recent version of the VSPP program before vSphere5 was announced, VSPP points were calculated based on allocated RAM (just like the new vRAM pricing on the Enterprise side) per powered-on VM, per month.  If you were a Premier partner, had 100 VMs with 4GB each powered on and a point value of $1, you would pay $2,800/mo, as an example (100*4*7=2800).  If you made a commitment of 2800 points per month, maybe you could get your point value down and that would carry through to the bottom line.

With the release of vSphere5, VMware have made some huge changes that are really, really going to help Service Providers.  First, they have changed the measurement from allocated RAM to reserved RAM.  You read that right!  If you give a VM 8GB of RAM but only reserve 6GB of it, you are only going to pay for the 6GB.  There is a requirement included in the small print that 50% of all committed RAM must be reserved (from a licensing standpoint, not from an operational one), but even there it puts the virtualization of tier 1 apps and over-commitment of RAM back on the table!  Next, VMware has capped the amount of billing at 24GB of RAM per VM versus 96GB for enterprise customers.  Finally, like with the enterprise licensing, there’s no charge for powered off VMs which is also a welcome change from previous versions.

Let’s look at an example that is close to my heart: a VCE Vblock, building off of the numbers that Aaron Delp generated for his scale-up vs. scale-out discussion.  VMware has been adamant about making sure we don’t use single server examples when discussing licensing, so we’ll use a couple pretty standard builds that customers are putting into production everyday.

First, we’ll look at the low end: 8 hosts, 96GB of RAM each, dual CPU.  First, let’s look at the setup:

In this example, we’ve allocated for N+1 protection on the environment, meaning that we’ll have 672GB of RAM to allocate to VMs once the environment is full.  Yes, I’m assuming that the customer will end up using all of the capacity their hardware provides, at least over time.  One of the nice things about the new vSphere5 licensing is that, unlike vSphere4, customers can ramp up their licensing as they go, rather than having to license everything up front, so as we continue please understand that these numbers are based on RETAIL costs assuming a FULLY POPULATED ENVIRONMENT.  In this example the average VM has 2GB of vRAM allocated, and 50% of that vRAM is reserved.  The reservation level doesn’t matter for the enterprise licensing, but it definitely does for VSPP, as you’ll see.  With that (extended) caveat in place, let’s look at the licensing cost of this environment with vSphere5:

OK.  So over three years (the “refresh period” I chose) you will pay $97,872 in total (retail) cost for the vSphere5 licensing at the Enterprise Plus level.  We took the total allocated RAM and divided by 96 (the amount of RAM entitlement at the Enterprise Plus level) to get the number of licenses needed, rounded it up, and then multiplied by the $3495 retail cost to get the license cost.  For SnS we take the retail cost ($874), multiply it by the number of licenses purchased and then multiply that by 3 years.  In straight vRAM we only needed 10 licenses thanks to the new upgraded 96GB entitlement that is included, but we had to purchase 16 licenses anyway to cover the number of sockets in the cluster.

So now that we know what the costs are on the enterprise side, let’s look at VSPP.  Again, we are going to assume the environment is full and at a steady state for the sake of an apples-to-apples comparison, and we are going to assume full retail pricing.  Please note that I fully understand both of these assumptions are unlikely to be found in the real world, but we have to put a stake in the ground somewhere, right? 

Let’s talk about how we came up with the “Premier Bundle Points Used” value.  The “cost” for the Premier Bundle is 7 points, per GB of RAM reserved, per VM, per month.  It’s OK, we’ll walk through it! :-) If you look at the setup at the top, you’ll see that we have an average of 2GB of allocated RAM in our cluster, with a 50% reservation rate.  These numbers are 100% variable from environment to environment!  The VSPP program puts a lower limit on reservation percentage at 50%, so that’s the lowest we can go there, but we can always go up, even reserving 100% of the RAM if we wanted to.  In this example, we end up with a total of 672 VMs running, or 11 VMs per core which is pretty high for an SP environment, depending on the service being offered. With 672 VMs, at 1GB reserved each, multiplied by 7 points each comes to $4,704 a month or $169,344 over three years.  That’s a 73% premium over the “new” vSphere5 licensing.  Fortunately, there’s more to look at…

Looking at just the licensing head-to-head leaves out all of the extra goodness that the VSPP program includes:

• vCenter Server for free
• vCenter Chargeback for free
• vShield Edge for free
• vCloud Director for free

All of those have a $$ that can be tied to them if the customer chooses to use them, right?  What kind of value?  Using the retail enterprise licensing and previous VSPP license costs, here’s my estimation:

• vCenter Server Standard + 3yr SnS = $8179.95
• vCenter Chargeback + 3yr SnS = $2,048.15 per 25 VMs/yr
• vShield Edge + 3yr SnS = $6,141.90 per 25 VMs/yr
• vCloud Director = $3,750 per 25 VMs/yr + $937.50/yr SnS

If we plug those in and add them to the enterprise licensing costs, here is what it looks like:

Holy schnikes, Batman.  We end up with $1,051,252 in additional “licensing benefits” in this environment (assuming you are using all of them), which means if you are using all of the additional features, that the VSPP program ends up giving providers MORE in free licensing than they actually collect in costs by a factor of 10.  Because vCloud Director and vShield Edge are licensed by VM, the value here is going to scale with the number of VMs.  In total, when compared to the vSphere5 license costs, we end up seeing the VSPP cost being 1001% cheaper over three years.  Ladies and gentlemen, that’s officially ridiculous.

There is a LOT of opportunity here for SPs who understand the licensing model and can leverage it for customer-facing services.  As you start to scale your infrastructure out, the additional value of the VSPP program tracks almost directly with the number of VMs you have in production.  If you have large, fully-reserved VMs, the VM count comes down and so does the value of the additional licensing.  This actually makes me happy from a Service Provider standpoint, since it puts the business decisions back in the hands of the provider.  Here’s what I mean:

Every VM in production at a service provider is simply a math equation.  One one side, we have the costs; capital equipment purchases, refresh and depreciation rates, staffing costs, marketing, licensing and the rest.  On the other side we have the revenue being generated.  Some SPs do revenue as a two-factor equation.  First, you have the direct revenue, usually the resources that can be directly tracked back to the capital and staffing that was invested in the platform.  Of course every SP wants to try and drive “attached” services such as backup, disaster recovery, OS and application management, professional services and maybe even co-location and data center services.  Each of these offerings has its OWN equation for cost and revenue, but since they are “attached” to the VMs being provided, you track that revenue in a separate column.  It’s important to understand the whole revenue ecosystem in play so that you can be responsive to customer trends!

Each object on the cost side also has a possible multiplier: risk.  If an SP wants to go cheap on the infrastructure or staffing investments they can, but each of those carries risk that may increase the cost of the service, or worst case bring the revenue down as customers leave.  The same goes for oversubscription and VM density.  The more VMs you have on a host, the more exposure you have during an HA event, and the more you oversubscribe resources the more you risk impacting customer performance.  Both oversubscription  and density also drive more staffing since you have to keep a closer watch on things from an operational standpoint.  Are we all seeing the complexity here yet? :-)  Math is sexy, kids.  It makes the world go round.

All that to say this: every running VM has at least one actual, real dollar figure attached to it.  Way back in the early days of VMware hosting when everyone was using perpetual licenses with a EULA amendment, the business could determine how much risk they wanted to introduce (and how much additional revenue they wanted to realize) while being able to keep the licensing costs flat.  I think it’s safe to assume those days are over, but this is the first version of the VSPP program that gives the business the possibility to increase the number of VMs supported without increasing the licensing costs! If I decide to move my reservation level from 100% to 50%, I have the ability (although I may not choose to) increase the number of VMs by 50% at no additional licensing cost.  Since the SP is the one making the capital and staffing investment and managing the offering on the open market, it’s only fair that they have the ability to choose their own risk/reward levels, and now they can do that with their VM licensing again!

So why is VMware doing this?  I don’t know, they haven't told me!  I have some opinions though, and I’ll be happy to share them if you ask. :-)

Where are you on the continuum? Here is a link to an Excel spreadsheet that I’ve put together to help my SP customers figure out where their “sweet-spot” is, and help position the costs to the business development side of the house.  I’ve tried to be thorough and make sure my math is correct, but please let me know if you find any issues.  I’ve protected the sheet to only allow editing in the appropriate fields, and I want to give a shout-out to Kendrick Coleman who made it “pretty” for me!

VSPP Calculator v0.4

At the end of the day, I think this is a huge break for the Service Providers, many of whom have been waiting for a long time for VMware to give them one.  The VSPP licensing model is definitely trying to drive behavior with the SPs, and we are going to see more and more IaaS environments that heavily utilize memory oversubscription in order to drive as much value as possible out of the program.  It’s also going to drive adoption of the “extras” that are included, namely vCloud Director and vShield Edge.  You can’t argue with free, and even though there are other options out there in both of the spaces these products live in, it’s going to make SP customers think twice about paying for alternatives.  In particular I think that the Cisco VSG product is going to take a hit seeing how it largely competes directly against the vShield Edge product.  It also means that VMware is going to have to step up to the plate with features and functionality in those products, but if they do so they'll have a huge install base to leverage as they start moving more of the application and database services to the cloud.

Hopefully this has helped you understand more about the VSPP program, the recent changes that were made and how they relate to environments large and small from a licensing cost standpoint.  If you have any questions, comments or suggestions for the calculator tool, polite comments by readers who are willing to disclose their vendor affiliations are welcome!

VMworld 2011: Recap Day 0

Like Aaron Delp I’m going to try and keep a running account of the show and it’s goings on, however I’m not as dedicated as he is, and these will tend to come out the morning after, instead of the night of.  Let’s just say I’m not sure my typing skills are up to the task after a long night in Vegas… :-)

Ah, Vegas; my old friend.  I never thought I’d say this, but I honestly wish VMworld had been back at the Moscone Center in San Francisco.  After five trips to Vegas this year, I’m pretty much over it, and would gladly trade the craziness for some time down on the water.  That being said, if you have to be in Vegas, the Venetian/Palazzo is definitely not a bad place to be!

Despite Hurricane Irene (or as my brother has nicknamed her, Hurrican't Irene) causing some travel issue on the east coast, Charlotte was far enough inland that we didn’t see more than a passing shower.  Good thought go out to everyone who was affected, especially Kevin Kelling and his family after some flooding issues in New Jersey.  I hope everyone made it through safely.  After being on the road all of last week, I spent the weekend at home with the family and flew out Sunday afternoon.  It was a lucky coincidence that I had most of the Varrow team on the plane with me!  It’s always good to hang out with Dan Weiss, Jason Nash and Joe Kelly, and even though Dan sometimes “forgets” to take his laptop out of his bag and gets strip-searched by the TSA, the flight was fairly unremarkable. 

Once we landed, the race began!  After checking in, I immediately headed to Chad Sakac’s 4-hour power meeting with the entire vSpecialist team.  Holy smokes what a group.  We got to hear from a bunch of different functional groups on where they are going technology-wise, and the VCE executives got to talk about how things are going on our side of the house.  In the spirit of work hard, play hard, there were two fantastic presentations, one by Chad to the “vOscar” winners highlighting the vSpecialists who went above and beyond in the last year.  I’m lucky enough to get to work with the vSpecialists a lot, but to see guy like Chris Horn, Tee Glasgow, Simon Seagrave, Nick Weaver, Anton Zhbankov and Rich Swakla get recognized for their efforts really makes you respect that organization.  Tee’s video montage of the Enablement team and some of the craziness they’ve gone through this year to make events like EMC World and VMworld happen gave me goose bumps. 

The highlight of the meeting came at the end where Chad turned on the fire hose and gave us 60 straight minutes of VMworld content prep including an overview of the keynotes, the micro and macro roadmap and a bunch of great interlock information.  That man is a machine, and it’s awesome to watch him in full “event” mode.  Looking at what’s coming this week, all I can do is tell you to buckle up.  There are at least 4 different announcements that floored me with the scope of the implications they will have on the industry and how we do what we do.  Yes, seriously.  Just watch.

From Chad’s meeting we got on the bus and headed straight to the v0dgeball tournament.  VCE had a team (I’ll spare you the details on how we did…) and everyone was there to support a great charity (The Wounded Warrior Project).  The fun level was high, as was the pain level afterwards.  I’m pretty sure Wade O’Harrow blew a hamstring, Chris Hoff from Juniper took a ball to the eyeball and was having some issues, and that was just the injuries I saw!  We are old geeks, what can I say?  I gave more than I got in the hitting with balls department, but I’m not going to audition for a team anytime soon!

Back to the hotel, and then to dinner.  Kenny Coleman and his (long suffering) wife Lauren, along with Rob Marković from VCE invited me over to I Heart Burgers at the Palazzo where we had a great time.  Kenny ordered too much beer, Rob asked the waiter too many questions and Lauren begged for less “geek talk” but it was great to see everyone!  After dinner, we wandered down to the casino and ran into Jonathan Donaldson, Eric Oberhofer and Chad Sakac playing craps.  It’s nice to watch the high-rollers in action!

And with that, day 0 ended.  Now it’s time to put the final touches on my theater presentation (vCloud on Vblock, come see it in Booth 1121, register and get my badge, finish up my blog posts for the week and track down John Troyer and introduce myself!  Tonight is the vGeekfest party with the vSpecialists and then #CXIparty at the Cosmopolitan, so we’ll see how things go!

VMware vCloud on Vblock

More and more of the discussions I have with service providers revolve around how to tackle the challenges being placed on the operations side of the house.  Fortunately, VCE and the Vblock have lots of tools and features that we can bring to bear on that issue, and one key piece of that story is VMware’s vCloud Director.

One of the challenges that vCloud Director poses for Service Providers, is that it’s not a management application.  Most SPs will leverage the included Vblock Advanced Management Pod to house all of the non-revenue generating pieces of the infrastructure, but vCloud Director, with it’s incredible active and important database, vCenter instance and web nodes takes some special consideration and planning.  It’s much more a Tier 1 app than it is a management tool, and that changes how things get laid out somewhat.

With that as background, and driven by the rate of adoption we were all seeing in the field, a group of VCE and VMware folks got together and started working through how vCD could be best implemented on a Vblock.  Kendrick Coleman from VCE and Chris Colotti VMware will also be posting this document and their thoughts, and we broke the design up into pieces and contributed to the final solution, focusing on four key areas:

• Management Infrastructure
• Networking Infrastructure
• Storage Infrastructure
• Scalability

It’s important for me to make a couple things clear before you dig into the design itself.  This paper is the culmination of a series of discussions and whiteboard sessions that the authors had, and not necessarily a solution that is supported or endorsed by either VCE or VMware.  It is designed to help people understand one of the more complete ways to implement the official VMware vCD reference architecture while staying true to the core of the Vblock standard.  There are definitely some suggestions in here that would need to be approved by VCE before they were implemented, so please make sure that anything you do with your Vblock isn’t going to impact your ability to take advantage of the one-call support process!  It’s possible that there may be an “official” vCD on Vblock reference architecture at some point in the future, and if that happens we’ll definitely point people in that direction…

This is also very much a work in progress.  Hopefully it will change due to feedback from you, our readers and the customers who are using these designs in the field!  Obviously there will have to be some changes made based on the soon-to-be-released vCD1.5 release, but the core of the architecture holds true for both versions. 

This was a blast to work on.  Kenny and Chris in particular, but the entire group including Sumner Burkhart and Sony Francis, are great to work with, incredibly smart, responsive and always focused on the customer and how we could best serve them.  There were some long days in conference rooms drawing on whiteboards, weeks and weeks of e-mail exchanges, at least one instance of Kenny almost losing his sanity and a very memorable dinner which included all of the New England Patriots cheerleaders and someone using the pickup line “I was there when you were born”.  These guys are the reason I love this job, and I can’t wait for our next project.

Enjoy the paper, and please make sure to send your feedback!

VMware vCloud on Vblock Design Considerations

vSphere 5 - vCenter Web Client Comparison, Part 1: Hosts and Clusters

If you haven’t seen it yet, the new vCenter Web Client is beautiful.  Seriously, it looks great.  There seems to be a good bit of Flash used which can be painful at times, but it’s definitely the best looking GUI that VMware has ever created IMO.  Is it a replacement for the full vCenter client? Probably not.  There are a half dozen or so posts that will give you all the screenshots you’d like, so this won’t be one of those.  I’m going to try and detail the differences between the full UI and the web version and talk specifically about a couple places I think it could use some improvement.  I’ll go through each of the four pieces of the management environment (Hosts and Clusters, VMs and Templates, Datastores and Networking) in a separate post to keep them from getting too long…

 

Overall, I’m pretty impressed.  The GUI is in a traditional 3-column layout, with the left column handling the inventory trees and the right column handling the Tasks and Alarms.  Each of the elements in the center and right columns can be moved, re-sized or maximized as you see fit, which is a nice touch.  I couldn’t find anywhere that there were additional widgets that could be added, although it looks as if that’s at least a possibility in the future.

Some notes about and minor differences in the general components:

• While the “My Recent Tasks” widget does refresh if you initiate an action from the web UI, it doesn’t refresh otherwise, so you don’t get to see a real-time task list like you do in the full client.  In an environment where multiple people may be connected, it’s nice to be able to see what other tasks are being performed.  If you manually refresh, you WILL see the other tasks that have been run, but you won’t see them in real-time.
• The concept of the “Work in Progress” window is actually pretty cool.  Anytime you are working in a dialog box or wizard, you can minimize it and it’ll show up on the “Work in Progress” list.  For instance, if you started provisioning a VM and needed to go do something else, you’d see this in the WIP widget.  All you have to do is click on it, and you’ll be put right back into where you left off, with all of the data you’d already entered having been preserved.  This is another fantastic feature I’d like to see moved into the full client, for sure.  Having a wizard open and having it lock the entire client sucks.  Don’t know what I mean?  Open up the “New Virtual Machine” wizard, minimize it and then try to do something in the main client.  See?
• There’s no access to permissions at all.  You can’t view or edit anything to do with permissions inside vCenter/vSphere at all.
• The pop-up notifications are a nice touch although I seem to get “Error:null” notifications a lot…
• You can drag-drop VMs from host to host and into resource pools/vApps, but there’s a LOT of lag so you need to do them slowly and wait for the process to complete, and for the view to refresh.  I’ve had a couple instances where I had to exit out of the Web Client and log back in to get everything straightened out. With both the clients open side-by-side, there was an additional 7-15 seconds that I had to wait while the web client figured things out after the full client had registered the changes.

vCenter Management

The lack of permissions access is the only real difference between the versions here.  For some reason the Web Client doesn’t enumerate the names of the actual datacenters that have been defined, but other than that it’s pretty close to 1:1.  You also don’t have any ability to add new datacenters or to add new hosts or folders.  It’s 100% read only at this level.

Datacenter Management

No Maps or Storage views here, and the Performance tab has been dumbed down (no advanced option, no individual cluster detail) but the Hosts, VMs and IP Pools lists are identical.  There is a new tab on the Web Client called “Storage” which shows all of the datastores that reside in the datacenter, which is nice. Like elsewhere, there is no access to permissions or any ability to add clusters/hosts.

Cluster Management

The cluster management is fairly poor in my opinion.  You don’t have access to DRS or DPM recommendations, you don’t have Maps or Storage Views and you can’t see profile compliance from the Web Client.   You can’t add hosts, and you can’t do anything with the existing hosts except put them in maintenance mode.  The Web Client does have a pretty “Utilization” overview on the “Resource Management” tab that shows guest and host RAM and Host CPU.  There’s a poor rendition of the DRS resource distribution chart that doesn’t give much detail, and there’s no summary of the DRS/DPM configuration like there is in the full client.   You can create resource pools and vApps from here, which is good.  You can also edit properties like VM start order inside the vApps once they have been created, and you can drag-and-drop VMs into both objects.

Host Management

One of the things I’ve come to rely on (a lot) in the full vSphere Client is the keyboard shortcuts that jump you back and forth between the different management modes.  Ctrl-Shift-H gets you to “Hosts and Clusters”, Ctrl-Shift-N gets you to “Networking” and so on.  In the Web UI, those shortcut keys don’t work, but VMware has made up for that somewhat by putting nifty little tabs at the top of the left-hand column that correspond to each function.  This is one feature I’d LOVE to see make it’s way to the fat client, since using the “breadcrumbs” at the top of the full client is very unwieldy. 

The “Summary” screen for each client is pretty similar, however there’s some pretty important information left off the web client including:

• License status
• CPU/RAM Usage (no, the mouse-over on the CPU graphic doesn’t count…)
• HA Status
• EVC Status
• Quick views of attached storage and networks. I don’t know that this is a huge loss since those small windows on the full client end up being pretty useless on a production server anyway…There are links on the web UI to the full host storage and networks tabs.
• No FT information

Including the general information that’s missing, you also have a very limited set of “commands” that you can use on the server.  Missing are the “Reboot”, “Shutdown” and “Enter Standby Mode” options.  “New Virtual Machine” and “Enter Maintenance Mode” are still present, although you have to go to the “All Actions” object at the top right of the pane to find them.

On the “Monitor” tab we can look at Tasks, Events, Performance and Alarms.  It’s nice that all of these are consolidated onto one tab, whereas on the full client they are spread over three separate tabs.  I like that you can still Show/Hide Columns from the right-click menu, and the options to lock the first column and size all columns to fit are nice (another one that would be nice to have in the full client).

There are some differences that can be annoying however.  For instance, compare the “Tasks” view from both clients:

There are a whole bunch of Tasks that the web UI just wouldn’t show no matter how much I refreshed.  In order to get it to show the most recent entries I had to log out of the UI altogether and log back in.  I suspect this is a bug, but it’s definitely annoying.  I didn’t find the same behavior in the “Events” tab, strangely enough.  It still didn’t auto-refresh, but when I clicked the button to refresh manually, it did.  The Tasks, Events and Alarms tabs all worked much the same as their full client counterparts, including the details and related events with links to the relevant objects. 

The Performance tab is particularly impressive from a comparison standpoint, in that the default Web Client and full client views are 100% identical.  Same stats, same graphs, same color, same everything.  The drop-down for setting the time-range includes the same options, and custom ranges can be set the same way in both.  The glaring difference is that you have the “Advanced” option in the full client which doesn’t exist at all in the Web Client.  Especially for those of us who are old-school and have custom charts defined for looking at the hosts, this can be a big difference.

The Virtual Machines tab also comes pretty close to being a 1:1 match.  It appears that all the same columns are available for viewing, although there’s no option to export the list like there is in the full client.

The Networks section, on the other hand, is pretty disappointing.  It shows the basic network objects that are defined on the host (or that the host belongs to), but that’s literally all it shows.  There’s no listing of the VMs that reside on the port groups, there’s no listing of the uplinks or NICs that belong to each vSwitch/dvSwitch, and there’s no way to edit anything that I could find.  This is very much a read-only version of the absolute basic information, and isn’t very much help at all.  This part could definitely use some additional functionality.

The Storage tab is better, but only a little bit.  Content-wise, it’s a duplicate of the “Storage” section on the “Configuration” tab inside the full client, complete with all of the same columns to view.  What it doesn’t have is any of the information from the “Storage Adapters” section or any of the Storage Views, both of which can be very useful.  There’s no way to edit any of the information here, and most baffling, there’s no way to ADD additional storage volumes. 

Overall, the Hosts and Clusters section isn’t bad.  There’s definitely some content missing, especially in the Networks and Storage sections, but it’s much better than anything we’ve had previously.  In fact, there’s only one omission that I’d consider unforgivable.  Take a look at the screenshot to the left and answer me this:  Are those hosts powered on?  Are they in maintenance?  Are the VMs powered on?  Are they FT protected?  You have no idea, because THEY DON’T SHOW ANY OF THE STATUS ICONS ON THIS VIEW!!! 

To make matters worse, the icons they do use for the VMs are the same ones use for powered off VMs in the full client so I have a mini-panic attack every time I log in.  It’s not that they didn’t have the icons available, since the VMs are represented properly on the Virtual Machines tab (see pictures to the right) and the hosts are represented correctly on the Hosts tab under the cluster objects.  So why are they not represented that way on the navigation pane on the left?  This is easily one of the most annoying things about the entire web client.

I think that covers everything on the Hosts and Clusters tab, and as you can tell it’s a mixed bag.  Some of the functionality is recreated faithfully, some is missing altogether and there are some new things (like the “Work in Progress” widget) that have been added.  Next we’ll look at the VMs and Templates section…

vSphere 5: System Logging Not Configured By Default

Playing around in the lab this week, I noticed something interesting.  I’m not sure if it’s a bug, or by design, but it’s new with vSphere 5 so I wanted to share.

I have one lab that has vSphere 5 installed on internal hard drives, and everything looks just like it should.  When I look at the logging options under Configuration Tab | Software | Advanced Settings | Syslog | Global I see the following:

Note that the logDir is set to “[] /scratch/log” putting the logs on the local disk.

In my other lab, I have vSphere 5 installed on USB sticks (as detailed in this post).  After install, there is an alarm on each host that looks like this:

Interesting, right?  Same install, just to different media, and it appears that vSphere knows that there’s no persistent disk and has decided to leave the logging unconfigured since it doesn’t know how much space is available for it.  Now I’m totally guessing on that point, but it does make sense.  Does anyone out there have a more “official” answer for the change in behavior?

If I check the Advanced Settings on the host with the alarm, I can see that there’s no logDir defined at all:

Sure enough, when I browse the /scratch/log directory, it’s completely empty.  Just for kicks I decided to put the same default value in on one of the unconfigured hosts and see what happened.  The value is accepted and the alarm goes away.  When I look in the /scratch/log directory now, I see all of the expected log files:

In the abstract, this seems like a welcome change.  For hosts that are being installed diskless, vSphere appears to be acknowledging that fact and leaving the logging options unconfigured.  While this means that you have to manually add the logging path (whatever you want it to be) after the fact, it would seem to be intended to prevent a situation where the default logging settings would cause an issue.  A small feature, sure, but one that could be very useful.  If you have any thoughts on this “feature” or if you have a more official answer to why the install process is different based on the media being installed to, let me know in the comments!

vSphere 5 Licensing: Tempest in a Teapot

So right up front, I’ll tell you there’s no way I’m offering an opinion on this topic. There’s no upside, and I’ve learned my lesson the hard way. For all of you who have sent me a DM, e-mail, etc.. asking if I was going to put up a blog post with my feelings, there’s your answer. I am willing, however, to (carefully) lend some perspective and advice to both sides as the discussion continues.

What I want to do is make sure that everyone understands the resources that are out there so that you can figure out what this means for you and your company.

First, the official documentation: http://www.vmware.com/products/vsphere/overview.html

• vSphere 5.0 Licensing, Pricing and Packaging Whitepaper
• vSphere 5.0 Licensing Model Video
• vSphere 5.0 Licensing Webcast
• vSphere Purchase Advisor

Next, the official VMTN thread: vSphere 5 Licensing. By my count, the thread has:

• A load time of around 30 seconds
• 33,000+ views since July 12th at 5:23pm (>5,000 day, or >208 per hour)
• 113,150 words of text
• 685 individual posts
• 231 mentions of Hyper-V
• 98 mentions of Xen (interesting numbers between those two…)
• 128 mentions of the words “money”, “cash” or “revenue”
• 30 mentions of the word “fair”
• 35 instances of posters cussing
• 9 different vExperts contributing a total of 63 posts.
• 6 different posts complaining that the vExperts aren’t chiming in enough.
• 1 post complaining that a specific vExpert is posting too much and just trying to drive traffic to his website. O-o

This thread certainly isn’t the only one on VMTN! I counted around 20 more that were active all discussing the same issues. Feel free to jump in wherever you see fit…

Finally, here’s a link to the licensing presentation and “Q&A” sessions that are happening over the next few weeks: http://www.vmware.com/go/vsphere5-licensing-webinar. There is a text-based chat Q&A included, so ask what you need to ask (respectfully) and push to get the answers your business needs.

Whatever your feelings, and however the licensing is going to affect you, speak up! VMware can only react to what they know about, so if you have a beef, or if you think the new model is fantastic, please speak up on VMTN, to your reseller/partner, to your VMware rep, to anyone who will listen. If you just want to vent, let me know where to meet you and the first few rounds are on me.

To the customers, I say do the math yourself. Don’t just look at what you are going to pay today, look at what you are going to pay to license the infrastructure you’ve already invested in. Make sure you account for HA, be realistic with your consolidation and RAM/VM ratios, and figure out if there’s a delta and how much it will be. If you have an ELA, you are not required to change to the new licensing model until your current contract expires, and if you don’t have an ELA maybe you should think about it. After you do the math, of course. If you are confused, there are a ton of great VMware partners out there who can help. I know Jason Nash (@nash_j) over at Varrow LOVES to answer VMware licensing questions, so hit him up on Twitter!

To VMware, I say back away from the speaking points and listen to your customers and partners. Stop telling them to run the tool when they are asking about the capacity they could have used in 4.1 versus the capacity they will have to pay for in v5. Customers buy hardware to last 3-5 years, so if someone just bought hardware, and you tell them that they won’t have to pay more today (like 95% of your other customers, we got it…) understand that they are talking about tomorrow as well. Yes, they know you have removed the core-count cap, but you can fit an awful lot of RAM into the 6-core CPU that was covered under the old Enterprise licensing… Finally, be willing to discuss the consolidation ration and RAM numbers that you cite as the basis for the allocations. I’m hearing from customer after customer that they don’t make sense in their environment. I’m not saying you have to agree that a 60:1 allocation or a 32GB VM is “typical” today, but seeing how the vSphere 4.1 licensing included targets (6-core for Enterprise and 12-core for Ent+) that are still, 2+ years later at the very, very far end of the hardware use-cases I think asking why you used today’s numbers versus where you project them to be two years from now is fair. Yes, over the next two years customers might have to choose to limit their core count in order to use the old licensing model, but don’t push that as “simplicity” or “evolution” when customers would be happy to make that choice, and in many cases already have as evidenced by their most recent hardware purchases. Finally, please stop using the word “fair” in the discussion. The statement that you are aligning costs with “value derived” is extremely subjective (at best) and I have yet to find a customer who has responded well to this word-play.

vSphere 5.0 is an incredible product, made by an incredible group of people. Let’s ALL, customer, partner and VMware alike, work hard to make the licensing model supportive of everyone’s desire to grow the use and adoption of the greatest hypervisor in the history of the world. All this focus on the licensing is taking away from the hard work done by thousands of VMware developers and partners, and keeps all of us from being able to focus on continuing to push into the Tier 1 app space.

PCs and Home Labs and Data Centers, Oh My... (Part 2)

With part one of my whole home computing make-over complete, it’s time to get finish up on the lab!  There are tons of other great blogs out there that talk about home labs (like this one, and this one, and this one, and this one, and this one...), and I definitely recommend you go through and pick what makes sense for you.  There are definitely wrong ways to build a VSphere lab, but there are also many, many right ways to build one.  Don’t get hung up on host count, NIC density, switch models and case types, just find something that fits into your budget, fits into your environment and does what you need it to do. 

In my lab (as you may remember from this post) there are a total of five hosts.  One host does nothing but management functions (vCenter, AD, etc…) and the other four are completely configurable.

In my case, I built a list of use cases that I wanted to support:

1. I wanted to stay away from virtualized hosts, if possible.  This is 100% a personal preference, because when I’m buzzing around in the lab I don’t want to worry about how many layers nested in I am.  There are still situations when I’ll use them, but one or two physical hosts wasn’t going to cut it for me.
2. I wanted the hosts to be as close to 100% supported on the VMware HCL as possible.  I hate troubleshooting some strange performance issue only to find out that it’s not supported by VMware.  NICs, CPUs, I want to get as right as I can up-front.
3. I want to be able to simulate multiple sites.  Whether it’s SRM, storage replication, application testing or what have you, being able to simulate real-life network conditions was a must for me. If you want to simulate WAN conditions in the lab, there are a couple tools out there.  WANem has both a bootable ISO version as well as a virtual appliance, and both work pretty well.  I’ve used both (I have a couple old beater Dell PE650 servers with dual-NICs that work great for this), if there’s any interest maybe I can put up a blog post on it…
4. I wanted to be able to use VLANs to segment the different setups in the lab, but still be able to route between them. This was especially critical for the vCloud Director and multi-site labs (SRM, RecoverPoint, etc…), but in a classic case of want my cake and to be able to eat it, I needed something that was quiet and that I could use my existing Cisco skill-set to manage.
5. I needed to be able to support multiple hypervisors and switch the lab hosts from one to another quickly.
6. I needed to be small enough to fit on an existing shelf in my office, I needed to be able to support it with a single UPS and it needed to be quiet/cool enough to not get my wife upset at me. :-)

So the hardware order was a go, and if you want to see that again here’s the link to the NewEgg.com shopping cart. I went over the rationale for the hardware in my previous post, so we’ll skip to the actual Lab build process.

First, I love these cases.  They are small, they look good, they have two USB ports on the front and they are easy to work with internally, which isn’t always the case with micro-ATX cases.  The edges are nicely rounded, so there was no blood shed, and the drive casing at the front disengages and slides out to make everything easy to get to.  There’s also enough slack in the cables that I was able to do a decent job of cleaning things up.  Finally, the BIOS has a “green” setting which allows you to turn off the annoying LED lights on the board, so once it’s up it’s completely dark.  The single 80mm fan on the side is OK, although it’s a little loud.  If I can find a good deal on some silent fans I might replace these and see if it helps any.  Honestly, one of them running isn’t a big deal, but five of them side-by-side and it’s louder than I’d like.  Overall, I was very impressed with the cases and the ease of construction.  In fact, after building the first one and figuring out the cabling between the case and the motherboard, I decided to see if the kids wanted to help.  My little boy is 4 and his sister is 3, so there was definitely the potential for disaster, but they did awesome.  Both built a whole PC from scratch (mount the motherboard, plug in the power cables, seat the CPU and RAM), with Dad to help with the CPU cooler spring mount and the little case cables.  I was very impressed, and both got to write their names inside the case so everyone knows that they built it.  My daughter was particularly excited with this and asked if I could take a picture of her!  I’ll have to start teaching her how to deal with stupid IT boys…

The switches are sweet.  Completely fan-less, able to do ACLs/layer-3 routing and with a Cisco logo on them for less than $250, they may be the best lab switches I’ve ever had.  The version of “IOS” that they run is…ummm…different, but the web interface is actually very, very good so that balances out. 

The NICs have also been perfect.  I was worried about spending more for the dual-port Intel cards, but hearing about the issues some of my friends have had here, I’m glad I did.  They have been 100% solid from day one, support wake-on-LAN and haven’t given me a bit of problems.  I like the dual port card as well, since there aren’t a ton of available slots on the motherboard.

After putting everything together on the rack and cleaning up the cabling, I have to say I’m pretty happy.  Here’s a couple pictures so you can see how it looks, and then we’ll talk about the VMware side of things. The rack is one I’ve had forever, and it fits perfectly under the desk in my downstairs office.  It has three shelves, but I’ve removed the middle one to be able to fit all of the hosts in on their sides.  The single fan on the sides of the hosts works great here! A liberal application of zip ties and everything is in place.  On top, you can see the lab switches, and then the firewall, internal LAN switch and one of the wireless access points use in the house.

The only real hole in the lab right now is the storage. I have an old ReadyNAS NV+ array that doesn’t do iSCSI and that gets easily crushed by too many VMs. I’m going to have to figure something out there, but I’ve been looking longingly at the Synology DS1511+ as a replacement. We’ll have to see how it goes.

If you looked at the pictures, you may be asking what the USB keys are doing plugged into the front of the hosts.  That was my answer to being able to quick switch between hypervisors!  I have figured out how to boot almost every hypervisor I need from USB, so if I want to switch from one version to another, I just reboot the host and swap out the USB key!  I have all the standards, including vSphere 4.1, XenServer, Xen 4.1.1 and Microsoft Hyper-V server.  I’ve also got some new VMware goodness in there, and some stuff I can’t even talk about, but having everything on USB makes it super convenient.  Want to see how different hypervisors work together?  Want to split the four hosts into two sites?  Now it’s easy!  There’s more than enough horsepower to go around, so I can do what I want to do.

So now we’ve gone through the home PC and the home lab, so what else is there?  Oh boy, now it gets crazy.  I don’t want to give too much away but the next post in this series will detail the offsite lab?!? Stay tuned…