I am beginning to think that it is a universal truth that geeks love to build some sort of playground or work area for their experiments. Whether that is physical or digital, it becomes a sort of escape--a place to try out new ideas without destroying something of value. If you are nodding your head in agreement then you know what I am talking about.
Before virtualization, we had computers stacked on top of computers just to run a digital playground. After all, what fun is a single host network? Then came the prevalence of virtualization and the market was easily dominated by the trailblazing VMWare. Virtualization not only helped me save on hardware, but also power, space, and cooling. Matter of fact, I think this was about this time that I discovered that HVAC was required to heat the house during the winter months. ;)
In the past, I have exclusively used VMWare for my at-home virtual environment. Starting way back with VMWare Server for Linux (a type 2 aka a hosted hypervisor) and eventually moving to ESXi (a type 1 aka native or bare metal hypervisor). However, I recently built a new rig and had to make a decision of whether to go back to VMWare ESXi or try out Microsoft’s Hyper-V Server 2012 (both type 1). I took a little bit of time to document my experiences and I think you will be as surprised as I was with the results. Hopefully this will save you a few sleepless nights that I lost along the way.
**Keep in mind, most hackers are not independently wealthy and thus we will be using FREE PRODUCTS ONLY! Sure, our employers can drop 10k or even 60k on all the bells and whistles, but we are trying to build a production-like environment for our at-home use.**
On with the geek holy war! (Maybe for the next set of articles, we will throw in XenServer to really spice things up).
Comparison
For a quick summary, I threw everything together in the below table. Read on for more details:Category | VMWare | Hyper-V | Winner |
Hardware Compatibility | Horrible - NIC driver missing on my setup | Excellent - no issues | Hyper-V |
RAM supported | 32 GB | All 64 GB - up to 4TB | Hyper-V |
Stability | Bad - Purple screen of death | Excellent - no issues so far | Hyper-V (in my case - most likely due to HW compatibility) |
Boot speed | Slow - minutes to complete | Fast - 30 seconds or less | Hyper-V |
Features | More developed and refined features | Free Hyper-V is missing File transfer to datastores and Free robust virtual machine converter | VMWare - Mainly due to Linux OS support, more robust management and conversion clients |
Management | vSpere client, Workstation, SSH | RDP, Hyper-V Manager | Tie - VMware wins in VM management and conversion, however Microsoft wins in hypervisor management |
Size of hypervisor | 4GB | 8GB | VMWare |
Hardware
The hardware for this hacker playground is as follows:- Cooler Master 932 High Air Flow (HAF) full tower chassis
- Intel i7-3930K processor - Liquid cooled
- ASRock X79 Extreme6 motherboard
- 64 GB Corsair RAM
- 180 GB Intel SSD hard drive
- 2 x 2 GB Western Digital Hard Drives
- 2 x ASUS DVD Burners
Note the amount of RAM, it becomes important later.
Playground requirements
When designing the virtual arena, I had a few requirements in mind:- Utilize the hardware above
- Ease of installation
- Stable environment
- Fast VM provisioning
- Quick snapshot recovery
- Ability to share VM access
Marketing
Both VMWare and Microsoft obviously spend money on professional marketing and comparisons. Depending on who you listen to--each will claim that they are cheaper than the other:"VMware (finally) admits that its costs are higher than Microsoft’s" - From Microsoft
VMWare Responds: “Flawed Logic Behind Microsoft’s Virtualization and Private Cloud Cost Comparisons”
And third parties chime in: “HYPER-V VS. VMWARE COMPARISON”
However, for my hacker playground, I won’t need platinum support. Ultimately, it will come down to how many features I can get for the low, low price of free. On paper, it appears that Microsoft steals the show.
In fact, here is a competitive feature comparison from Microsoft. This comparative analysis paper points out VMWare’s greatest limitation of the free ESXi product. ESXi has a hard limit of 32GB of RAM unless you license the product. Remembering back to the hardware statistics stated earlier, my rig has 64GB of RAM. A few years ago that amount of RAM would have been out of the price range for the average consumer--however, now it is only $300! Wake up VMWare! We can build home boxes that exceed what you support in the free ESXi. How about Microsoft Hyper-V Server 2012? Do they have this limitation? Not even close! Check out the graphic below, they support up to 4TB!

Well, this cannot be true. Can we find something from VMWare that confirms this 32GB limit for free ESXi? Sure can, see the image below from VMWare:

Wow. That is a tough obstacle for VMWare to overcome. That leaves me three options:
- Stay with their product and disable half of my memory
- Pay thousands to license it for my house
- Adopt Microsoft’s Hyper-V or another virtualization product
Ok, this crippling limitation aside, let’s see how they compare in overall experience.
ESXi experience
To describe my personal ESXi experiences, I grouped the experiences into a handful of categories below.Hardware compatibility
I must be honest in saying that my most recent overall ESXi experience has been horrible--mainly due to the very limited hardware compatibility list (HCL) and the 32GB limit on the RAM. If your hard drive controller is on the list, your NIC isn’t or vice versa. The average home system running a reasonably priced motherboard is most likely not going to be 100% compliant with the HCL which means building your own custom image with added drivers. I had to hunt around to find an article that explained how to create a custom USB ESXi image with extra drivers for the components I was missing. On top of that, the version of ESXi that I could get the drivers loaded on was very unstable. Overall, it was the RAM limitation and the hardware compatibility issues that prompted me to look at other virtualization solutions--the hardware compatibility list for VMWare is horrid.RAM supported
Did I mention ESXi embarrassingly only supports 32GB of RAM.Stability
The next category of importance was stability. With VMWare ESXi, I would get a purple screen of death--yes, that’s right, purple. After much research I find lots of other people with the same problem, but no real solution other than upgrade the version of the image which would cause me to lose my drivers. I would just purchase another NIC (that is on the HCL) in order to overcome the stability and driver issues, but I still have the RAM limitation to deal with.Boot speed
VMWare ESXi takes forever to boot/reboot. Minutes. Sometimes as long as 5 minutes to completely start all of the services. I hated rebooting ESXi and avoided it when possible.Features
VMware is great about supporting ALL operating systems--including Linux. The data store management is a breeze with included file transfer capabilities. Included performance meters are useful and sometimes necessary for a bit of troubleshooting. Resource pools are very handy in provisioning. Free stand-alone VMWare converter is also a very useful and flexible tool.Management
vSphere Client as well as VMWare Workstation can both be used to manage your virtual machines. Management of the server itself is either done at the console or via SSH. I feel like Microsoft edges out VMWare here by providing RDP to the Hyper-V server.Size of hypervisor
VMWare has Hyper-V beat here. My guess is that VMWare has a smaller footprint because it has less hardware support (since it didn't support mine). As a result, ESXi can be installed on a 4GB thumb drive instead of an 8GB. Is it really that big of a deal? Probably not.Hyper-V experience
Here's my experience with Hyper-V in the same categories:Hardware compatibility
No compatibility issues here! I did not have to build a custom image with third party drivers. Download the ISO, burn it, install and done.RAM Supported
My entire 64GB of RAM is utilized. In fact, if I could afford 4TB of RAM, free Hyper-V would support that too. No competition here.Stability
So far, this has been rock solid. Even with multiple VMs running and really stressing the box, I have had no issues.Boot speed
Wow is Hyper-V reboots fast. Not only did it install in less than 10 minutes, but it reboots in 30 seconds! Moreover, 20 of those seconds are spent during the POST of my computer. (64GB of RAM takes a little bit of time to check). So, really Hyper-V was booting in 10 seconds.Features
Here is one category where Microsoft falls short. Keep in mind, I am purely going off of what is available for free (Microsoft System Center is not free)--but I miss being able to upload and download files to the datastore. With ESXi, we could use vSphere to upload and download ISOs or even VMs. With Hyper-V Manger, you have no file transfer capability. FREE stand-alone VMWare converter could convert many types of VMs into other VMWare VM’s--including Hyper-V virtual machines. Microsoft’s converter tool is much more limited. This will be discussed in Part III of this series.
Management
One added bonus in Hyper-V is the ability to remotely administer your VMServer using RDP with Hyper-V as opposed to command line over SSH with VMWare. This is a really nice convenience when you don’t want to get up off the couch to go into the office.
To manage the virtual machines, you can use Hyper-V Manager. This tool even looks similar to vSphere client which helps to reduce the learning curve.

But before we even get to being able to use Hyper-V manager, stay tuned for Part II of this series detailing my experience plus tips and tricks on how to do some under the hood work to get remote management working. This is BY FAR Microsoft’s biggest shortcoming on this product--closely followed by its lack of support for Linux and very limited conversion tool. To be honest, the difficulty with enabling remote management almost made me give up on it before even installing the first virtualized OS.
Size of hypervisor
As mentioned before, VMWare has a smaller footprint, but I would take a larger footprint combined with an easier install and more stability any day of the week.Final Thoughts
I have been a long-time adopter and advocate of VMWare, however I feel that they may have been riding on the waves of their success for too long. Unfortunately, in the absence of real competition, they could afford to do so. Now that Microsoft (and others) have moved into this arena and become a major threat--especially with Hyper-V 2012, I am looking forward to seeing both companies continue to innovate new features and enhance the end-user’s experience in the near future. I am very impressed with Hyper-V (after remote management is setup), but I still virtualize both Linux and Windows, plus I consume virtual appliances which are almost always created as VMWare images--both of these issues may be a problem with Hyper-V. However at this time, it appears the Hyper-V has the edge, but you won’t know the real winner until the final article.Stay tuned for part II of this series which outlines the painful steps in setting up remote management. There may be tips and tricks that will hopefully save you lots of time. Thanks for reading. :)
It's probably worth noting that for VMWare, the Memory available to your VMs is actually not available as it's own feature, it is tied to the number of cores you have available to your license. The free license allows 32GB RAM and 8 cores (if I remember correctly). If you need more cores or RAM you need to buy the combined feature license that increases both (you can't increase them individually).
ReplyDeleteThis put us into the tens of thousands of dollars to ultimately run 32 core systems with 98 GB RAM.
Thanks for the additional detail. :) Good thing companies have tens of thousands of dollars. ;) Now, if I could only find my money tree...
DeleteYour article have one big flaw: everyone know that esxi have hardware computability list and outside of it hardware is not supported. Windows usually have great driver support for desktop machines and you are proving it, but that make your hardware stability test not very trustworthy. The boot speed test is really interesting and need to be double checked, but with you using incompatible with esxi hardware ..... You can easily deceive someone new to virtualization.
ReplyDeleteUsually you don`t use home pc`s to create virtual environment, except for test purposes. Using only one host with the free editions you can hardly use or see the benefits of virtualization. I really doubt your tests have any real life value.
PS Sorry for my bad english.
No problem for the bad English and thanks for the comment. :) You are absolutely correct that VMWare publishes a hardware compatibility list and that outside hardware is not supported. My intent is not to deceive someone who is new to virtualization as I was pretty clear that this set up is a home test environment. I was merely providing my experiences with setting up both environments. One host with the free editions still creates a pretty nice playground to test out software, exploits, and other things that will trash your physical installs. Again, sorry to say, but this was not intended to guide corporations into using free virtualization on a desktop PC. Lulz.
DeleteNice! This was the very comparison I'm stuck in right now! Thanks for saving me a few sleepless nights... ;)
ReplyDeleteThanks Luke, I hope you get to the end of the series... I had plenty of sleepless nights trudging through it. My hope is that both MS and VMWare take note and fix the minor annoyances. :) Let the games begin!!
Delete@Anonymous:
ReplyDeleteActually the article was upfront that this was a testing environment. Sure ESXi isn't intended to run on workstations, but then again that's going to be a lab environment more often than not.
Yeah I'm not going to use VMWare to it's extent that I might in an enterprise environment, but at the same time, that wasn't the point.
I'm contemplating both for one reason- datastore Server 2013 offers an unlimited number of virtual machines on a single Server box. If I want I can install core mode akin to Hyper-V standalone and benefit from the lowered memory requirements of running Server 2013 VMs.
Ultimately I've got both environments in my test lab. You can build a rather beefy i5 processor based VM lab for like 600 bucks. Build 2 and use both.
Thanks for understanding the intent of the article :) (whew!!) However, you are a gluten for punishment running both environments... ;) My hat it off to you though. (deep bow)
DeleteFlawed test by running esxi on unsupported HW platform.
ReplyDeleteGet supported HW and run ESXi as demo. You will have all the features for 60 days.
Unfortunately, I did not want it just for a 60 day demo. That is a lot of work to set up for a temporary environment. Thanks for the comment though. :)
DeleteI have ESXi running on my Proliant DL 160 G6 at home. Its running on HP's ESXi install that made for this server. I have ESXi installed to a 1GB thumb drive so unless the VMWare unmodified is that much bigger, your size has to be wrong. My HP version appears to be the full version, just with added HP drivers. I am unregistered and can use the full 72 gb's of RAM in my machine. Once the 60 days ends, it drops to 32 GB's but I just down the server, DOD wipe the flash drive and reinstall ESXi with a fresh 60 trial period. Then readd the OS's from the datastores. Takes me less than 15 minutes but saves me a ton. I agree about the time for rebooting, i HATE when ESXi goes down and takes FOREVER to reboot. Other than that, I love it. No experience with HyperV yet but i found this researching it as i'm going to set it up within the next week. Nice blog.
ReplyDeleteHalllo, if you want to upload files to hyper-v server, you just use the comand mkdir and the netshare.
ReplyDeleteBeen working with visualization a long time now and in my opinion Hyper-V is overall better - but if you run *nic hosts - I would prefer VMware
ReplyDeleteTo get shared storage access with Hyper-V 2012, run this command in powershell
install-windowsfeature FileandStorage-Services -IncludeAllSubFeature
You don't need setup shares - you can just access your administrative shares eg. \\hyper-v\C$
I have the paid version of ESXi and I much prefer Hyper-V to that - boottime, accessing vm - hyper-v is a winner
One downside to ESXi is the host management. The downloadable client is not supporting all features only the webclient does but it requires its own infrastructure to install and run.
Hyper-V is designed with the big infrastructure in mind, and thus it can be an excruciating pain ITA to get Hyper-V administration working if you are not backed by an AD.
I recently came accross this resource http://blogs.technet.com/jhoward which is great when getting a non-ad Hyper-V environment running.