I noticed this week that the Transaction Processing Council (TPC) is working on a new benchmark designed to measure workloads across virtual machines: TPC-VMS. This benchmark builds on the existing benchmarks out there (TPC-C, TPC-E, TPC-H and TPC-DS) with the idea that companies want some idea of how various hardware and software might compare in virtual environments. In this benchmark, there are three systems consolidated onto one host running some type of hypervisor. If you are interested, you can read about the current benchmark, v1.1 (1.9MB PDF).
Companies choose virtualization for efficiency reasons, sacrificing some stable, known level of performance from their systems. In many cases that trade-off isn't a problem as we have many, many systems that are using only a fraction of their power. However as data professionals, we are often very concerned about the possible complications from virtualization. It's our phones that ring and each of us that gets the blame when systems are not as responsive as users would like.
This seems to be an ambitious undertaking from the TPC, and I suspect that more than a few hardware and software vendors will be nervous about submitting their wares for evaluation. The very nature of virtualization would seem to imply that as the load increases, the performance from any particular VM might vary from test to test. I will be curious to see how they present these results and how we can interpret them.
I don't know if the benchmarks will have any relation to the real world. As it stands today, the TPC results don't seem to relate to actual systems in the real world, though they do confer some bragging rights for platforms. I've enjoyed seeing SQL Server in the various rankings, if for no other reason than to show it can perform at the same level as other RDBMS's.