Windows Vista Team Blog : Windows Vista, performance

The right time to assess Windows Vista's performance

Nick White — Sat, 01 Dec 2007 01:29:00 GMT

Measuring the performance of an operating system is a tricky thing. At the same time, it's the right and necessary thing to do, because performance is one of many criteria important to customers. Part of the trick of measuring performance is to time testing execution with the product cycle such that the results are as meaningful as possible for customers; this helps them make a better decision by making use of the full array of available information. As one example, about a year ago we commissioned a firm called Principled Technologies to conduct a study comparing Windows XP SP2 to Windows Vista RTM. That study found the performance measures of the two operating systems were within the same range for many tasks that home and business users frequently perform under real-world conditions.

My point is that we waited to conduct these benchmarking tests until Windows Vista had reached the RTM milestone in the product cycle, as this allowed us to provide our customers the most meaningful data available at the time -- the data most likely to directly affect their decision to upgrade to Windows Vista. We do a whole range of performance tests at every stage of the OS development process, but, as a general rule, we avoid sharing benchmark tests of software that hasn't gone RTM (i.e., final code). This explains why we have not to date published any findings of benchmark tests (nor commissioned anyone to do so) on performance improvements brought about by Windows Vista SP1. Publishing benchmarks of the performance of Windows Vista SP1 now wouldn't be a worthwhile exercise for our customers, as the code is still in development and, to the degree that benchmarking tests are involved, remains a moving target.

Aside from that point, let me also emphasize that there are a variety of ways to benchmark the performance of a PC. Different techniques can yield different results. Some benchmark techniques simply test PC hardware performance by running a series of tasks at superhuman speed. Such tests tend to exaggerate small differences between test platforms and consequently are used less frequently nowadays, replaced in favor of benchmarks running tasks at human speeds with realistic waits and data entry. Benchmarks that run at superhuman speeds often deliver results that don't tell the whole story. In fact, we made deliberate choices during the development of Windows Vista to focus on real-world scenarios affecting user experience, rather than focusing on improvement of microsecond operations imperceptible to the user. In addition, in Windows many operations can require additional processing time for work that is done for reasons that benefit the customer; these can include security, reliability or application compatibility checks conducted when a program launches. These operations may add microseconds to an individual application's launch that under real usage isn't perceivable to the human eye. When thousands such operations are strung together through automation, those few microseconds can have a cumulative effect on the benchmark result, causing performance to appear much better or worse than expected.

I've included below a video we captured depicting a "benchmark test" running a window-open, window-close routine at accelerated speed. You can see that it isn't representative of real-world user behavior and hence isn't an accurate gauge of the actual end-user experience. Further, tests like these only measure a very small set of Windows capabilities and so aren't representative of the user's overall day-to-day experience of working with Windows and running applications.

Video: Windows Vista benchmark testing

Methods like those of Principled Technologies that actually approximate the experience of using the PC, taking an OS through the paces of completing actual tasks at the approximate pace a user might click through them, tend to provide results far more useful to our customers. The typical Windows customer generally wants to know how his/her actual computing experience will change (read: improve) with an upgrade. The Principled Technologies tests do that.

For what it's worth, I can personally attest that I prefer to get my work done on Windows Vista SP1 RC bits. I run Windows Vista RTM on two production machines and SP1 RC bits on two others; in fact, I'm writing this post on a machine with SP1 RC bits installed. As a part of our internal SP1 testing program, I know that we continue to develop and improve SP1 every day, in large part based on feedback and bug submissions from external an internal Beta-test program members. IMO, the perceived gains in performance between SP1 Beta and SP1 RC code are significant. As I said at the beginning, though, performance is only part of the story -- don't forget that SP1 also brings support for new types of hardware and several emerging standards, and further eases an IT administrator's deployment and management efforts.

But don't take my word alone for it. We'll broaden the testing pool of SP1 RC bits soon (very soon), so when I post that notice here on the blog, you'll be able to put Windows Vista SP1 RC through its paces yourself. I think you'll find the experience worthwhile and satisfying.

Aero and battery life

Nick White — Mon, 14 May 2007 22:00:00 GMT

Dwight Silverman, the tech reviewer at the Houston Chronicle, has written about the impact that the Aero theme has on battery life and performance (under the provocative title "Vista, Aero, battery life . . . and Doom" -- he's referring to the game Doom3, BTW). It's a good post and I'd like to take a minute to expand on what Dwight has written.

First off, yes, as Dwight correctly points out, the Aero theme drives the GPU harder and therefore uses more power. But in the big picture, it's really not that much more. For example, the display on most laptops will consume somewhere between 15-25% of your "power budget" when you are running on battery. Nevertheless, in our testing we've seen that turning on Aero consumes only about 1-4% more of battery life. In terms of making your battery last longer, turning off Aero will not go very far while at the same time costing you some of the cool features that make Windows Vista fun to use, such as Flip 3D, taskbar previews, window transparency and so on.

Further, as Dwight correctly points out, your mileage can vary widely depending on the workload the machine is running (he uses the example of playing Doom3.) To use an Aero-specific example, you would use more power enabling transparent window borders and stacking a lot of windows upon a portion of the screen showing a video clip. The GPU would be required to constantly re-render those parts of the screen that make up the transparent window edges, which in turn drives the GPU harder.

Now, we know there are times when you want to fully maximize battery life and every little bit helps. It's for this reason that we automatically turn off things like window transparency when the machine is put into a power-saving profile. We don't turn off Aero wholesale because in the end, doing so is not going to save you much more power. So we turn off the transparency effects and maintain a smooth user experience. We know that it results in a tradeoff, but we also think it's a fair one to make.

P.S. As an aside, I recently learned that testing the impact of software features on battery life is exceptionally tricky, even by benchmarking standards. A huge number of variables must be controlled for. In fact, something as simple as the way a battery is handled can introduce enough variability into testing to make it difficult to identify the true cause of any observed changes in performance or battery life. This is because the charge a battery requires can vary considerably based on, among other things, the battery technology (Li-poly, Li-ion, Ni-MH, etc.), how recently the battery was power-cycled, how old the battery is, and even the temperature of the battery when it was charged. Wow.

Windows Vista Hardware Assessment v2.0 Beta Program Now Open!

Baldwin Ng — Wed, 18 Apr 2007 04:28:00 GMT

Good news! The Windows Vista Hardware Assessment v2.0 Beta Program is now open for registration! Please join our beta program and test drive new features including OU-targeting, machine names import, localized UI/readiness reports and more.

Visit the Microsoft Connect site to join: http://go.microsoft.com/fwlink/?LinkID=88282

For those of you who are not familiar with Windows Vista Hardware Assessment, it is basically a network-wide PC readiness assessment tool that helps organizations understand their readiness to migrate to Windows Vista. It provides organization-wide reports on hardware compatibility, device compatibility, upgrade recommendations and more. The coolest part is that this tool does not require any software agents to be installed on individual computers.

Visit our TechNet Web Page and download the tool (v1.0) today at: http://go.microsoft.com/fwlink/?LinkID=83405

Baldwin Ng (baldwin.ng@microsoft.com)

Sr. Product Manager, Windows Vista Hardware Assessment Solution Accelerator

New team blog: Client and server performance

Nick White — Thu, 28 Dec 2006 00:41:00 GMT

Back from an all-too-short holiday with a quick note on a new blog started by the Windows Client and Server performance teams: the Windows Performance Blog.

According to Matt Ayers, a Program Manager from the Client Performance Team and the inaugural contributor to the new blog, you can expect the team to address performance-enhancing features and technical programming tips, as well as general thoughts on responsiveness in Windows. Whether you’re a technical enthusiast, hard-core benchmarker or just want to learn a bit about how we improve Windows 'perf,' head on over, have a look around and leave them a question or comment.

Enjoy your New Year's celebrations, and be safe!