Principled Technologies Tests Windows Vista SP1

re: Principled Technologies Tests Windows Vista SP1

Ceinach — Fri, 07 Mar 2008 19:00:07 GMT

Nick,

After doing some additional research and talking with a number of other consultants and companies I work with on a daily basis, I've got to say I am absolutely now convinced that thess test results are absolutely bogus. Not a single person I talk to agrees with these test results. Simply put, they don't reflect real-world usage at all according to everyone I know.

I also found and interesting note on another Windows Blog that you wroteas well regarding other tests that Microsfot has performed regarding Vista. The developer of the testing software HIMSELF wrote in that blog the following regarding Microsoft's testing procedures:

"Nice "hit piece," Nick! First you generalize about benchmark "purity," then you do a hatchet job on OfficeBench without so much as a shred of technical evidence....As the author of the OfficeBench test script I take personal offense at your cavalier handling of what is a very real and pressing issue for enterprise IT organizations."

http://windowsvistablog.com/blogs/windowsvista/archive/2007/11/30/the-right-time-to-assess-windows-vista-s-performance.aspx#comments

Simply put: I don't believe you anymore Nick. The testing company doesn't believe you (which is probably why you wrote the test scripts yourself for this one). I don't believe these tests were in any way carried out in the best interests of your customers. I believe the testing scripts and procedures were optimized to show Vista in the best light possible. That was the sole purpose of the test...not an honest assessment of the state of Vista versus XP performance.

This will probably be my last post on this as I can see I'm simply wasting my time trying to understand this process. It doesn't make any sense.

re: Principled Technologies Tests Windows Vista SP1

Mondoman — Thu, 06 Mar 2008 22:16:07 GMT

"A machine was excluded (though as you note, the raw scores remain part of the report) due to highly variable results. The results were not consistent in test after test"

I also take exception to this. If you view the other test results, they too show high variability. In any case, out of 5 machines it just happened to be the one thrown out? Too conincidental for my tastes and smacks of poor testing procedures.

re: Principled Technologies Tests Windows Vista SP1

Mondoman — Thu, 06 Mar 2008 21:56:03 GMT

"And finally, the original tests were designed before the responsiveness benefits of SP1 update were known, or even planned, so there was no way to skew these new tests to highlight the benefits of SP1."

This may be correct, but you've carefully avoided mentioning if those tests were designed to promote Vista in general over XP. From the pdf, it appears that the tests were in fact designed by Microsoft, not PT. Microsoft also supplied the hardware and sponsored the test. This is NOT an independant test and is the reason why responsible and trustworthy testing (like Consumer Reports) does not stoop to levels like this to try and skew test results in favor of Vista over XP.

In any case, it would be great to understand how PT got such great results on such minimal hardware on Vista. I, like other posters, have never had any success with Vista running less than 2 to 4 gigabyte of memory modules. I'm extremely interested how PT managed to pull this off with half the resources and truely have arrived at an independant and comprehensive test. Perhaps Microsoft can convince me to reload Vista on my laptop if only I could understand how my performance magically triples under XP versus my 6 months of Vista disappointment.

re: Principled Technologies Tests Windows Vista SP1

Korn1699 — Thu, 06 Mar 2008 15:01:28 GMT

Don't you need a new Mac for that..? I have Vista on a machine that is 5 years old..

Ubuntu takes up a lot of resources too if you turn on all the effects, the way Vista is by default.

re: Principled Technologies Tests Windows Vista SP1

tjmk — Thu, 06 Mar 2008 11:26:38 GMT

nobe no we love MAC vista is fake os. sorry microsoft, but vista is a sink boat (collapse)

MAC run

Mac OS X v10.5 Leopard

Add a new Mac to your Mac. Mac OS X v10.5 Leopard is packed with over 300 new features, installs easily, and works with the software and accessories you already have.

http://store.apple.com/1-800-MY-APPLE/WebObjects/AppleStore.woa/wa/RSLID?nnmm=main&mco=EC571521

re: Principled Technologies Tests Windows Vista SP1

pworthy — Thu, 06 Mar 2008 07:55:20 GMT

Well said MikeJay. I'm curious and have a question for the folks here. Has anyone actually met a real person (not just anonymous posters, but someone in the flesh) who likes Vista?

I'm an IT Pro. I deal with dozens of small businesses. Since Vista has been released I have talked with probably hundreds of people who have had it forced on them when purchasing new systems. As well as dozens of people who had the misfortune of upgrading to vista.

Seriously, not a single person I have spoken to has had a good experience. They have all had serious problems and the most common question I am asked about Vista is how to remove it. Dozens I've spoken to tell me that their new systems are just sitting and gathering dust because they are unusable.

Recently, because of the lawsuit against Microsoft, even the top executives at Microsoft have admitted that their own personal systems were unusable. So how does the average user fare? I particularly enjoyed the comment by Mike Nash, vice president for Windows product management.

"I know that I chose my laptop (a Sony TX770P) because it had the Vista logo and was pretty disappointed that it not only wouldn't run [Aero], but more important wouldn't run [Windows] Movie Maker," Nash said in an e-mail on Feb. 25, 2007. "Now I have a $2,100 e-mail machine."

So again, has anyone here actually met a real person who likes Vista?

re: Principled Technologies Tests Windows Vista SP1

MikeJay — Thu, 06 Mar 2008 03:51:47 GMT

"it's important to remember that Windows Vista is a more advanced operating system and includes many more capabilities than Windows XP. By default, Windows Vista includes antispyware, a search indexer and the shadow copy service."

Nick, The anti-spyware and anti-virus I can get for free from AVG, Goole has a great seach indexer (among other free options), and XP already have shadow copy services built in...except it's called VSS and works great with NTBackup. Of course I can't restore them or any of my previous 3 years worth of backups in Vista without downloading a separate tool since Vista natively can't read the standard .BAK format either.

Let's just say that antispyware and indexing hardly make the case for consuming 2 GB of memory.

"All these work in the background, providing important services to customers, but they of course take up a portion of the PC's memory and processor resource."

Really? Why? Why, for example should the search indexer be running in the background, constantly, taking up memory and processor? No, really, why? It's taking up valuable resources and accomplishing nothing because either: a> I'm trying to use the computer or b> there's nothing to index. Why should I care one bit whether I have a properly indexed version of the Word document I just finished writing 15 minutes ago? I'm never going to need the indexer indexing constantly. Isn't this why services can be started and stopped by other system processes? Isn't Vista intelligent enough to handle this without just running everything, including 80% of the crap I'm not using at the time I'm trying to get my task done.

In short, more is DEFINATELY not better. The ideal OS would be the OS that nobody notices. It gets out of the way and prioritizes the applications, not itself. Vista is too narcissistic for it's own good. It spend too much time looking in the mirror, primping and preening itself for naught. It's NOT spending enough time getting it's work done because it's got exactly what you siad, service after service after service running, checking, rerunning, flagging prompts, checking devices, etc. It's so complicated and convuluted it keeps tripping all over itself. It does indeed, look great however.

re: Principled Technologies Tests Windows Vista SP1

Nick White — Thu, 06 Mar 2008 01:23:20 GMT

Hey everyone:

Let me provide you more context around our engagement with Principled Technologies and the tests they conducted: We chose to contract Principled Technologies for benchmarking of Windows Vista RTM and SP1 because they provide trusted independent analysis to the high-tech community. Further, Principled Technologies follows a strict and carefully documented methodology, thus maintaining their integrity as a benchmarking firm. We would expect that other benchmarking labs conducting similar tests will find similar results.

You can find a partial list of their clients here: http://www.principledtechnologies.com/Clients/Clients.htm.

In order to keep the results comparable the same machines and the same procedures were followed. A machine was excluded (though as you note, the raw scores remain part of the report) due to highly variable results. The results were not consistent in test after test. This is not unusual in compiling benchmarking results (or any statistical analysis where "outliers" are eliminated). The average of the excluded data can be found in the appendix of the report.

And finally, the original tests were designed before the responsiveness benefits of SP1 update were known, or even planned, so there was no way to skew these new tests to highlight the benefits of SP1. In fact, as you've noted, the responsiveness gains are modest with SP1 under these test criteria. Windows Vista SP1 remains on par with Windows XP SP2, just as the initial tests with RTM code revealed.

While the Principled Technologies tests show that - as with Windows Vista RTM - overall performance is on par with Windows XP, it's important to remember that Windows Vista is a more advanced operating system and includes many more capabilities than Windows XP. By default, Windows Vista includes antispyware, a search indexer and the shadow copy service. All these work in the background, providing important services to customers, but they of course take up a portion of the PC's memory and processor resource.

re: Principled Technologies Tests Windows Vista SP1

MikeJay — Wed, 05 Mar 2008 23:03:29 GMT

Brandon,

In fact, right now he's only listening to an MP3 file and running Outlook 2007 and the system has used 1.880 GB of memory already. Running IE will push this over 2GB.

Vista is using 2gb of memory to play an mp3 file, read e-mail, and browse the Internet. We also just disabled his anti-virus. This can be accomplished faster on my XP laptop system with 1GB of memory and a Duo-core without disabling anything.

Vista is simply a pain in the ass with little/no ROI value. You should NOT need a quad core w/ 4gb and dual SATA drives to achieve acceptable performance from ANY OS. XP doesn't require this level of hardware. Neither does OS-X or Ubuntu. Bloat, bloat, and more bloat is what is going on IMHO.

Just for fun Branon, you should dual-boot an XP load off the same hardware you have. I'll bet it would be so fast you wouldn't even see the windows XP blink open. There is no reason you should be satisfied with the performance you are getting out of Vista if you compared it to XP.

re: Principled Technologies Tests Windows Vista SP1

Brandon LeBlanc — Wed, 05 Mar 2008 21:52:34 GMT

MikeJay, I am running a quad-core Dell XPS 420 with 4GB of RAM and 2 SATA drives and I can tell you that it does not take no where near 2 minutes and 53 seconds for my machine to "cold" boot. It probably takes less than half that. And I'm running Vista as-is with no services off or anything. I find it odd your friend's PC takes that long to boot on a PC with those specs.

- Brandon

re: Principled Technologies Tests Windows Vista SP1

Korn1699 — Wed, 05 Mar 2008 19:54:43 GMT

On my machines, XP ran the same OR SLOWER after my memory increase due to it not needing that extra memory. When I went with the cheaper memory, I think the latency was higher, so that could be why that machine preforms slower with 2 GB of ram than it did with 1 GB..

re: Principled Technologies Tests Windows Vista SP1

pworthy — Wed, 05 Mar 2008 19:37:15 GMT

My 2 cents. Many people justify keeping Vista by saying that they get a "jump in performance" by increasing memory. No one seems to mention the fact that increasing memory also gives a huge performance gain to XP. So XP on that same machine would still outperform Vista.

In the last week alone I have had 3 friends with small businesses ask me to Upgrade (yes Upgrade) their machines to XP simply because many of their programs wouldn't work properly,their hardware was now buggy and they complained about speed. For instance on one system after installing a corporate vista version of a virus scanner, the install disabled windows defender. Said it wasn't compatable. Fine, but now he started getting nags. So he said he turned off the nags and his wireless stopped working. He checked the web and found it was a common problem so he turned them back on and the wireless worked again. Go figure. Every time he used a memory stick to back up his data, half his files and directories were deleted from the stick. After about a week and dozens of other problems he had enough. He just couldn't use it with his business. He is a real world typical user who needs something that just works and doesn't require his constant attention. He needed to focus on running his busIness not spend his day trying to get Vista to do what XP did without all the time and attention. That's why he went back to XP.

re: Principled Technologies Tests Windows Vista SP1

eponymousnyc — Wed, 05 Mar 2008 17:22:37 GMT

Benchmarks or not, I can't connect to an Xserve with anything close to the speed I could with XP, so it looks like I get to come in this weekend and roll my machine back to XP.

re: Principled Technologies Tests Windows Vista SP1

Korn1699 — Wed, 05 Mar 2008 15:44:52 GMT

Vista has been faster for me than XP on the machines that I have had both OSs, BUT that was with increased memory. I had Vista on my old laptop that had a 1.7 GHz Centrino and 1 GB of good memory. It ran slower than with XP until I increased that to 2 GB of cheaper memory, and now it is faster than when it runs XP. I also noticed a big jump in performance with Vista on my 5 year old desktop after I increased the memory from 1 GB to 3 GB.

re: Principled Technologies Tests Windows Vista SP1

MikeJay — Wed, 05 Mar 2008 14:31:11 GMT

I'm at my friends house right now. He has a quad-core desktop tower, a SATA drive, 4 gigabytes of memory, and whatever the latest ati card is that costs hundreds of dollars.

We timed it. From a cold boot it took 2 minutes 53 seconds to boot and log into Vista. After that it took at least 5 minutes before the hard drive calmed down.

He's an MCSE and works on computers for a living. That's the fasest he can make his Vista run right now and he's shut of necessary services, etc. That's reality. That's not whatever world Priciple Technology lives in.

It would be great if we could all write customized non-independant scripts that show our products in great light and get our jobs done in the timeframe Principled can. The problem is, I live in the REAL world, where pictures of Vista dancing in fields with pixies and only taking 30 seconds to boot are mere figments of my imagination.