Microsoft creates mini internet to reliably test and improve Internet Explorer

With the grandiose bluster that only an aging juggernaut can pull off, Microsoft has detailed the Internet Explorer Performance Lab and its extraordinary efforts to ensure that IE9 is competitive, and that IE10 is the fastest browser in the world. Here’s a few bullet points to reel you in: 140 computers, 20,000 tests per day, over 850 metrics analyzed, and a granularity of just 100 nanoseconds.

First some background: The average Windows user spends 50% of their time in a web browser. If the web browser is slow or temperamental it reflects poorly on the underlying OS, which 90% of the time is Windows. With Windows 8, Internet Explorer will be even more important because itpowers the Metro interface and any Metro apps written in HTML and JavaScript — and, at least in the US, it looks like IE10 will be the only browser available in the Metro interface, which is where tablet users will spend most of their time.

On another front, Internet Explorer and Firefox are rapidly losing ground to Google’s Chrome, whose success has hinged almost entirely on speed. Internet Explorer 9 has started to turn the tide on Windows 7, but it’s down to IE10 to continue the upwards trend.

With this in mind, Microsoft has built a 140-computer lab that tests the performance of Internet Explorer 9 and 10 before and after each change to the codebase. According to Microsoft, this means they measure IE’s performance 200 times per day, collecting over 5.7 million measurements covering 850 discrete metrics (TCP bytes received, GPU utilization, CPU time spent rendering, etc), and 480GB of runtime data. This data is then parsed by 11 server-class machines (16 cores, 16GB of RAM), and finally stored on a big SQL server (24 logical cores, 64GB of RAM). This data is then visualized (pictured right) and analyzed to see if the latest code changes have improved or worsened performance.

The setup

What about the other 128 computers in the lab, then? Well, it turns out that Microsoft’s IE Performance Lab is basically a mini internet. The Performance Lab is a completely closed network, disconnected from both the internet and Microsoft’s intranet. The 128 computers break down into the following categories: content servers (i.e. computers hosting websites), DNS servers, network emulators, and test clients. There is network gear as well, of course.

The idea is that it’s impossible to perform reproducible, actionable testing on an open network. When you make a minute change to the codebase, you don’t want it to be hidden by a router hiccuping half way around the world. The Performance Lab’s tools are accurate to within 100 nanoseconds — 0.0001 milliseconds — and so the tiniest of hiccups is enough to ruin a test. This is why the lab includes every piece of hardware and software that a “normal” internet would have.

The vast majority of the computers are test clients, which are broken down into high-, mid-, and low-range devices, spanning everything from 64-bit desktops, to Atom-powered netbooks, to ARM tablets. The DNS servers are simply DNS servers. The network emulators, however, are interesting. Basically, the Performance Lab has no variation at all. This is by design, and ensures that test results are actionable; if you run the same test on the same hardware, the result will be the same. Obviously the internet isn’t actually like this — and that’s where the network emulators come in. Network emulators can be tuned to inject conditions that real-world users might experience, such as latency and packet loss. Network emulators are, in effect, “the internet” portion of the Performance Lab.

If you thought Microsoft’s attention to detail was fairly impressive, there’s more! Before every test, each and every computer receives a fresh install of Windows (Vista, 7, or 8). If a test fails for whatever reason (a bad code push), Windows is reinstalled. Furthermore, if a piece of hardware fails, the entire computer is thrown out. Apparently, newer hardware is faster than older hardware — so replacing a broken stick of RAM with a new stick can throw off the entire test. When you are working at a granularity of 100 nanoseconds, every little detail counts.

The test

In essence, a Performance Lab engineer tweaks the testing scenario — the content delivered by the web servers, the latency on the network emulators, the local settings of Internet Explorer — and then simply presses a big red button, which triggers the installation of Windows on a test computer, and then hours of repeated web page fetching. As mentioned before, a total of 850 metrics are captured, each one measuring one of four benchmark categories: Loading content (from pressing enter to finishing rendering); interactive web apps (clicking through interactive, JavaScript elements on a page); synthetic benchmarks (SunSpider et al); and the application itself (is the “File” menu more or less responsive, does “Print” still work, and so on).

At the end, data is funneled back to the analysis and SQL servers for inspection.

A complete test cycle is a lot more complicated than this, but fortunately Microsoft has provided a flow chart of the process.

It’s all a bit over the top

To be honest, the Performance Lab feels slightly over-compensatory. I mean, Internet Explorer 9 is certainly fast, and IE10 will undoubtedly be very fast as well… but there’s more to web browsing than raw performance. By the numbers, Firefox is as fast as IE9 or Chrome, and yet it doesn’t feel as fast. Likewise, IE9 is theoretically very fast, but you have to remember that it doesn’t have add-ons, sync, or many other features found in Chrome or Firefox.

Just like when Microsoft turned to calculus to defend its murder of the Start Menu, the Performance Lab feels like the digital equivalent of Steve Ballmer breathlessly chanting developers, developers, developers; it’s impressive, and even a little bit scary, but not actually all that effective.

Another big question is whether Mozilla and Google employ the same testing methods. We know that Mozilla does performance testing of add-ons, but as far as we know they rely on Test Pilot for the browser itself. Google probably has a similar setup to Microsoft. We’ve reached out to Google and will update this story when (or if) it replies.

Read more at Building Windows 8 or watch a video about the Performance Lab

Source

The death of Firefox

It doesn’t look good for Firefox: Almost every month for the last three years, Firefox has lost ground to Internet Explorer, Chrome, and Safari. For most of 2009 the trend was fairly straight as it fended off Chrome and nibbled away at IE, but between 2010 and today Firefox has lost a third of its market share, from a worldwide peak of around 30% down to 20%.

You can look at this two ways. First, the total number of people on the internet is growing, so while Firefox’s share has decreased, the total number of people using Firefox is increasing. The other point of view is that Firefox, whether you like it or not, is declining in popularity.

I love the Fox as much as the next bearded geek, but the numbers just don’t lie: Chrome is breathlessly decimating Firefox’s userbase at a breakneck rate. It took Firefox more than four years to prise 20% of the market from Internet Explorer; Chrome did it in almost half that, and is fast approaching 30% in just over three years. Internet Explorer’s graph is a little harder to interpret, but it looks like it might have finally turned the corner and stopped hemorrhaging market share.

Compounding Firefox’s losses is the stark reality that it’s unlikely to make any gains. Google has obviously spent a lot of money advertising Chrome, but there’s no way that ads brought it nearly 30% of the web’s two billion surfers. People are migrating to Chrome because of word of mouth: Geeks and power users picked it up first, and they’ve been installing it on the computers of friends and family ever since. Microsoft, too, is using a dollar bill tourniquet, and when Windows 8 tablets roll around with IE10 as the default browser, you can be sure that its market share will climb. Mozilla is adding some exciting new features to Firefox, and Firefox for Android is an interesting enterprise, but I don’t foresee anything that will turn the tide.

But is that really a problem? The entire reason that Firefox was such a success is that it appealed to the geeks and power users who weren’t happy with Internet Explorer 6′s 95% share of the market. Microsoft effectively put the dampers on web innovation for five years. Firefox was conceived with one purpose in mind: To revitalize the web.

In that regard, it has succeeded. The web, with three browsers vying for supremacy, has never been more exciting. Within a few short years of launching, Firefox had shown the world what CSS and a gutsy JavaScript engine were capable of. Firefox triggered the HTML5 revolution. It is because of Firefox that Metro-style Windows 8 apps can be written in JavaScript. And ironically enough, it is because of Firefox that Chrome was created.

If you used Chrome in 2008 and 9, you will remember that almost all of its early adopters were disaffected Firefox users who had grown tired of an ever-increasing memory footprint and sluggish interface. Chrome had almost zero features when it first arrived, but it didn’t matter: When the only two choices were a slow Internet Explorer or a bloated Firefox, Chrome was exactly what the people (and the internet!) needed. Mozilla has spent the last year trying to trim the fat, but it hasn’t caused an upswell of users to return to the motherland. Much in the same way that Firefox cannibalized Internet Explorer, Chrome capitalized on just a single feature — speed — and has been riding the wave ever since.

Despite its ridiculous rate of growth, though, Chrome will eventually reach a zenith. The Big Three all have enough gravitas to ensure that no one browser has the power to monopolizingly choke the web. Even if a browser does get a little too big for its britches, Mozilla will always be there to knock some sense into the community — and, if need be, do the grassroots thing all over again.

What will happen now, assuming Microsoft and Google continue to barrage each other with their full arsenal of cannon, is that Firefox will gradually fade into a position of feared, revered veterancy — kind of like an aging grandfather who sits on a rocking chair in the middle of the World Wide Web with a loaded shotgun. It might even get to the stage where Firefox has to occasionally loose some crazy feature onto the web, just to remind everyone that it’s still alive. Eventually, if Microsoft really is serious about open web technologies and Google does no evil, Firefox might even die.

But, having completed everything you set out to do in life, is death really that awful?

[Image credit: Mike Smail]

Source

Windows Patch Tuesday: Duqu and BEAST among critical updates

Microsoft has just delivered a large Christmas present to IT directors and users alike in the form of its last “Patch Tuesday” of the year. Addressing bugs in Windows, Internet Explorer, and Office, the updates repair twenty issues including a fix for the dangerous Duqu worm that took the internet by storm recently. In addition, the browser exploit against SSL/TSL cookies (BEAST) is being fixed in this package.

Because Redmond is understandably tight lipped about critical updates, you can’t know for sure exactly what is being patched. However, of the fourteen security bulletins that Microsoft published today, seven deal with the Windows OS. Three of these are critical updates, and one requires a system restart. This leads forensic experts to believe that the Duqu exploit is being addressed since a forced reboot usually indicates a kernel level change in code. With Duqu threatening so many systems, it makes sense that this first critical update would be geared towards closing the hole in the system the worm uses.

As mentioned above, the proof-of-concept hack known as BEAST that was discovered by researchers Thai Duong and Juliano Rizzo last September is being defeated with this patch. BEAST allows an attacker to be able to decrypt user data stored in cookies, allowing them to gain access to restricted systems. There has been some debate about how much of a threat BEAST actually is, since a simple update to the latest version of SSL/TSL renders the hack useless. However the threat of the evolution of the process to overcome the update is enough to merit patching.

Because of the way the calendar fell this year, IT pros are getting the patch a bit later than normal. If you fall into this category, we wish you luck in getting your networks updated so that you can enjoy the rest of the holiday season!

Read more at Microsoft Technet

Source