Revisiting browser and connectivity defaults

WebPageTest Announcements
1

WebPagetest provides a bunch of browser and connectivity options but by far the most testing us done using whatever the defaults are set to. It is a pretty big deal to users if they change which is why I try to do it when only absolutely necessary but I think it’s time to re-visit what the defaults are and at least have the discussion if it’s finally time to change them again.

Browser

The last time the browser default was changed was in May 2011 (close to 2 years ago) when it was changed from IE 7 to IE 8. IE 8 is getting pretty long in the tooth and market shares have changed a LOT in the last two years.

At least according to statcounter, IE 9 has eclipsed IE 8 in every region except for Asia and outside of the US Chrome and Firefox both have more usage than IE:

US
Europe
Asia

Global

IE 9, Chrome and Firefox are all much more modern and comparable with each other than IE 8.

Akamai also provides browser usage share as part of their IO data set which is reporting much higher shares for IE.

I’m leaning towards switching to Chrome for the default but I could probably be convinced to use IE 9 instead (though with the auto-upgrade to IE 10 coming shortly there is a good chance that the IE mix will get murkier).

Connectivity

The default connectivity has never changed since WebPagetest’s launch back in 2008. 5 years is quite a long time and it has done us well but it is becoming increasingly difficult to justify. Getting good data on broadband speeds and adoption has been really difficult.

Akamai makes a bunch of data available regularly as part of their state of the Internet reports. The Q3 2012 report calls out the percent of users with > 4Mbps connections (62% in the US, over 80% in Asia and in the 70’s for Europe).

Some other data I have been looking at also shows significant bandwidth increases across all percentiles, particularly over the last 2 years.

At 1.5Mbps (the current default), the majority of pages that I have seen are bandwidth constrained and any improvement basically requires reducing bytes (not a bad idea but may not move the needle for a lot of users that are not bandwidth constrained). Given the growth in sizes of pages, maybe it’s a good idea to keep the focus there but it feels like it should be time to re-consider.

I’m inclined to switch to the “Cable” profile which is faster and with lower latency (5Mbps and 1/2 the latency) which should be a lot closer to the current medians (and more forward-looking) but I’d love to hear input.

Any changes will just be to the default settings, all of the options will continue to be available so the plan isn’t to take anything away, just to update the defaults for where we steer people to optimize.

2

I couldn’t agree more with you. We were just talking about this yesterday, and how it would be better to indeed change default to Chrome and Bandwidth to ‘at least’ the current Cable setting, maybe with an even higher download.

  • C
3

I’m not too concerned about the default browser choice I would be tempted to go for IE9 given the market shares in North America but aware Chrome has a lead elsewhere - in traffic terms where do most of WPTs users come from?

Bandwidth wise I think the cable settings are too optimistic - many people have faster connections these days but are they really achieving faster throughput?

I was quite disappointed when I saw the Netflix stats (http://ispspeedindex.netflix.com/) but aware they’re an average rather than a distribution (know anyone at Netflix who can give us 50th, 90th percentiles?).

4

Definitely agree with changing the default connectivity to Cable.

As for browsers I’m a bit undecided. I pretty much always test IE9 and Chrome. I think it’s more likely that people find performance issues when they test with IE9 though (although yesterday we found an issue with Chrome’s speculative downloading that resulted in poorer performance than with IE9…). Either one would be fine though.

5

I agree with the change:

  • At least move to IE9
  • Bump to CABLE profile
6

Maybe Cable is too optimistic, however, I’m referring to http://www.netindex.com and looking at their stats, the average is higher than the default Cable, that’s why I suggested and even higher download.

7

I’d choose for cable and IE 9

8

It would be good if users could save their own default. But Cable & IE9 as the default sounds good to me

9

The Akamai data and other data I have seen are as-measured so it’s safe to assume that’s actual thruput and not just advertised or subscribed speeds. I’m less confident in the Netflix data (Google Fiber is 3Mbps, seriously?) I think they are limited somewhat by the bitrate of the encoded movies depending on how they measure or there’s congestion somewhere else.

The bulk of the users doing testing are from the US and Europe and I can use different browser defaults by location and tune them for the different regions but the Dulles default is the main one I want to make as neutral as possible since the vast majority of users just drop in an URL and hit test.

IE 9 will probably be the most seamless change since it uses the same agent code as IE 8 and there are no missing features (like the PageSpeed score being missing for Chrome for example). There’s a chance I’ll have to re-visit it again in a couple of months but Chrome and Firefox change on their own every 6 weeks so it’s still not that bad.

The more data I look at the more comfortable I get with switching to the 5Mbps cable profile as well though I still worry for the sites that do no testing at lower bandwidths and shove 4+MB of data down to users on each page view. I don’t think that necessarily outweighs getting people to optimize for more modern browsers though so maybe there are other things I can do to alert them of that kind of issue.

10

Chrome and Cable.

11

I don’t know how many people test from multiple locations but I wonder if having different defaults for different locations is likely to create more issues than it solves?

12

Maybe it’s an idea to add a line or table to the final results page with a calculated (rough) estimated load time for the other line speeds. I’m sure you have enough data on the line speeds to make some pretty close estimate calculations. Even when people just tend to test only on speed, the default or not, you still make them aware in the overview of the (estimated) other line speed load time.

Adding ‘warning colors’ for page load times higher than whatever currently ‘the golden rule’ is, might get that 4+MB warning and awareness job done better. I know from my experience with customers, that most people are very motivated to get most of those colors on the WPT page as green as possible, even in the EU, where F->A doesn’t tell them that much (We’re more 1->10 kinda people :wink: )

13

I agree about the default’s being cable & maybe for the browser you could set a cookie to the last used. I would be intrigued to see some mobile speed settings on there maybe 3G / 4G setting’s would be useful to some.

-Sean

14

We should look into the native broadband speed setting some more. Xfinity is talking about doubling their speeds. So fiber/cable-modem is getting even faster. Probably due to improvements and price drops in fiber transceivers, as well as networking backbone equipment moving to 10gpbs and in some cases 100gbps. People are also using triple-play (voice, video & data) more often. They are also using their Internet connection for HD video now. I guess that’s a whole other beast though.

15

I’d like to see the default connectivity move to cable as well, and I’m pretty neutral as far as the IE 9 vs. Chrome debate goes. I certainly think that it should be one of those two instead of IE 8.

16

ok, I haven’t seen any opinions or data opposing it so I’ll be switching the defaults to IE 9 and Cable over the weekend (just upgrading some VM’s right now so there is enough IE 9 capacity). Thanks everyone.

17

Quick question, I’ve been running tests pointint to stage.mint.com for a couple of weeks now and it seems that today around 5pm Pacific time the outcome of the test changed, apparently some contents I preload from the login page are getting reloaded (not cached) on the page after login and the second refresh seem also to start from scratch.
Not sure how it would be related to this change, any idea?

If you dont have much time to spare, no worry i’ll try to look into it.

Thanks

18

Perhaps I am not using the REST Api correctly, but it appears that the speed variation that we had it coded to (in case the defaults changed) has been overwritten. It looks like all of our pages load MUCH faster than previous runs with this morning runs.

This TERRIBLE page here:
http://www.webpagetest.org/result/130315_8W_M0R/
Normally takes around 13 seconds to completely load the page with 2.* mb of data.

With this morning’s run it was less than 4 seconds.
http://www.webpagetest.org/result/130316_Y0_FAV/

While we will likely revisit the params we’re using down the road, the unexpected change in speeds will throw off our data. Which is why we used the hard coded values on the API rather than depend on defaults.

I’ll look into this more tonight when I get a chance to see if something is not set right for our calls to your service.

FWIW I agree with the settings you are defaulting to, though I might put Chrome ahead of IE9 just because of the number of webkit related versions. Well, at least for our customers.

Thank you,
Mike

19

Do you have before and after test URLs? It’s easy enough to just run a test on DSL with IE8 and see if the behavior reverts. My guess is that it’s more tied to the browser than the connectivity speed.

20

Looks like your query parameters are fine. I’m checking to see if I had a tester that didn’t traffic shape correctly (the latency numbers look like there was no traffic shaping whatsoever). I re-ran the test it it came out as expected: WebPageTest Test - Running web page performance and optimization tests... so I’m hoping it’s just a specific tester with an issue.