Chrome 33

WebPageTest Private Instances
1

Has anyone tried Chrome 33 with the wptdriver in 2.13 yet? I’m seeing some very significant regressions in Chrome 33. This is a histogram over 350 runs on Chrome 32 and Chrome 33 on one of our pages in production.

Runs were executed at the same time from an IE9 WPT AMI and an IE10 WPT AMI running WPT 2.13. Both in EC2 East Virginia.

Average deltas for 32 vs 33

Document Complete  Fully Loaded  Start Render      Visual Complete  Speed Index
490 (26%)	      634 (15%)     -413 (-28%)       181 (4%)         147 (11%)

Do you think this might be problems with the wptdriver or is 33 simply this much slower?

2

Try the latest wptdriver from here to see if it helps: https://sites.google.com/a/webpagetest.org/docs/private-instances#TOC-Updating-Test-Agents

What size were the instances (small or medium)?

Chrome 33 also introduced some resource scheduler changes that should be a lot faster in the general case but there may be situations where it made things slower.

3

medium instances. Thanks, I’ll try this out!

4

Chrome 33 wptdriver 166 had similar perf regressions, but a little better than wptdriver 148.

Chrome 33 to Chrome 32 (positive is slower)

Document Complete  Fully Loaded  Start Render      Visual Complete  Speed Index
416 (22%)          -231 (-6%)       154 (12%)      10  (0%)         142 (10%)

DomLoadedStart was significantly faster -460 (-32%)

Chrome 33 wptdriver 166 to 148, got faster in some metrics but not back to Chrome 32 speed.

Document Complete  Fully Loaded  Start Render      Visual Complete  Speed Index
-67 (-3%)          -871 (-19%)       -8 (-1%)       -165 (-5%)       -6 (-1%)

This test was run with >300 runs on the 148 and 166 driver versions.

This driver is not compatible with the IE8/9 instances we are running. Tests against chrome is no longer complete on those instances.

5

Sorry, there was a bug in the driver that got introduced earlier this week that I just fixed (not related to performance, the reason it wasn’t working at all on the EC2 instances).

6

No problem, you made downgrading/upgrading drivers really simple and I appreciate that.

I’ll do another performance run on this driver just for kicks and giggles though.
[hr]
Tests are starting and never completing in Chrome on the 168 version.

Firefox/Chrome still appear on the webpagetest page for manual testing, but getLocations isn’t reporting any browsers that use wptdriver.
[hr]
Screenshot of wptdriver on ie8/9 AMI