After following several of the other resources for testing pages, I am completely confounded. Other sites show our site loading much quicker, even if you count the webpagetest.org repeat view with cached data.
There is an addon for Firefox and Chrome called apptelemetry that is showing our site loading in 1.45 seconds (may be cheating since I am internally connected to the network). However, the other sites are all recording much faster load times.
Can someone help me understand why webpagetest.org is so much slower or if there is an accuracy problem with webpagetest.org? I very much want to get an accurate representation of what our visitors are experiencing.
WPT uses real connections, real devices, real browsers. Not algorithms or programs. I believe therefore it is really the ONLY objective, repeatable, falsifiable, truly accurate, REAL WORLD tester out there.
It also has the advantage that it’s not trying to sell us anything.
Do you happen to know if there is any checks when a testing location is set up to make sure that the times are reliable from that location? I am just needing to clarify as our IT department’s own software lists the times differently and I have an uphill battle explaining the times I am getting from WPT.
Do you happen to know if there is any checks when a testing location is set up to make sure that the times are reliable from that location? I am just needing to clarify as our IT department’s own software lists the times differently and I have an uphill battle explaining the times I am getting from WPT.
[/quote]Since it is real connections just like your visitors and users have, the vagaries of the web come into play. This is why when I use WPT as a tool to optimize sites, I test from only one location and one browser.
Location: Dulles VA (Because I have found it to be the most consistent and reliable device/connection, and it is Patrick Meenan’s own personal setup.)
Browser: IE11 (Because this is “worst case” it counts ALL of your requests, doesn’t hide any like Chrome does.
Explain to your IT eggheads that WPT is real world, not virtual like everything else you have mentioned. And have them google “Patrick Meenan.”
More often than not, timing differences will come from the type of connectivity selected for testing. WebPageTest defaults to a 5Mbps cable connection (traffic-shaped) while most testing services will test from the backbone (1Gbps, no latency). If you change the WebPageTest connection type in advanced settings to “Native Connection” it will usually get a lot closer to what you see in other tools.
Remaining differences tend to come down to browser and the physical location where testing is done from. WebPageTest runs the actual browsers and just watches what they are doing. The different commercial tools sometimes implement browser-like fetching or parts of browser engines and you can sometimes see pretty significant differences in the waterfalls.
Thank you very much. I added another script to run “Native” to see the differences. I immediately saw a closer number to what my IT department is seeing from Google Analytics. However, now I am more concerned than ever that we are just turning a blind eye to real world problems and high-fiving successes based on numbers that are not realistic.
Is that safe to say about the “Native Connection” results? Testing without traffic is more of an “Everything is optimum utopia” result and running on the 5Mbps is more of a real world result?
For comparison to your analytics you may be best off looking at repeat view numbers, assuming some percentage of your population is coming in with a pre-warmed cache and may be bouncing from page to page in the same session.
“Native” is certainly best-case as far as connectivity goes but it is worst case as far as browser cache, particularly if you JS resources are long-lived and your users tend to visit frequently.
When looking at Analytics, make sure to look at percentiles/histograms. Averages just hide way too much information. I usually use SOASTA’s mPulse for RUM data because of the visibility into the percentiles, user timing metrics and a bunch of other stuff (free for some levels of usage).