Dear All, i’m in the process of setting up a 4x way test between laternative Content Delivery Networks (CDN’s). What should i be testing/examining for?
eg
[list]
[]time to first byte
[]full page load
[]above the fold load
[]mobile / tablet / desk top
[]page size
[]compression: css/img/html
[]image size
[]3rd party plugins
[]cache hit
[]query strings (ignorred or processed)
[*]1st view
[/list]
In terms of speed, I look at:
Much of the things you mention are independent from which CDN you use (e.g., image size, compression, plugins). By the way, a more honest assessment of a CDN is probably performing a WPT against an asset hosted by the CDN - so not the actual website itself.
Then you can rule out variables that are outside of the influence of a CDN (like time to first byte, DNS lookup of the webpage).
thanks , TTFB was in there but totally missed DNS lookup which is a great suggestion.
The metrics we are looking to measure will include image size and compression as we’re engaging advanced facilities bundled with the CDN’s.
Interesting idea to test performance against a specific asset - to cut down variables.
Lots of bits to reflect on thanks.
I’ll second JMTC’s recommendations for metrics – I like TTFB and time-to-render using WPT. The main bit of advice I’d have is general stats work: averages are exceptionally misleading – one slow sample will seriously skew your results – so you want to make sure that you get at least a few percentiles.
I typically look at 50th, 75th, 90th and 98th and, where possible, try to ensure that those tests include both cache hits and misses. In my experience, most CDNs are in the same ball-park for cache hits – modulo regional connectivity, of course – but you can see more substantial variations for cache misses and those are going to be the most painful user experience (e.g. a miss on an HTML page which spends 5+ seconds in the CDN’s network).
I would add everything related to SSL, SPDY and HTTP/2 (including length of certificate chain). Also response size in bytes ( from additional headers ) and TCP congestion window.
Last but not least - distance of CDN node from your website users (if they are clustered around city / state ).