Best way to get started and collect data

[israelda]

Hi All

We’ve been talking about speed at our company for a while now, and are at the point where we want to move from just discussing it to actually gathering data and settling on goals. The plan is to collect data for the four pages we want to measure and then decide what whether we’re happy with those numbers or if we want to work on bringing the time down. My question is, what’s the best way to run these consistent tests for a couple weeks and then work off of that? What automation options are there? Basically looking for the best way to get a baseline. Any feedback as we’re getting started would be greatly appreciated!

Is

[akshayranganath]

It depends on the kind of measurement you want to run to get the metrics. You can use synthetic monitoring like scripted WebPageTest or more productized tests like Catchpoint / Keynote or Gomez. For your use case, just using a scripted WebPageTest would work. You can write tests for the 4 pages of interest across different test agents. Each test run should access the URL at least 5 times and your tests should run at least once every hour or half hour for a day. You can then use this data for benchmarking. Once you gain confidence, you could explore the paid product options.

Another alternative would be to collect the data using real user monitoring (RUM) solutions. These provide you real numbers from real browsers. You can then slice and dice around the browser, OS, geo and more. You could potentially use a RUM solution like Boomerang but you’ll need to setup infrastructure to receive the beacons that are fired from browsers. Alternatively, you could use products like mPulse or New Relic.

(Disclosure: I work at Akamai and mPulse is a product of Akamai)

[israelda]

@akshayranganath

It depends on the kind of measurement you want to run to get the metrics. You can use synthetic monitoring like scripted WebPageTest or more productized tests like Catchpoint / Keynote or Gomez. For your use case, just using a scripted WebPageTest would work. You can write tests for the 4 pages of interest across different test agents. Each test run should access the URL at least 5 times and your tests should run at least once every hour or half hour for a day. You can then use this data for benchmarking. Once you gain confidence, you could explore the paid product options.

Another alternative would be to collect the data using real user monitoring (RUM) solutions. These provide you real numbers from real browsers. You can then slice and dice around the browser, OS, geo and more. You could potentially use a RUM solution like Boomerang but you’ll need to setup infrastructure to receive the beacons that are fired from browsers. Alternatively, you could use products like mPulse or New Relic.

(Disclosure: I work at Akamai and mPulse is a product of Akamai)

Thanks for the reply! The key point you made about running the webpagetest every hour for a day is what I was looking for. We have speedcurve running once a day and those number are so inconsistent - there’s no way we would have seen a 500ms savings. Will also revisit RUM - never had it, this might be the time. We’re akamai customers - will take a look at mPulse.