What do you want to see next?

Feature Suggestions
21

There’s a documented automation API available now (essentially what the form was doing anyway): http://www.webpagetest.org/docs/Automating_Pagetest

Beacon support isn’t there right now but it shouldn’t be hard to add. Was going to look at supporting having it send up HAR files and raw results so that falls right in line with what you are talking about.

Also on the list to consider is a way for doing recurring tests (at a minimum for the private installs but maybe a fixed set of pages on the public site as well) which combined with ShowSlow would give you interesting trending reporting. I’ll ping you offline because I’m working with someone on possibly standardizing the various interactions which would allow for even more creative mashups.

22

All sounds very exciting, let’s work on that.

Meanwhile I’ll look into PageSpeed automation to attach it to ShowSlow in some way.

23

Hey there,

First off, I really love the idea of automating page tests. It looks like you have a great start for doing just that.

Currently, when I run a test, I look over the results and pretty much never come back to it ever again - at least no more than a week.

I would love to see hourly, daily, weekly, monthly, and yearly trend data for a particular webpage or even an entire website. I realize this could be done manually but it would be very time consuming to do - especially when the entire process can be automated.

The first thing I would focus on is getting the testing process using a database instead of files. This way you have better control over the data.

The test history page could be setup a bit differently. You could have unique URL links (or titles such as My Homepage) on this page. Click one of these links could open up a new page that lists dates and times that tests were run for this particular URL. You could have links here to view hourly, daily, weekly, monthly, and yearly trend data if enough data exists for these particular URL. You could also have a link here to run test now so the testing data is already propagated into the test form.

On the actual test form, you could have a drop down box thats labeled something like “Run Test” and then “Once, Hourly, Daily, Weekly, Monthly, or Yearly”. You could have this option only available to registered members so it would encourage people to register.

If a test fails to run successfully for various reasons, you could have an automated email setup that informs the website owner that something went wrong with the test. This could be an indication that maybe their website is currently down and they need to fix it.

For the trend data, you could have flash graphs where data points are represented by circles. Clicking on a circle could goto the test results for the corresponding point.

Let me know what you think.

Sincerely,
Travis Walters

24

I’ll be adding trending support at some point but I’m not sure if it will ever see the light of day on the hosted instance here (more for use in the private installs). The main reason is the test system capacity. Doing recurring tests hourly for a large number of sites would consume a significant amount of the testing infrastructure. I may do something where I have a set of industry sites that are trended and may do something where I allow registered users a couple of pages but I haven’t figured out the details yet.

It’s also a space where there are a fair number of commercial offerings (Keynote, Gomez, BrowserMob, WebMetrics, etc) - some of which have free offerings to an extent.

Internally at AOL we already use the guts of the WebPagetest infrastructure for exactly that purpose so I’m pretty familiar with the different pieces that would have to be covered for a useful offering but we also have hundreds of test systems and a fairly large database cluster for running it on (and even there we gather exponentially more data than people end up actually using).

Thanks,

-Pat

25

I agree with Pat, doing the automated testing is not cheap and can’t be done just like that.

On a second topic, ShowSlow is the open source solution intended to be this source for recurring data keeping track for you and that’s why I’m trying to work with Pat on integrating Pagetest.

Same problem though - ShowSlow.com (public version) can’t run infinite runs of tests for you for free, but both products are open source and can’t be installed locally (and many people do).

I don’t know if Pat has an intention to provide paid service as part of WebPageTest.org, but ShowSlow is definitely going to - this might be a solution to your problems.

Sergey

26

Hey there,

Pat, I never knew you worked for AOL. That is pretty neat.

The first thing I thought of when I saw you put the automated testing in place was the test system capacity. I know a lot of automated tests would put a heavy load on the server.

First, perhaps you could have a limit on the number of tests per time period that can be run from a single domain name. That number could be tweaked over time as more people use the website.

Second, you could have a script to automatically delete hourly data if it is more than 24 hours old. Thus, the trend data for a particular webpage would only show the last 24 hours. Likewise, for daily data, you could automatically delete data after 7 days. For weekly data, data could be deleted after 52 weeks. For Monthly data, you could delete data after 12 months. For yearly data, you could delete it after 10 years. If done this way, a single webpage would never have more than 105 database entries.

Third, you could have it where members have to log in every 60 or 90 days or their trend data would get deleted. This would encourage people to come back to the website.

Fourth, you could have a limit on the number of webpages that track trend data for a particular member. This number could also be tweaked over time.

Fifth, paying for a service would also be a possibility. If you do this though, I think you should have some sort of trial period or maybe even a number of free tests a single IP can do per day. That way you could still reel people in to use the website.

Sincerely,
Travis Walters

27

WebPagetest will always be free and there won’t ever be subscription services. I like it that way as it’s less complicated and simplifies building a collaborative community (also makes it easier to not have to support SLA’s, etc). I am looking at various options along the lines of what you are talking about.

The code already supports running tests at different priority levels so the recurring testing would be given a lower priority than on-demand tests. Would just have to keep track of capacity to make sure the system didn’t get over-provisioned.

As far as storage goes, even the hourly results are not a problem - where things get to be intensive are the storing of the screen shots, http headers and the transaction details needed to generate the waterfalls. For recurring testing it’s likely that screen shots and http headers wouldn’t be kept at all and the details for the individual requests would only be kept for X days.

It’s all easy enough to do, it just takes time to implement :slight_smile:

28

One other option I’m considering is that users who run/sponsor test locations would get X testing slots (well above what would be made available for regular registered users) which would encourage more hosting of the testing infrastructure.

29

Hey Patrick,

One suggestion I have is to allow members to specify FTP information to their own servers.

It would be very easy for a website owner with their own server to setup a FTP login that only allows access to a specific folder, which could hold such waterfall images generated by this website.

Let me know what you think? The Google Affiliate program does this in a way. Affiliates provide FTP login information and product data files are sent to third party servers for processing.

Sincerely,
Travis Walters

30

I’m actually probably going to support a beacon rather than a FTP dump so that when a test is complete it can post the data realtime to an external system as well.

In either case you’d have to build something to deal with the data (or use something off the shelf). I’m working with a few people in the industry to try to standardize the interfaces which would make it a lot easier to plug and play.

31

Hey Patrick,

I am looking at the differences and similarities between the first view and the cached version of this test:

http://www.webpagetest.org/result/100613_3XS/1/details/
http://www.webpagetest.org/result/100613_3XS/1/details/cached/

Notice how there is a huge gap after the document complete event on the cached version. One thing I have always wondered about (and I am sure others have as well) is what goes on during that period and how do I fix it?

I am sure cached requests occur during that period. Is there anyway you could visually show cached results perhaps in a different color? I think there needs to be a way to help people understand what goes on during those periods with huge gaps.

Sincerely,
Travis Walters

32

Yep, I had started the work on it, just need to finish it off and figure out how to display it so it doesn’t confuse people. Might make them a faint yellow or something like that so it’s clear when you’re just looking at the waterfall.

33

The yellow might confuse people with 3xx result.

Perhaps some sort of magenta?

34

Hey Patrick,

Another thing that would be really useful (but I am not sure it would be possible at least easily) is some sort of JavaScript Analyzer that could go through all JavaScript code for a webpage and do some or all of the following:

  1. Specify Unused Functions and Variables
  2. Create a JavaScript File with all Used Functions

I think firebug does the first item to an extent. It would be nice to take that a bit further and create something that would do step the second item.

This would be great for getting exactly what you need out of third party JavaScript libraries.

Sincerely,
Travis Walters

35

That may be a little more difficult than it may even appear (to the point that I’m not sure you’d ever want to trust something that did). The difficulty comes in figuring out if you got 100% of the possible code paths to execute before you go peeling away code and it’s unlikely that you’ll get that coverage with something like WebPagetest.

I do want to see about adding code coverage and profiling at some point but that will probably be a ways out still.

36

I would like to be able to automate the entire process. So that i can set the time and page i would like tested and it would check it daily and output the test results into a csv or similar type of file.

37

How many pages?

It’s pretty high on the list to implement recurring scheduled testing but mostly for the private install version. I’ll probably enable it on the public version for a handfull of urls for “approved” registered members but need to be a little careful to protect the testing capacity.

38

maybe 10 domain names or so

39

Hi Pat

I use Connection rather than Waterfall view almost exclusively. For me it’s a really standout feature.

Might we have this as the graph under the Video, please?

Also, could it be the first graph on the Details tab, not the second?

Guess it’s a consensus thing, of course.

BR

N

40

Actually, I could make it selectable by preference so you won’t have to get consensus. I am pretty swamped for the next few weeks but should be able to take a look at it after I get clear. It would be tied to your forum login (been wanting to add more capabilities for “registered” users anyway).

I prefer the cascade view for my day-to-day because it’s easy for me to see where a site needs to focus, even from a thumbnail but the connection view is helpful for the implementation details (could be a difference between shallowly looking at a ton of sites vs. looking deeper at a few).

Thanks,

-Pat