Move Dulles testing to EC2?

WebPageTest Announcements
1

I’m looking at possibly moving the Dulles test systems to use EC2 instead of running on hardware in my basement :smiley:

I’ll do a lot more testing and provide the details of the analysis before committing but I wanted to get a feel for how people would feel about making the move.

Pros:

  • The test environment will be consistent for a long period of time (and there’s no risk of it changing just because I move)
  • It looks like EC2 provides more consistent results when running multiple tests
  • It also looks the the times with and without video capture are identical
  • (essentially) infinite capacity. More test systems can be brought up as needed
  • The public instance will be using test systems that are available for anyone to replicate for private instances

Cons:

  • Test results from EC2 are not identical to what is currently running in Dulles so comparison of times to previous tests will not necessarily be valid (this will be a problem if I ever change out the hardware as well)
  • The EC2 instances are VM’s without hardware acceleration for video so things like IE9’s hardware acceleration will not be active (don’t think this impacts most “pages” and is more relevant for highly active apps but hard to say)
  • The costs (to me) will be higher but probably cheaper than if I put dedicated servers in a co-lo

Initial reactions?

2

Go for spot instances… in the case of test servers the state really doesn’t matter much…

3

Yep, that’s what I’ve been testing with - the testers are completely stateless so it’s a great fit and the pricing is pretty good (5-12 cents/hr)

4

btw, the results from my initial testing is available here: Performance Matters: Performance Measurement consistency and Video Capture impact

5

This definitely seems like it would be a good move, the only major downside that I see is the increased cost to you personally. I know that you have sponsors for other testing locations, but have you ever considered putting a donation button or something similar on the site? I’m sure there are plenty of people who would contribute money to help defray your hosting costs. Demand seems to be growing for WebPagetest, and the last thing we want is for it to become prohibitively expensive for you to run.

-Jonathan

6

Thanks. The ads are doing a good job at offsetting the costs (with some headroom) so I don’t think it will be a big deal to cut over. One thing I’m looking at doing is adding support for spinning up instances as needed so some of the less-used configurations stay powered down when they are not being used (and more instances get spun up to handle big bulk requests).

The one I’m having troubles justifying so far would be to move to S3 for archiving the test results. I need to do some modeling to see how expensive it would be but it would be nice to have the raw results archived to S3 where I’ll be a little more sure about the reliability (right now it’s all on Dreamhost’s SAN and hopefully backed up :slight_smile: ). Last I checked it was around 700GB of data though. My intent is still to keep all test results for ever but you might see something where authenticated user’s test data gets archived to S3 as well for more reliable storage or something like that (or everything but the screen shots gets archived since it’s nicely compressible).

Thanks,

-Pat