Test Error: Data is missing in scheduling large number of runs per job

WebPageTest Private Instances
1

Hi There,

Running the latest private instance 2.15 and mobile js agents.

I am trying to schedule anywhere between 100 - 500 runs for a single URL job. When I schedule these jobs, the results start coming in one by one.

“Waiting for test result…” is what is seen on the result page for the runs that are yet to execute.

Then, somewhere around run 50(the number varies quite a bit) or so, the test result page changes all the “Waiting for test result…” runs to “Test Error: Data is missing.”

What’s even more peculiar is that the missing data runs are still running on my mobile devices. And results do seem to come in as and when each run is getting executed.

The issue is that I am using WPT-API to determine if the test finished executing or not. When the test status flips to “Test Error: Data is missing”, the API reports that the job has finished running - even though the remaining runs are still running and the next job is still waiting for to be started.

Any idea what is going on here?

Thanks,
Sid

2

You are not going to have a good time scheduling that many runs at once. The mobile agents don’t currently know how to shard tests so each test must all be run by a single agent. Is there a reason you aren’t (or can’t) submitting it as several different tests?

There are probably timeouts in various parts of the code that look at the test and see that it started X hours ago and force it to fail because it has been running for too long. I think I fixed most of them as long as results keep coming in but there are probably a few left.

3

I am writing my own wrapper tool that uses WPT-API by Marcel Duran to integrate WPT scheduling and result reporting into our own continuous integration tooling. We don’t yet have WebPageReplay and the associated hardware setup right now to ensure consistent network conditions.

So in order to compensate for the network variability, I run each of our pages ~100-500 times until we can get an acceptable confidence intervals on the result.

Since WPT already has functionality to calculate the median, stdev + a bunch of other statistical measures and even has a nice charting UI for displaying these distributions, I thought I would simply use it in WPT, instead of scheduling each run individually and then parsing the results and graphing them myself using my wrapper tool.

Does WPT have a way that can do the statistical analysis across multiple jobs? This way, I can schedule 500 tests individually and then have the same charting and statistical functionality.

About the timeouts, I believe I can see this in the log, although it is not very descriptive on what line of code that produces this. Maybe you can spot where this is instantly.

The timeout is not a 100% consistent behavior, from what I can see. Some jobs can run for hours with no trouble. Some other jobs last less than 15 minutes.

Any way to track where these timeouts are triggered and set different values for them? (or maybe eliminate them?)

Log lines below:

2014/07/24 20:08:57 - Test Run Complete. Run: 82, Cached: 0, Done: , Tester: D0FBA0A034320A19-10.28.109.141
2014/07/24 20:09:19 - Test Run Complete. Run: 83, Cached: 0, Done: , Tester: D0FBA0A034320A19-10.28.109.141
2014/07/24 20:09:19 - Test has been running for 29 minutes and it has been 100 since the last update, forcing the full test to finish.
2014/07/24 20:09:40 - Test Run Complete. Run: 84, Cached: 0, Done: , Tester: D0FBA0A034320A19-10.28.109.141