Issues with tests not starting up and stopping in between

WebPageTest Private Instances
1

I have set up new WebPage test server instance via the AMI available on AWS with auto scaling EC2. However, I have been facing problems such as the tests stopping midway, waiting for the next iteration, new tests being queued, even if there are no tests in queue.

This is only resolved when the older EC2 Agent Instances are cleared and a new one is set up. Also, ec2 logs show that the server is trying to boot up new machines (but is unable to do so as max limit for each location is 1 by default)

Also, nginx logs show that the agents are trying to get work from the server so the agents are alive and healthy I guess.

Here are the appropriate logs:

EC2 LOGS:

[quote]09:15:11 - Launching m3.medium ami ami-4a84a220 in us-east-1 for us-east-1,us-east-1_IE11 with user data: wpt_server=10.2.157.67 wpt_loc=us-east-1,us-east-1_IE11
09:15:12 - Instance i-03a08f68283b7b6b2 started: m3.medium ami ami-4a84a220 in us-east-1 for us-east-1,us-east-1_IE11 with user data: wpt_server=10.2.157.67 wpt_loc=us-east-1,us-east-1_IE11
09:20:15 - Error: Listing running EC2 instances: Request limit exceeded.
09:40:16 - Error: Listing running EC2 instances: Request limit exceeded.[/quote]

Test Logs:

[quote]Test log

2017/09/04 09:11:44 - Test Created
2017/09/04 09:20:35 - Starting test (initiated by tester i-03a08f68283b7b6b2)
2017/09/04 09:20:35 - Run 1 assigned to i-03a08f68283b7b6b2
2017/09/04 09:21:26 - Test Run Complete. Run: 1, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:21:26 - Extracting 71441 byte uploaded file ‘/tmp/phpHN2jvT’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:21:26 - Done Processing. Run: 1, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:22:04 - Test Run Complete. Run: 1, Cached: 1, Done: 1, Tester: i-03a08f68283b7b6b2
2017/09/04 09:22:04 - Marked shard 1 as complete: {“1”:true}
2017/09/04 09:22:04 - Extracting 42916 byte uploaded file ‘/tmp/phpzie5D2’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:22:04 - Done Processing. Run: 1, Cached: 1, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:22:04 - Run 2 assigned to i-03a08f68283b7b6b2
2017/09/04 09:22:51 - Test Run Complete. Run: 2, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:22:51 - Extracting 66721 byte uploaded file ‘/tmp/php9DrbER’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:22:51 - Done Processing. Run: 2, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:23:27 - Test Run Complete. Run: 2, Cached: 1, Done: 1, Tester: i-03a08f68283b7b6b2
2017/09/04 09:23:27 - Marked shard 2 as complete: {“1”:true,“2”:true}
2017/09/04 09:23:27 - Extracting 42913 byte uploaded file ‘/tmp/phpU4ltPg’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:23:27 - Done Processing. Run: 2, Cached: 1, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:23:27 - Run 3 assigned to i-03a08f68283b7b6b2
2017/09/04 09:24:15 - Test Run Complete. Run: 3, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:24:15 - Extracting 67297 byte uploaded file ‘/tmp/phpkYRVVg’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:24:16 - Done Processing. Run: 3, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:24:54 - Test Run Complete. Run: 3, Cached: 1, Done: 1, Tester: i-03a08f68283b7b6b2
2017/09/04 09:24:54 - Marked shard 3 as complete: {“1”:true,“2”:true,“3”:true}
2017/09/04 09:24:54 - Extracting 43532 byte uploaded file ‘/tmp/phpGnmBEi’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:24:54 - Done Processing. Run: 3, Cached: 1, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:24:54 - Run 4 assigned to i-03a08f68283b7b6b2
2017/09/04 09:25:43 - Test Run Complete. Run: 4, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:25:43 - Extracting 66573 byte uploaded file ‘/tmp/phpvnikHo’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:25:43 - Done Processing. Run: 4, Cached: 0, Done: , Tester: i-03a08f68283b7b6b2
2017/09/04 09:26:21 - Test Run Complete. Run: 4, Cached: 1, Done: 1, Tester: i-03a08f68283b7b6b2
2017/09/04 09:26:21 - Marked shard 4 as complete: {“1”:true,“2”:true,“3”:true,“4”:true}
2017/09/04 09:26:21 - Extracting 42909 byte uploaded file ‘/tmp/php7RmwC6’ to ‘./results/17/09/04/ZG/2’
2017/09/04 09:26:21 - Done Processing. Run: 4, Cached: 1, Done: , Tester: i-03a08f68283b7b6b2[/quote]

Nginx access logs

[quote]
10.2.22.229 - - [04/Sep/2017:09:36:07 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:14 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:18 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:21 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:26 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:29 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:31 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:36 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:42 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:48 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”
10.2.22.229 - - [04/Sep/2017:09:36:55 +0000] “GET /work/getwork.php?shards=1&reboot=1&location=us-east-1,us-east-1_IE11&screenwidth=1280&screenheight=1024&winver=6.1&winserver=1&is64bit=1&software=wpt&version=17.8.0.391&ver=391&pc=IP-0A0216E5&ec2=i-03a08f68283b7b6b2&ec2zone=us-east-1a&dns=10.2.0.2&freedisk=6.202&GPU=0 HTTP/1.1” 200 5 “-” “WebPagetest Driver”[/quote]

2

I’m also experiencing this issue.

Watching the AMIs, tests seem to just stall out on the agent. Also, the agent is requesting work but seemingly not being given any. It then is shut down and another AMI is re-initialized. Sometimes it is given a bunch of work and will churn through it. Sometimes it’ll stall out mid test.

3

I am also having the same issue exact issues. Any update on this? I’d be happy to update this post with any log info as needed.

Thanks!

4

It looks like a bug in the newer versions. Not sure if this has been resolved yet.
A workaround for this is to set up a permanent agent by setting the minimum agent in configuration. We have currently set it up as 1 as per our needs.