Correct location/browser/connectivity settings for mobile test triggered via API?

WebPageTest General Discussion
1

I’ve been using a fork of Eduardo Bouças’ SpeedTracker tool to collect historical performance data on sites our team has in development, but recently I’ve run into trouble when attempt to run tests with mobile settings. I’m using the webpagetest command line tool in an attempt to determine the correct Location:Browser.Connectivity settings, and have posted some of my attempts below.

Here’s an example of a test that runs successfully, but gives me an error on the WPT results page:

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --location Dulles:Dulles_iPhone5c.LTE
{
  "statusCode": 200,
  "statusText": "Ok",
  "data": {
    "testId": "161201_HP_65CJ",
    "ownerKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "jsonUrl": "http://www.webpagetest.org/jsonResult.php?test=161201_HP_65CJ",
    "xmlUrl": "http://www.webpagetest.org/xmlResult/161201_HP_65CJ/",
    "userUrl": "http://www.webpagetest.org/result/161201_HP_65CJ/",
    "summaryCSV": "http://www.webpagetest.org/result/161201_HP_65CJ/page_data.csv",
    "detailCSV": "http://www.webpagetest.org/result/161201_HP_65CJ/requests.csv"
  }
}

Here’s the error on the results page:

It’s clear to me I’m using the incorrect ID for the browser here, but I’ve been unable to determine what the settings should be. I was able to find “Dulles_iPhone5c” in the getLocations.php list, but setting that value as the location returns the following:

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --location Dulles_iPhone5c { "statusCode": 400, "statusText": "Invalid Location, please try submitting your test request again." }

Are there specific settings that would allow us to run an LTE or custom throttled test on an iOS browser? Are theses browsers/locations not available to the API? Would we need to set up a private instance attached to a mobile device?

Thanks for taking a look!

2

The json result from getLocations.php still lists the browser names.

getLocations.php?f=json

3

Thanks Vincent. The json data will definitely be useful, but it looks like many of the locations in those results aren’t recognized by the API as valid:

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxx --location Dulles_Nexus7
{
  "statusCode": 400,
  "statusText": "Invalid Location, please try submitting your test request again."
}

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxx --location Dulles_IE8
{
  "statusCode": 400,
  "statusText": "Invalid Location, please try submitting your test request again."
}

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxx --location Dulles_GalaxyS5
{
  "statusCode": 400,
  "statusText": "Invalid Location, please try submitting your test request again."
}

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxx --location Dulles_iPhone6
{
  "statusCode": 400,
  "statusText": "Invalid Location, please try submitting your test request again."
}

$ webpagetest test barrelny.com --key xxxxxxxxxxxxxxxxxxxxxxxx --location Dulles_iPhone5c
{
  "statusCode": 400,
  "statusText": "Invalid Location, please try submitting your test request again."
}

I dug a little and was able to find an inactive thread referencing the same issue: [url]http://www.webpagetest.org/forums/showthread.php?tid=13664[/url]

Does anyone have any additional info on the disconnect between what the getLocations page displays and what the API recognizes as a valid location?

4

Not sure how the speedtracker tool works, but going by your example, if you want iphone5c and lte, and you want to use this location:

“Dulles_iPhone5c”: {
“Label”: “Apple iPhone 5c (iOS 9)”,
“location”: “Dulles_iPhone5c”,
“Browsers”: “iPhone 5c iOS 9,iPhone 5c iOS 9”,
“status”: “OK”,
“relayServer”: null,
“relayLocation”: null,
“labelShort”: “Dulles, VA”,
“group”: “Mobile Devices - Dulles, VA”,
“PendingTests”: {
“p1”: 0,
“p2”: 0,
“p3”: 0,
“p4”: 0,
“p5”: 0,
“p6”: 0,
“p7”: 0,
“p8”: 0,
“p9”: 0,
“Total”: 0,
“HighPriority”: 0,
“LowPriority”: 0,
“Testing”: 0,
“Idle”: 1
}
}

location:browser.speed

should be:

Dulles_iPhone5c:iPhone 5c iOS 9.LTE

5

That’s what I would’ve thought myself, Vincent. I’ve actually been testing this outside of SpeedTracker, using the OSX command line tool and Marcel Duran’s API console. The setting you mentioned for location:browser.speed is still throwing an error:

webpagetest test barrelny.com --key xxxxxxxxxxxxxxx --location Dulles_iPhone5c:iPhone 5c iOS 9.LTE
{
  "statusCode": 400,
  "statusText": "Invalid Location, please try submitting your test request again."
}

Have you been able to successfully use that location by any chance?

Instead of using this technique I’ve started using the “mobile” parameter to have Chrome emulate a mobile browser. Not entirely sure what I’m losing by using this method, but it does seem to be giving us fairly complete test results (the mobile testing agents seem to have some trouble recording for the film strip view).

6

http://www.webpagetest.org/runtest.php?url=http://www.google.com&f=json&fvonly=1&private=1&runs=9&video=1&location=Dulles_iPhone5c:iPhone 5c iOS 9.LTE&k=[key]

This worked for me.

Results: https://www.webpagetest.org/results.php?test=161207_TJ_bdf6b03fe749112231f82c1646301593

7

Just tried the same thing, substituting my key, and got this:

{"statusCode":400,"statusText":"Invalid Location, please try submitting your test request again."}

Did you just paste the URL in the address bar of your browser? If so, which browser do you happen to be using? If not, how did you trigger the test?

8

I think for the results I posted, I just pasted it via browser (Chrome), but I’ve also got a powershell script calling it for a daily test.

9

Thanks Vincent! Both myself and a colleague tried it without any luck. Going to let it rest for the time being since the mobile emulation through the Dulles Chrome testing agent has been working well for us.

10

I get the exact same error with the exact same settings. Is there a new or different way of doing this that we are not understanding? Would definitely prefer the device over the chrome emulation.

11

What device are you trying to test against? Only the Dulles_MotoG location is available for API testing. Specifically location=Dulles_MotoG:Motorola+G±+Chrome.3G should work

12

Thanks for the answer Patrick, it saved me submitting every permutation of Dulles / Moto G4 / Chrome 3G to the API :slight_smile:

We’ve been working against Google’s PWA Checklist and found WPT to be an excellent tool for identifying and resolving performance opportunities identified by Lighthouse tests.

The Moto G4 devices accessible through the regular test web page flow are the perfect fit for the perf challenges we’re usually looking at.

I realize this is a totally free service and extremely grateful but I’ve got to ask :smiley:

  1. Could you expand the set of mobile devices available for API testing?
  2. Alternatively, are there any vendors offering hosted WPT instances with mobile browser? I understand that we could host a private instance, but I’d rather just pay someone.

Thanks again for the wonderful service!

13

Chrome’s DevRel team just purchased a bunch of G4’s for me to deploy so they should become available to the API over the next couple of weeks (depends on when they come in and how long it takes me to set them up).

I’m not aware of anyone else running a WPT instance for public access (paid or otherwise) with physical devices. There are a few private instances that have them configured.

14

I should probably also mention that Lighthouse is also coming to WPT sometime this quarter (fingers crossed) so you’ll be able to run the full audit remotely and also get the traditional WPT waterfalls, filmstrips, etc.

15

? Amazing new Patrick thank you!

16

Most of the devices are online now so Dulles_MotoG4 is now available with the API as well (same browsers as Dulles_MotoG).