Differences in API and website UI

Web Performance Discuss Test Results
1

Hi,

I have been collecting web page load speed data using the webpagetest.org UI by visiting the website everyday and also using the API. The issue I am having is that both methods provide different results and I am interested in the reason for this. My aim is to migrate to using the APIs to automate the collection of this data. But at the moment, the two sets of data over the past month do not correlate and I cannot explain the difference.

This is the procedure I follow when testing using the website:

  1. Enter URL
  2. Set test location as Ireland - EC2 (Chrome, Firefox)
  3. Set browser as Chrome
  4. Open advanced settings
  5. Set connection to 3G Fast (1.6Mbps/768 Kbps 150ms RTT) in test settings
  6. Set number of test runs to 3
  7. Click first view only
  8. Tick capture video
  9. In the chromium tab, tick capture lighthouse report
  10. Click emulate mobile browser and choose Nexus 5
  11. Click capture dev tools timeline
  12. Click run test and wait for results to finish
  13. Note down speed index and date from table

I am also attempting to replicate the same results using the API. I am using Python and here are the parameters I am setting. Note - I am aware that some of the parameters are different, but I believe they have no real impact on the results but let me know if I’m wrong!

parameters = {
‘k’ : #api here,
‘f’ : “json”,
‘location’ : “ec2-eu-west-1:Chrome.3GFast”,
‘mobileDevice’ : “Nexus 5”,
‘url’ : url,
‘runs’ : 6,
‘lighthouse’ : 1,
‘timeline’ : 1,
‘video’ : 1,
‘fvonly’ : 1,

}

I take the speed index value from this process as this is the KPI my company uses. I record one result manually and kick-off 6 automated runs using the APIs. I then average the 6 API values of speed index into 1 API result. Both processes are testing the same URL. I believe I am using the same parameters as the website so in theory I should be getting the same results, however they are different. Here are some examples in difference in speed index.

Manual result:
date: 20/09/19, 9:35:44 AM
speed index: 4017

Api results:
date: 20/9/2019 9:26:57
speed index: 3051
date: 20/9/2019 9:27:17
speed index: 2798
date: 20/9/2019 9:27:31
speed index: 2613
date: 20/9/2019 9:27:38
speed index: 2852
date:20/9/2019 9:27:40
speed index: 2608
date: 20/9/2019 9:27:53
speed index: 2775
average: 2337

Here are the graph results over a month:

Manual process

2349×1402 117 KB

API

2441×1417 223 KB

The results from the API has been consistently lower than manually entering in the URL and testing on the website. Could you please help me identify why the results are different? What am I doing differently in the code compared to the website?

Thanks in advance.

2

Looks like your API tests are not enabling mobile emulation (mobile=1) so it is probably running the test as a desktop test.

3

Thank you! That looks to have increased the API results closer to the results from the manual process.