Lighthouse results out of sync with WPT results for scripts that use logData 0

Support
1

Hi,
I am wondering if I can get some help here.

I have a test whereby the webpage test results do not correspond with the lighthouse results

Here is the scenario
I am setting a cookie in a webpage test script in order to test a new version of a page. However, the results from webpage test seem to correspond to the new version of the page, whereas the Lighthouse scores correspond to the old version of the page

Webpage script steps

  1. Turn off logging (logData 0)
  2. Set cookie to inform the browser that the new version of the page should be used
  3. Navigate to page url (the cookie should be stored now, so next time we navigate to the page the new version should be loaded.
  4. Clear contents of page to start test with clean slate
  5. Turn back on logging (logData 1)
  6. Navigate a second time to page url

Expected Results: Webpage test results and Lighthouse scores are run against the new version of the page
Actual Results: Webpage test results correspond to new version of page. Lighthouse scores correspond to old version of page (i.e the first navigation rather than the second navigation)

I can verify this is happening by looking at the Lighthouse report and seeing that the resources for the old page are being loaded rather than the resources for the new page.

This to me looks like a bug. Am happy to share more info if that helps.

Thanks,

2

I also posted this question on Speedcurve support which is built on top of webpage test. I got the following answer which I will try now

The official stance is that LH doesn’t support WebPagetest scripting. However, some things work more consistently with LH than others. If you are using setCookie, it will be ignored/dumped when running the LH test. However, you can try to use setHeader to set the cookie value. Some folks have had success with that. Lighthouse: pass through cookies from WPT script · Issue #167 · WPO-Foundation/wptagent · GitHub

Steps will also very likely not work. But, you shouldn’t have to do two navigations using either method (setHeader or setCookie). The cookie will be pre-populated for the first navigation.