I recently submitted some tests to WPT through Perlscript (via API calls) for URL=www.cnn.com
First, please consider the “default” case (i.e. Dulles_IE7.DSL, runs=1, fvonly=0). Mostly the “requests.csv” file I download has all the info as expected - e.g. Default_IE7_1. Column C shows that both “Cleared Cache-Run_1” and “Cached-Run_1” tests were conducted - as expected.
However, on 2 occasions, I saw that only “Cached-Run_1” test was done - e.g. see Default_IE7_2 - even though I didn’t change any of the parameters.
Question #1: Would you know why this arbitrary behavior occurs?
Now consider a slightly different case from the “default” - the only difference being that instead of IE7, the browser was Chrome (i.e. Dulles_Chrome.DSL). In this case, the “requests.csv” files I receive seem to consistently:
a) Conduct only one of the 2 tests: “Cleared Cache-Run_1” and “Cached-Run_1” (I can’t tell which - most probably the former), and
b) Leave the first 3 columns, and a few others, empty
#1 - Do you see it regularly? I had some system issues this morning (and one of the two servers is actually not responding right now so I need to go physically kick it). It isn’t supposed to run the repeat view test if the first view test fails.
#2 - I’ll look closer at the CSV export for Chrome. Looking at the actual tests it looks like all of the data is there for first and repeat view, it’s just not making it to the requests.csv. A lot of the fields aren’t populated because we basically implemented just enough to get the waterfalls working and haven’t had time to go back and fill in some of the other fields yet.
First of all: Wow, amazingly fast response Pat - Thank you!
#1 - Actually it has happened only twice so far: at 9am and 9:30am (CST) today. Server problems would probably explain it.
#2 - I understand why lot of the fields may not be populated. But the first 3 cols were populated for the IE7 case, shouldn’t they be filled for all browsers? Unless you are using separate code for different browsers of course…
Would appreciate it if you can fix it such that all the relevant data makes it to the requests.csv files. Would such a fix apply to all the locations around the US/world?
First of all: Wow, amazingly fast response Pat - Thank you!
#1 - Actually it has happened only twice so far: at 9am and 9:30am (CST) today. Server problems would probably explain it. Will discard those files as anomalies.
Regarding “It isn’t supposed to run the repeat view test if the first view test fails” - so if I see only one test result (as in the Chrome case), is it perhaps an indication that the first view test is repeatedly failing?
#2 - I understand why lot of the fields may not be populated. But the first 3 cols were populated for the IE7 case, so shouldn’t they be filled for all browsers? (Unless of course different code is being used for different browsers…)
Would appreciate it if you can fix it such that all the relevant data makes it to the requests.csv files. Would such a fix apply to all the locations around the USA?
#1 - Chrome is running both, if you go to the actual results page you can see that the data is available for both tests, it’s just not making it through in the CSV stage.
#2 - Different code is being used for the Chrome support. It’s a new codebase that will support multiple browser and eventually we’ll migrate the IE testing to use it as well (the IE code is VERY IE-specific).
The fixes will automatically get deployed globally.
#1 - By “actual results page”, do you mean this GUI results page? I think the small table at the top of this page is a snapshot summary of the First View and Repeat View runs?
In this post, you said that “the CSV is actually used to draw the UI” so it’s strange that the CSV does not have all the data. Perhaps there’s another source from which both CSV and GUI draw…
#2 - When might the fixes be deployed? (time frame of days/weeks/months?)
BTW, except for Dulles_Chrome which explicitly states Chrome 11, other locations like Chicago_Chrome, LosAngeles_Chrome, NewYork_Chrome, and SanFrancisco_Chrome do not state the version number (info extracted from this XML file). How to determine the browser version we’re dealing with?
#1 Yes, but click on the waterfall for either to go to a page-specific run. The CSV data is actually stored in the filesystem as tab-delimited files with one file for each run (and separate files for first and repeat view). The CSV code just merges the files together into a single file and changes it from tab to comma delimited.
#2 somewhere between days and week (wekk, to get al of the data in there and make enough data columns available to be useful). Full data parity would be months (if ever) but a lot of the fields aren’t useful.
All of the Chrome locations are running whatever the current production version is. Chrome is super aggressive about automatically updating itself. I’ll see if I can add the full version number to the data.