Over the course of the last 2-3 weeks I have noticed more and more of my Tests in my test history are starting to fail with “Test not found” when I click on the results, for example the below test WebPageTest - Running web page performance and optimization tests...
What’s even stranger other tests in my history now link to a Google Maps test made from Tokyo in July when the url shows it was a test from September
WebPageTest - Running web page performance and optimization tests...
Am I hitting some kind of limits in my account or are there storage issues?
Thank you,
Andreas
Not an account problem or a limits problem, sorry. I transitioned to a different way of archiving tests (http instead of NFS mount) and there were a few hiccups where tests were lost but it shouldn’t be an ongoing problem. I may have broken something more than I thought though if it is still happening from tests from the last few days.
The maps test is also my fault. Some user’s automated test scripts were stuck in a loop constantly re-requesting tests that were not found and not handling the error condition so I copied an old valid test to backfill some of the missing tests and make their scripts stop hammering the server. Unfortunately it looks like I was probably a little too aggressive with the backfill.
Thank you Patrick,
I will run my usual battery of tests today and see if everything is working again. If I see something not working I will let you know.
Hello Patrick,
well it looks like the tests from last week are gone again.
Link to October 9th Test WebPageTest - Running web page performance and optimization tests...
October 7th Test
WebPageTest - Running web page performance and optimization tests...
September 29th
WebPageTest - Running web page performance and optimization tests...
September 28th
WebPageTest - Running web page performance and optimization tests...
I looks like I have lost all test history data since mid September and none of the test have saved for longer than maybe a day. Going to run again my Wednesday test suite hoping it will save this time.
[hr]
I just noticed something. One of the tests from last week did not get lost
The biggest difference between that test and the other tests is that I did that one fresh through the Form on the Home page for all the other tests I used the “Re-run the test” button for example rerunning this old test
Could the re-run have something to do with the storage issue?
That’s bizarre. The test infra did change where the long-term storage is archived to and there were a few days of hiccups along the way but it shouldn’t be persistent and there is around the same volume of data being archived so there is some edge condition that isn’t being handled properly.
A few questions if you don’t mind that might help me isolate it:
- Were the tests run manually or through the API?
- What location?
- Do you know if the tests completed quickly or were in the queue for a really long time?
- Did the tests complete and then disappear or were they queued and then disappeared?
I significantly simplified the archiving logic, particularly the part that deletes tests to make sure it never deletes a test that hasn’t been archived. Hopefully that helps:
https://github.com/WPO-Foundation/webpagetest/commit/d969766eccf464fe1e1e09f5eeaec03a373a09d6
To answer your questions.
- Were the tests run manually or through the API?
The tests were all run manually through the webpagetest.org UI.
- What location?
I am assuming you mean Geography. They ran from “Toronto, Canada - Chrome - Cable”
This is todays test currently accessible WebPageTest - Running web page performance and optimization tests...
- Do you know if the tests completed quickly or were in the queue for a really long time?
They were queued up just as expetced 10 tests with 9 first and repeat runs each
- Did the tests complete and then disappear or were they queued and then disappeared?
The tests completed, I saw the results and when I checked a week later they were gone.
Hope this info sheds more light on it.
Good Morning Patrick,
yet another set of tests got lost WebPageTest - Running web page performance and optimization tests...
and
WebPageTest - Running web page performance and optimization tests...
yet this one
and this one
Seem to have survived. So 2 test survived 9 tests vanished.
Let me know what I can do to help debug this further.
Do you know if those were after my change or before (they were on the same day)?
That said, I just added another level of protection to have it verify that the archived copy exists in the long-term storage by actively polling for it before allowing the local copy to be deleted. If it still gets deleted after this change then it is going to be from a different chunk of code somewhere (not the archiving code).
Hello Patrick,
everything is looking good this week so seems like your last change fixed it.
All tests from last week remained accessible.
Again thanks for your help
I maybe experiencing this issue? Seems inconsistent between tests.
https://www.webpagetest.org/result/240102_AiDcXP_DR3/