iPhone testing still a bit screwey?

I know there were some recent issues but from what I’d read, those looked to be addressed. That being said, I’m running into a rather odd issue when testing via one of the iPhone devices.

I haven’t tried the iphone 8, but when testing using the iPhone 7 on two different sites (both on cloudflare), the results are…odd

The filmstrip and screengrab look accurate, but the timing is WAY off and the only resources that show as being loaded are images from the site in question and third party scripts such as Google Analytics scripts. No scripts from the origin site, no css, fonts, or html…just images.

Got the same result with two different sites. However running a test against them via the Dallas server, the results are as expected with all resources accounted for.

Any ideas?

My first thought was a firewall or WAF, but that doesn’t seem to be likely since the filmstrip shows the site loading so it was obviously getting the proper resource files for the request.

Yeah, sorry - Safari effectively broke the inspector protocol in 12.3 and made it racy so it’s not possible to capture a bunch of the early requests in the waterfall. They landed a fix in December but it hasn’t rolled out in a release yet.

WebPageTest bug tracking it here: Faulty results when testing with 12.2+ · Issue #265 · WPO-Foundation/wptagent · GitHub

ahh…not sure how I missed that, but I was running tests using the iPhone 8 recently with no issues. The last accurate test I ran was on 2/15. It wasn’t until after that date that the results began missing so much of the resources.

Seems as if this may be related to something else (or at least in addition to that) since things were working fine until just over a week ago.

I also tested one of the sites using the Nexus 7 out of Dulles and ended up with a “The test completed but there were no successful results” error.

Looking in the logs (both server and cloudflare) I don’t even see a connection that could have been from WPT around the time I ran the test.

I’m guessing they aren’t related. I see at least one other person recently posting about the same issue.

I guess the part that is missing is that most of the iPhones were held back from updating and were still on 12.2 until a couple of weeks ago. Now that they are all on the latest iOS they are all plagued by the Safari issue.

The Nexus 7 issue isn’t related. The Nexus 7 is largely dead an needs to be wiped and re-configured to (hopefully) work. After running for ~7 years it’s also possible that it’s just dead beyond recovery (the flash back then wasn’t great for endurance). I just hid it so it won’t appear as available.

That makes sense. A case of upgrading being more of a downgrade. I know how that works at times.

I guess I’d just been lucky enough to not have hit any of the upgraded iPhones until they were all upgraded. Well I guess I’ll just wait until the fix is hopefully rolled out to do any iphone based testing.

Nexus 7 makes perfect sense. I wouldn’t normally have selected it for a standard test. It was just as a test/comparison to what the iPhone was giving…only to end up making things more muddy.

Thanks for the information!

What is the closest device to the iphone 8 spec wise? having the same issues on the metrics being weird due to the safari update

There’s nothing close. All of the android devices are significantly slower than iPhones. The desktop browsers running mobile emulation are the best bet but none of them are actually running safari.

FWIW, all of the visual metrics should be fine and the times pulled out of navigation timing should also be fine (loadEvent and domContentLoaded event): WebPageTest Test - WebPageTest Details

What is the location string that you use to run tests on iphone 8, Dulles, 4g?

You can’t access any of the iPhone’s from the API, only the Moto G4 and G are available through the API (Dulles_MotoG4 and Dulles_MotoG)