I am hoping someone can throw some light on this. Our site is running off a Vultr VM at their DC in Seattle with 32 GB RAM. The site is image heavy, so we moved the images to S3 with CloudFront CDN in the Mumbai region.
To the normal eye, the site looks like it’s loading fast, but tests here show terrible TTFBs for images pulled from the CDN - as high as 89 seconds. We found an old thread that suggests having the server closer to the CDN would help:
We would be happy to do that, but I would like to know why the TTFBs are so high when images are being served from a CDN - wouldn’t these be stored from the edge caching server/PoPs around the world and thus be served faster with lesser TTFB?
Be sure to also look at the TTFB for your document. It’s brutal - 11 seconds?
Moving S3 closer will help - buy typically if your images are requested frequently - cloudfront will keep them hot in the cache - avoiding the latency to re-download them from S3.
If possible move your server(s) as close as possible to your target customers.
What I would do is instead signup for cloudinary - and link it directly to the images on your primary server. You can see the image analysis here:
It says it could reduce the 9.5MB of image data on that page - down to about 1MB. Even it it was 2-3MB - that would be a massive improvement. As your test was using a DSL which has a speed limit. Also the latency between your test location (India) and your server location (Seattle) is probably pretty high - so there are probably 200+ms latency between those locations - multiply that by the packets required for a 10MB webpage - and it will start to add up fast.
I’ve used cloudinary on a number of websites with good results - they have there own CDN also and I believe a free tier.
Another option would be to look at cloudflare - they could terminate your website closer to the customer and help reduce latency - not sure if they do any image optimization yet.
The bandwidth utilization is saturated. The delays to start getting some of the images is because they are queued up behind the earlier images. Cloudfront supports HTTP/2 which means the browse will send it requests for all of the images that it knows about and then the server will send them back in some order (usually based on the priority and order of the requests but a lot of implementations are busted). The TTFB only looks long because the HTTP/2 connection is busy sending the other images.
Would this be the yellow bar seen for the wait time in the request for each image? Noticed this, but hadn’t connected this to bandwidth being a problem. If yes, this would be on the hosting provider’s side (Vultr) rather than CloudFront, right?
If the bandwidth issue could be fixed either by moving the server location from Seattle to Singapore (latter is closer to CloudFront’s mumbai region) and/or moving to a server provider with less bandwidth contention, would the TTFB for images automatically get sorted out?
The bandwidth saturation isn’t a provider problem, it’s a “10.7 MB of images” problem. The test is with a 1.5 Mbps DSL connection (little-b, bits). The theoretical fastest that amount of data could be transferred on that connection assuming no overhead is 10.7 MB * 8 Bytes per bit / 1.5 Mbps = 57s.
The 12 second TTFB on the HTML (for the application) makes that 69 seconds.
There are a few gaps in the download, possibly where some application logic is running or other network issues are coming into play
The yellow bars are time from when the browser discovered an image until it actually requested it on the wire. They are there because the server the images are coming from is HTTP/1.1 so the browser opens 6 connections and can only download 6 images at a time (all competing for the 1Mbps of bandwidth).
The images that take a long time but don’t have a yellow bar are served from cloudfront which supports HTTP/2. The images get requested immediately but are queued behind the other images that were also requested.
To make the content fast(er) without changing the experience will require a bunch of different things:
Fix the base HTML TTFB. 12 seconds is insanely long. Likely a result of the hosting for the application or the application code itself.
Optimize the images. The images are not compressed particularly well and you can get at least 3.5MB reduction just by compressing them better (even more by using new formats like avif): WebPageTest Test - WebPageTest Optimization Check Results
Thanks, Patrick Meenan for that wonderfully detailed explanation. I am much obliged and will consider and address every point you have made.
For some reason, clicking on reply brings up a 500 error and clicking on Reply with Quote says some quotes have been added, but nothing shows up in the Quick Reply box. Had filed a bug report about it.
Thanks to you, too, dreamwerx for the suggestions. We will most probably move the server to Singapore, which is much closer to the Mumbai AWS region than Seattle.