Notice that these were run on the same day, and I did nothing to the server between the two tests. You can do this yourself and get back to back A or F for first byte. Is my server really this inconsistent, or could your site’s measurement be off?
Your server really is that inconsistent. The difference is coming from various layers of caching on your server (from the OS to the database to the actual application).
If you visit the site (or a page) for the first time, the server will likely have to go to disk for everything, particularly if it is on a shared server. The next time you visit the page, the database queries may be cached in the database, the files (code and data) may be in a filesystem cache in RAM, etc.
It’s unfortunate that the first byte grade is not deterministic and can vary from run to run but I thought it was important enough to highlight to justify the variability.