Why is my website loading slow only for the first time?

I have a website which has a really long waiting time only on the first load. After that, when I reload the website, it loads fast.
Please check my performance report here: https://www.webpagetest.org/result/190110_WR_030914a71a1ea047ac2d7c2680d4bf87/

As you can see, when loaded for the first time, it is taking a long time to wait. After that, when I reload the website, it loads much faster. After that I do not open the site for an hour approx. Again, after checking after an hour or so, it is loading slow again in the first run.

Can you please help me and let me know why may this happen?

Is this a server issue?

That is usually a cache somewhere on the server (or CDN). The actual site itself is slow to build the page (more often than not a slow database) but the generated page is cached so it can be served quickly after that.

It could also be other caches in the application (database query cache, filesystem caches, etc) but more often than not it is a page cache.