Time To First Byte

I Have been absorbing all the info and comments on this subject. However, my SITEGROUND shared plan is not performing well in this area. I must first state my small blog is well optimized getting 99-100 on google page speed, gt metrix and pingdom ( accept for vary acceptance header on latter) . With https.www.com 98 but vary acceptance warning still there; no problem.
HERE >> Web page speed test ( and sometimes google page speed) DISMAL TTFB scores and I would say, with grow big plan SITEGROUND, the problem is there at least once a day if not more!!
My thoughts ( as a newbie) but reading everything I can get my hands on, is that ARGUABLY, the only solution may be a DEDICATED SERVER!

I do need some ideas and opinions, please!