Google doesn’t release detailed information on it’s ranking algorithm, even for page speed. Page speed is only one ranking factor (I’d see it as more of a penalty). What Google really cares about is content.
What we do know from released information is that:
- The scores on PageSpeed Insights isn’t the end goal. So don’t chase high scores on these tools. It’s also not accumulative so a higher score or faster website won’t make the website rank higher.
- Any ranking penalty relating to load time likely only affects a small amount of websites and it would probably have to be very slow.
- The algorithm takes into account the type of content and how long users might wait for this content. So load time plays a different role for different search terms
It’s possible Google uses a combination of metrics most likely related to the first byte time, actual load time, maybe even speedindex and the type of search made.
So in terms of SEO I would focus on improving the load time where as possible, but not at the cost of removing functionality that would affect user experience. If the site feels fast when your browsing it on mobile and desktop I would say you would be pretty safe from any penalty.
In saying that there is some evidence a faster load time correlates to improved conversion rates, so it could be worth optimizing the site for that reason.