Google’s John Mueller said on Twitter that time to first byte, as a page speed metric, is not used at all by Google in search or search rankings. John said “AFAIK we currently don’t use TTFB for anything in search/ranking.” “It can be a good proxy for user-facing speed, but like other metrics, don’t blindly focus on it,” he added.
Time to first byte (TTFB) is a measurement used as an indication of the responsiveness of a web server or other network resource. TTFB measures the duration from the user or client making an HTTP request to the first byte of the page being received by the client’s browser.
Again, page speed is one of those things that many webmasters and SEOs obsess about. But most of them obsess about them because of rankings in Google. To see a negative ranking factor for site speed, your site has to be really really slow. I am not saying you should not be concerned with site speed, you should, it has huge influence on conversion metrics. But from a Google ranking perspective, chill out.

AFAIK we currently don’t use TTFB for anything in search/ranking. It can be a good proxy for user-facing speed, but like other metrics, don’t blindly focus on it.
— John ☆.o(≧▽≦)o.☆ (@JohnMu) November 30, 2017
Forum discussion at Twitter.