A fast load time when a page is indexed does not guarantee a fast load time when it is served up to the actual viewer. Serving the page from cache is the only way to guarantee that the page will still be fast when the user wants to view it.
Because users want relevant search results much more than fast websites. Google already factors in a website's performance in their rankings, but weighing it too much over content relevance will make search results worse.
If they actually cared that much about making the results "relevant", they wouldn't mix a bunch of irrelevant suggestions into the results page, each marked with "missing: <query_term>" pointing out exactly how they ignored part of the user's request.
By that logic then, what's the point of AMP if Google is saying page load speeds aren't really that big of a factor? Why go through through the trouble of deriving a whole new subset of HTML?
I've yet to see a Google engineer, executive, or "fanboy" address this question adequately.
This thread will be no exception. Queue the crickets.