OK, interesting. TCP cookies allocate (practically) no resources to a TCP connection until the three-way handshake is complete, whereas Fast Open can start receiving data before the handshake even finishes. But even if you only do Fast Open on hosts that you have previously completed a normal 3-way handshake with, attackers will just do one full handshake at the beginning of the attack and then switch to SYN flooding.
SYN flooding is only really effective if it's spoofed, and you can't spoof a full three-way handshake unless you're in a privileged position on the network.
attackers will just do one full handshake at the beginning of the attack and then switch to SYN flooding
Except it's no longer SYN flooding at that point, it's full HTTP request flooding.
But in a sense isn't that really the goal of this design? To make it a bit more efficient to get requests to the application layer?
In any case, it seems like an application using this feature to have an efficient way of disabling it if it can't handle the current load. Kernels could add efficient heuristics to throttle it automatically too.
I'm more concerned that a bug in the entropy of the key generation process could turn these servers into massive reflected DoS amplifiers. E.g., the attacker sends 1 packet with the source address spoofed and the webserver replies with an entire HTTP result to the victim.
In fact, many of the security considerations are the same.