Your SYN flood defenses passed the last penetration test with flying colors. The test fired 50,000 SYN packets per second at your edge, your mitigation kicked in, and the report came back clean. You signed off, filed it, and moved on.
Here's the problem: the real attack won't look like that test. The attacker won't send naked SYN packets from spoofed IPs at a predictable rate. They'll use a botnet of 100,000 real devices, each completing partial TCP handshakes at randomized intervals, mixed with legitimate traffic from the same geographic regions your actual customers use. Your SYN cookies will either let the attack through or block your real users. When it comes to SYN flood mitigation best practices, the standard playbook has a blind spot most teams never test for.
Why SYN cookies fail under real-world conditions
SYN cookies are the standard defense taught in every networking course. The server responds to every SYN with a SYN-ACK containing encoded state, avoids allocating resources until the handshake completes, and lets legitimate clients through while wasting attackers' packets. Elegant in theory.
In practice, SYN cookies have three critical limitations that only manifest at scale. First, they strip TCP options from the handshake — window scaling, selective acknowledgment, and timestamps all get lost. For your customers running latency-sensitive applications, this means degraded TCP performance during every attack, even successfully mitigated ones. Second, SYN cookies respond to every SYN, including spoofed ones. Your server becomes a SYN-ACK reflector, potentially amplifying the attack toward innocent third parties. Third, sophisticated attackers complete the handshake — they ACK back, establish the connection, then flood at the application layer where SYN cookies provide zero protection.
The retransmission-first approach
Nexusguard's Bastions mitigation engine uses a different strategy: drop the first SYN from every new source, and only accept connections from clients that retransmit. Legitimate TCP stacks always retransmit a dropped SYN — it's built into the protocol. Spoofed packets, by definition, never retransmit because the spoofed source doesn't know a SYN was sent on its behalf.
This approach preserves full TCP options (the retransmitted SYN carries the same options as the original), eliminates the reflection risk (dropped SYNs produce no response to reflect), and scales linearly (the only state tracked is a bloom filter of recently-seen source IPs, not per-connection state).
How to test your own defenses
The next time you commission a DDoS penetration test, don't accept a simple SYN flood from spoofed sources. Require the testing team to simulate a botnet of real devices completing partial handshakes, mix attack traffic with legitimate client connections from the same subnets, and measure TCP performance degradation for legitimate sessions during mitigation. If your protection passes a basic SYN flood but degrades under these realistic conditions, you have a blind spot. Following SYN flood mitigation best practices means testing against realistic attack scenarios, not textbook ones.
FAQ
Q: What is retransmission-based SYN flood mitigation?
A: Instead of responding to every SYN with a cookie, retransmission-based mitigation drops the first SYN from each new source and waits for the client to retransmit. Legitimate TCP clients always retransmit dropped SYNs; spoofed sources never do. This preserves full TCP performance while filtering attack traffic — a core SYN flood mitigation best practice for 2026.
Q: Why do SYN cookies degrade TCP performance?
A: SYN cookies encode limited state in the SYN-ACK response, which means TCP options like window scaling, selective acknowledgment (SACK), and timestamps are stripped from the handshake. This forces connections to fall back to less efficient TCP behavior, increasing latency and reducing throughput for legitimate users during an attack.
