Developers: Stop Building DDoS Attack Vectors!

The recent memcrashed attack hit GitHub with a record 1.3 terabits per second of traffic.  A more recent attack just broke that record, hitting an ISP with 1.7 Tb/s.  Just for an idea of scale, that is the same as 170,000 high-speed cable modem uplinks (10 Mb/s up with 200 Mb/s down) going full blast.  Of course, attackers didn’t attack by saturating the uplinks of 170,000 hapless cable-internet customers.  Instead, they exploited an amplified UDP reflection attack known as memcrashed.

To understand how we developers can stop helping DDoS attackers, we first need to understand how UDP amplification attacks like memcrashed work.  Then, we can discuss some solutions that application developers should use…yet currently don’t. But by using these solutions, we could end amplification attacks in the long run.

Reflection Attacks – Mirrors Required

Reflection attacks work via a mechanism called IP spoofing.

The idea is like sending someone a letter with a forged return address.  When the recipient of the letter responds, the response goes to an unsuspecting victim.  In the same way, IP packets sent on the Internet have the destination IP address in the packet header. In our letter example, that’s the recipient’s address on the envelope. IP packets also have a source IP address, which is just like the return address on our envelope. It’s this return address that the recipient will send any responses to.

Tricking Ronald into responding to the White House – Credit: Peace Corps

For example, to start a TCP connection, the client initiates the connection by sending a SYN packet to the server and filling in its own IP address as the source (return) address so the server can reply.  The server replies with a SYN/ACK packet back to the address listed in the SYN packet.  If an attacker spoofs the client IP address in the SYN packet, the server will respond with a SYN/ACK to the innocent victim listed as the source in the SYN. When an unwitting server is used in an attack this way, the server is called the reflector.

TCP 3-Way Handshake – Credit: N-21

Fortunately, these attacks are difficult to scale since the SYN packet and SYN/ACK packet are about the same size. So in order to send 1.3 Tb/s to a victim, the attacker has to generate 1.3 Tb/s to the reflector servers – no easy feat.

Amplification – Ours goes to 11!

Amplification is when there are servers on the Internet that reply to a small initial packet with a very large packet before verifying that the client isn’t spoofing its source IP address. Or worse yet, if they reply with many very large packets, we now have the recipe for a massive DDoS attack.

Amplification turns up the volume on an attack

Any standard TCP server will refuse to converse with a client who cannot first complete the TCP handshake by responding to the server’s small SYN/ACK packet with the right magic sequence number.  But UDP servers do not have this limitation.  And the more data that a UDP server generates in response to a tiny request packet, the bigger the amplification.  For example, if a 100 byte request packet can generate 2,000 bytes of response data, that is a 20x amplification.

Memcached – Ours goes to 51,200!

Enter memcached.  Memcached is a popular key/value store used by many applications.  Each UDP packet sent to memcached will contain a request for one or more keys.  Each key value can be up 1MB in size by default, so in response to one small request packet, memcached can send hundreds of maximum-sized UDP packets at the spoofed victim IP address until all key values are delivered.

Even worse, memcached has no authentication requirements, which means that anyone can write a large key to ensure maximum amplification.  After writing a maximum-sized key value, an attacker can conduct a massive amplified attack by requesting that large value with a spoofed source IP.  Actual amplification seen in the 1.3 Tb/s GitHub attack – 51,200 times!

Creating Safe Applications

The memcached developers will be the first to tell you that memcached is not designed to be run on the open Internet. Obviously. But that doesn’t seem to have fixed the problem, does it? So here we are, with over 100,000 such servers out there. Inevitably, no matter how many times people are told not to run something on the open Internet, they will.

Vulnerable memcached servers – Credit: Rapid7

As developers, we need to design server code so that it can’t be used as an amplification vector. For the next application that uses UDP, first try to use TCP instead – problem solved.  If you must use UDP, then use DTLS.  If you can’t use DTLS, then read on.

For raw UDP communication, force new clients to make a TCP connection first, and then prove with every UDP packet that it made this TCP connection. You want to do this in a way that:

1. Requires a TCP connection rarely.
2. Is Stateless – the solution does not require the server to keep per-client state.
3. Is Efficient – the solution does not add a huge overhead to the UDP packets.

Here is a solution with all three properties.

1. Before transmitting UDP packets to the server, the client makes an HTTP connection to the server and requests a token.
2. The server responds with a plain text expiration time and an encrypted token that contains the source IP address, the expiration time, and a server-specific magic number. (Use the real source IP address, not the contents of the X-Forwarded-For HTTP header as that can be spoofed too.)
3. Every UDP packet the client sends must contain this encrypted token.

The server then does the following for every UDP packet received:

1. Decrypts the token with the secret key.
2. Checks the magic number, if it does not match the server’s magic number, drops the request.
3. Checks the expiration time in the token, and, if expired, or too far into the future, drops the request.
4. Checks that the source IP of the packet matches the one in the token and rejects the packet if it doesn’t.
5. If the above checks pass, it responds to the request.
6. If the token is close to the expiration time, say 1 minute, the response should include a new token with an updated expiration time to avoid unneeded TCP connections and network delays.

Because the token can be encrypted with a small block cipher like AES, with a 128 bit block size, the overhead in a UDP packet is small.  Since the token is encrypted, it can only be made by someone with the server’s secret key (if this weren’t the case, the underlying cipher would be susceptible to attack).  Since the token contains the expiration time, it can only be used until it expires (even if an attacker replicates it to their bot net and starts spoofing the source IP).  Moreover, since the token contains the source IP address, an attacker has to be able to conduct 2-way communication using the victim source IP.  The magic number is there to ensure that random data used as the token in an attack does not have a high likelihood of decrypting to a valid expiration time and a random source IP address.

It is essential, in the above sequence of steps, that the server not respond to any packet that fails the checks, as this behavior can then be used for reflection attacks.  Also note that I’ve built this protocol by distilling ideas from other protocols (like DTLS, and QUIC) that have been analyzed by real crypto-protocol experts.  That is why I recommend you use these tried and true protocols (TCP, DTLS, etc.), and only resort to this method as a last resort.  I think it is secure, but without widespread peer review and real world attacks, you really don’t know.

For paranoia’s sake, expiration times should also be short, say 10 minutes or so at most.  To be extra paranoid, the server should periodically rotate the key it uses for the cipher as well.  These measures make it less likely that a brute force or computationally intensive attack on protocol and cipher weaknesses can generate long-term large scale attacks.

If this all seems like too much, I’ll repeat, just use TCP, HTTP, Web Sockets, QUIC, DTLS, or another proven protocol built or hardened by real experts in secure protocol design.  Designing a protocol that will inevitably end up on the open Internet is not for the faint of heart.

Do NOT reinvent this! Or the wheel.

Conclusion – Do Your Part!

Whenever you open a network connection in your application, especially a server, you could be contributing to the next major DDoS attack or security breach.  Make sure you do your part by preventing the most well-known attack vectors, even if you think your service will never be on the Internet.  Use proven protocols whenever possible.  Carefully think through any custom protocols if the standard ones are inadequate. And remember, any service in the cloud is just one typo away from being on the open Internet.

When that happens, it’s up to us developers to save the day.