Deep Dive: Making Our Network 15% Faster Than Before

What is TCP?

TCP stands for Transmission Control Protocol. On the internet, TCP is the protocol that controls the speed and method by which HTTP data is transmitted. Developed in the mid-1970s, TCP has been the foundation of internet and intranet communication.

Hardware Speed Improvements

Well yes, the transmission control protocol has changed since it's inception. Computers have also come a very long way since then, and today's hardware is hundreds or thousands of times more capable than it was back then. As such, TCP has been improved upon and modern networks and applications can do even more to increase performance, especially on high-bandwidth low-latency networks.

What About UDP?

This may have popped into your mind. What about UDP, and how is UDP used in relation to TCP? UDP stands for User Datagram Protocol. TCP is a stateful protocol, ensuring the client is capable of and ready to receive the data from the server before sending it. UDP doesn't care - UDP just sends the data as soon as it can. For UDP, it is up to the application to deal with preparing the client for receiving data. For HTTP and web traffic, we have HTTP/3 a.k.a. QUIC, which starts with TCP and transfers data over UDP afterward. UDP is much faster since it doesn’t have to wait for the client to acknowledge that it has received the data - if the data never makes it to the client, UDP doesn't know and doesn't care.

How To Make TCP Faster

Increasing Window Scaling Factor

Increasing socket buffer sizes ultimately allows us to increase the window scaling factor our edge servers advertise to clients. The window scaling factor is the maximum volume of data allowed in transit before the client is required to send an ACK back to the server.

Increasing TCP Buffers

TCP performance is fundamentally limited by two key factors: latency and window size. The buffer size in the operating system must be large enough to accommodate the window size. If the buffers are too small, the window scaling won't help because there's nowhere to store the additional in-flight data. The combination of larger buffers with TCP Large Window Extensions (RFC1323) addresses these limitations, especially for high-bandwidth, high-latency connections.

Bandwidth-Delay Product (BDP)

When transferring data over TCP, there's a mathematical limit to throughput called the Bandwidth-Delay Product, or BDP, which is equal to Bandwidth multiplied by Round-trip Time (RTT). This formula represents the maximum amount of data allowed “in-flight.”

RFC1323 Makes The Difference

Traditional TCP has a max window size of 64KB because the window field in the TCP header is only 16 bits. This creates an extreme limitation. No matter how fast your connection, if you can only have 64KB "in-flight" at once, you'll be limited to:

Max Throughput = 65,535 bytes ÷ Round-Trip Time

For a 100ms RTT, this means a maximum throughput of only about 5.2Mbps, even on a 10Gbps link! RFC1323 introduces the Window Scale option, which allows the 16-bit window size value to be multiplied by a scale factor (up to 14), effectively increasing the maximum window size to 1GB.

How To Make UDP Faster

Increase UDP Buffers

Increasing socket buffer sizes helps UDP traffic as much as it helps TCP traffic, by allowing the OS to process larger quantities of data more quickly. Without increasing these buffers, there is a risk that the OS will drop packets entirely if the buffer gets too full.

Results of These Changes: 15% Average Latency Reduction

DSM-AMS 27% Latency Reduction Graph

Average reduction of about 110ms performing an HTTP GET from Des Moines, IA, USA to Amsterdam, Netherlands. (Roughly 27% reduction)

DSM-CHI 14% Latency Reduction Graph

Average reduction of about 10ms performing an HTTP GET from Des Moines, IA, USA to Chicago, IL, USA. (Roughly 14% reduction)

DSM-LAX 8% Latency Reduction Graph

Average reduction of about 20ms performing an HTTP GET from Des Moines, IA, USA to Los Angeles, CA, USA. (Roughly 8% reduction)

Consistent Quality Graph 1 Consistent Quality Graph 2 Not only did we see a reduction in HTTP response times overall, but we also see that results are significantly more consistent.

Learn more

Get Started