TCP Thin-Stream Modifications: Reduced Latency for Interactive Applications

Are you tired of having to wait for seconds for your networked real-time application to respond? Did you know that Linux has recently added mechanisms that will help reduce the latency? If you use Linux for VNC, SSH, VoIP or on-line games, you should read this article. Two little-known TCP modifications can reduce latency by several seconds in cases where retransmissions are needed to recover lost data. In this article, I introduce these new techniques that can be enabled per stream or machine-wide without any modifications to the application. I show how these modifications have improved maximum latencies by several seconds in Age of Conan, an MMORPG game by Funcom.

The communication system in Linux provides heaps of configuration options. Still, many users keep them at the default settings, which serves most causes nicely. In some cases, however, the performance experienced by the application can be improved significantly by turning a few knobs.

Most services today use a variant of TCP. In the course of many years, TCP has been optimized for bulk download, such as file transfers and Web browsing. These days, we use more and more interactive applications over the Internet, and many of those rely on TCP, although most traditional TCP implementations handle them badly. For several reasons, they recover lost packets for these applications much more slowly than for download traffic, often longer than is acceptable. The Linux kernel has recently included enhanced system support for interactive services by modifying TCP's packet loss recovery schemes for thin-stream traffic. But, it is up to the developers and administrators to use it.

A large selection of networked interactive applications are characterized by a low packet rate combined with small packet payloads. These are called thin streams. Multiplayer on-line games, IP telephony/audio conferences, sensor networks, remote terminals, control systems, virtual reality systems, augmented reality systems and stock exchange systems are all common examples of such applications, and all have millions of users every day.

Compared to bulk data transfers like HTTP or FTP, thin-stream applications send very few packets, with small payloads, but many of them are interactive and users become annoyed quickly when they experience large latencies. Just how much latency users can accept has been investigated for few applications. ITU-T (International Telecommunication Union's Telecomunication Standarization Sector—a standardization organization) has done it for telephony and audio conferencing and defined guidelines for the satisfactory one-way transmission delay: quality is bad when the delay exceeds 150–200ms, and the maximum delay should not exceed 400ms.

Similarly, experiments show that for on-line games, some latency is tolerable, as long as it does not exceed the threshold for playability. Latency limits for on-line games depend on the game type and ranges from 100ms to 1,000ms. For other kinds of interactive applications, such as SSH shells and VNC remote control, we all know how a lag can be a real pain. It also has been shown that pro-gamers can adapt to larger lag than newbies, but that they are much more annoyed by it.

We had been wondering for a long time how game traffic looked when one saw a lot of streams at once. Could one reduce lag by shaping game traffic into constant-sized TCP streams? Would it be possible to see when avatars interacted?

To learn more about this, we monitored the game traffic from Funcom's Anarchy Online. We captured all traffic from one of the game servers using tcpdump. We soon found that we were asking the wrong questions and analyzed the latencies that players experienced. Figure 1 shows statistics for delay and loss.

In Figure 1a, I have drawn a line at 500ms. It is an estimate of the delay that the majority of players finds just acceptable in a role-playing game like Anarchy. Everybody whose value is above that line probably has experienced annoying lag. The graph shows that nearly half the measured streams during this hour of game play had high-latency events, and that these are closely related to packet losses (Figure 1b). The worst case in this one-hour, one-region measurement is the connection where the user experienced six consecutive retransmissions resulting in a delay of 67 (!) seconds.

Figure 1a. Round-Trip Time vs. Maximum Application Delay (Analysis of Trace from Anarchy Online)

Figure 1b. Per-Stream Loss Rate (Analysis of Trace from Anarchy Online)

The high delays you can see in the previous section stem from the default TCP loss recovery mechanisms. We have experimented with all the available TCP variants in Linux to find the TCP flavor that is best suited for low-latency, thin-stream applications. The result was disheartening: all TCP variants suffer from long retransmission delays for thin-stream traffic.

We wanted to do something about this and implemented several modifications to Linux TCP. Since version 2.6.34, the Linux kernel includes the linear timeouts and the thin fast retransmit modifications we proposed as replacements for the exponential backoff and fast retransmit mechanisms in TCP. The modifications behave normally whenever a TCP stream is not thin and retransmit faster when it is thin. They are sender-side only and, thus, can be used with unmodified receivers. We have tested the mechanisms with Linux, FreeBSD, Mac OS X and Windows receivers, and all platforms successfully receive, and benefit from, the packet recovery enhancements.

TCP streams that are always busy—as they are for downloading—use fast retransmit to recover packet losses. When a sender receives three (S)ACKs for the same segment in a row, it assumes the following segment is lost and retransmits it. Segment interarrival times for thin-stream applications are very high, and in most cases, a timeout will happen before three (S)ACKs can arrive. To deal with this problem, you trigger a fast retransmission when the first duplicate (S)ACK arrives, as illustrated in Figure 2. Even if this causes a few unintended retransmissions, it leads to better latency. The overhead of this modification is minimal, because the thin stream sends very few packets anyway.

Figure 2. Thin Fast Retransmit

When packets are lost and so few (S)ACKs are received by the sender that fast retransmission doesn't work, a timeout is triggered to retransmit the oldest lost packet. This is not supposed to happen unless the network is heavily congested, and the retransmission timer is doubled every time it is triggered again for the same packet to avoid adding too much to the problem. When a stream is thin, these timeouts handle most packet losses simply because the application sends too little data to trigger fast transmissions. TCP doubles the timer, and latency grows exponentially when the same packet is lost several times in a row. When modification is turned on, linear timeouts are enabled when a thin stream is detected (shown in Figure 3). After six linear timeouts, exponential backoff is resumed. A packet still not recovered within this period is most likely dropped due to prevailing heavy congestion, and in that case, the linear timeout modification does not help.

Figure 3. Modified and Standard Exponential Backoff

As the modifications can have a negative effect on bulk data streams (they do trigger retransmissions faster), we have implemented a test in the TCP stack to count the non-ACKed packets of a stream, and then apply the enhanced mechanisms only if a thin stream is detected. A stream is classified as thin if there are so few packets in transit that they cannot trigger a fast retransmission (less than four packets on the wire). Linux uses this “test” to decide when the stream is thin and, thus, when to apply the enhancements. If the test fails (the stream is able to trigger fast retransmit), the default TCP mechanisms are used. The number of dupACKs needed to trigger a fast retransmit can vary between implementations and transport protocols, but RFC 2581 advocates fast retransmit upon receiving the third dupACK. In the Linux kernel TCP-implementation, “packets in transit” is an already-available variable (the packets_out element of the tcp_sock struct), and, thus, the overhead to detecting the thin-stream properties is minimal.

The modifications are triggered dynamically based on whether the system currently identifies the stream as thin, but the mechanisms have to be enabled using switches: 1) system-wide by the administrator using syscontrol or 2) for a particular socket using I/O-control from the application.

Both the linear timeout and the thin fast retransmit are enabled using boolean switches. The administrator can set the net.ipv4.tcp_thin_linear_timeouts and net.ipv4.tcp_thin_dupack switches in order to enable linear timeout and the thin fast retransmit, respectively. As an example, linear timeouts can be configured using sysctl like this:

$ sysctl net.ipv4.tcp_thin_linear_timeouts=1

The above requires sudo or root login or using the exported kernel variables in the /proc filesystem like this:

$ echo "1" >  /proc/sys/net/ipv4/tcp_thin_linear_timeouts

(The above requires root login.)

The thin fast retransmit is enabled in a similar way using the tcp_thin_dupack control. If enabled in this way by the system administrator, the mechanisms are applied to all TCP streams of the machine, but of course, if and only if, the system identifies the stream as thin. In this case, no modifications are required to the sending (or receiving) application.

The thin stream mechanisms also may be enabled on a per-socket basis by the application developer. If so, the programmer must enable the mechanism with I/O-control using the setsockopt system call and the TCP_THIN_LINEAR_TIMEOUTS and TCP_THIN_DUPACK option names. For example:

int flag = 1; 
int result = setsockopt(sock, IPPROTO_TCP, 
                        TCP_THIN_LINEAR_TIMEOUTS,
                        (char *) &flag, sizeof(int));

enables the linear timeouts. The thin fast retransmit is enabled in a similar way using the TCP_THIN_DUPACK option name. In this case, the programmer explicitly tells the application to use the modified TCP at the sender side, and the modifications are applied to the particular application/connection only.

We've successfully tested the thin-stream modifications for many scenarios like games, remote terminals and audio conferencing (for more information, see the thin-stream Web page listed under Resources). The example I use here to show the effect of the modifications is from a game server, a typical thin-stream application.

Funcom enabled the modifications on some of its servers running Age of Conan, one of its latest MMORPG games. The network traffic was captured using tcpdump. The difference in retransmission latency between the modified and the traditional TCP is shown in Figure 4.

During a one-hour capture from one of the machines in the server park, we saw more than 700 players (746 for the traditional and 722 for the modified TCP tests), where about 300 streams in each experiment experienced loss rates between 0.001% and 10%. Figure 4 shows the results from an analysis of the three first retransmissions. Having only one retransmission is fine, also when the modifications are not used. The average and worst-case latencies are still within the bounds of a playable game. However, as the users start to experience second and third retransmissions, severe latencies are observed in the traditional TCP scenario, whereas the latencies in the modified TCP test are significantly lower. Thus, the perceived quality of the game services should be greatly improved by applying the new Linux TCP modifications.

Figure 4. Modified vs. Traditional TCP in Age of Conan. The box shows the upper and lower quartiles and the average values. Maximum and minimum values (excluding outliers) are shown by the drawn line. The plot shows statistics for the first, second and third retransmissions.

If you have a kernel later than 2.6.34, the modifications are available and easy to use when you know about them. Since you now know, turn them on for your interactive thin-stream applications and remove some of the worst latencies that have been annoying you. We're currently digging deeper into thin-stream behavior—watch our blog for updates on how to reduce those latencies further.

Testing and traces by Funcom: Pal Frogner Hansen, Rui Casais and Torbjorn Linddgren. Scientific work: Carsten Griwodz and Pal Halvorsen.