- Paper
- Introduction
- Participants
- Installation
- System Configuration
- Usage
- Suitable Scenarios
- [How To Use] (#how-to-use)
- Demo Server
- Evaluation
- [Nginx] (#nginx)
- [HAProxy] (#haproxy)
- Online Evaluation
- New Features
- Contacts
ASPLOS 2016
http://dl.acm.org/citation.cfm?id=2872391
With a rapid growth of NIC bandwidth and CPU cores on one single machine, a scalable TCP network stack is performance-critical. However, stock Linux kernel does not scale well when CPU core number is above 4. It is even worse that the throughput could collapse when there are more than 12 CPU cores.
Fastsocket is a scalable kernel TCP socket implementation and achieves a straight linear performance growth when scaling up to 24 CPU cores. Meanwhile, The underlying kernel optimization of Fastsocket is transparent for socket applications, which means existing applications can take advantage of Fastsocket without changing their codes.
Currently Fastsocket is implemented in the Linux kernel(kernel-2.6.32-431.29.2.el6.x86_64) of CentOS-6.5 which is the latest version of redhat EL6, since CentOS-6.5 is our major production environment system. According to our evaluations, Fastsocket increases throughput of Nginx and HAProxy(measured by connections per second) by 290% and 620% on a 24-core machine, compared to the base CentOS-6.5 kernel.
Moreover, Fastsocket can further exploit more from the hardware:
- With Fastsocket, Hyper-Threading can make an extra 20% performance increase.
- With Fastsocket, NIC that support Flow-Director(like Intel 82599) can increase the throughput by 15% if the server works as a proxy(like HAProxy).
Fastsocket (V1.0) has already been deployed in the SINA production environment. Fastsocket is used with HAProxy to provide HTTP load balance service and has been running stably since March 2014 More details are in the Evaluation.
Fastsocket is released under GPLv2 and we promise that we would never ask for any payment to use our codes.
- Fastsocket is started and mainly developed by Xiaofeng Lin from the OS team of SINA.COM.
- OS center of Tsinghua is cooperating closely with SINA.COM to further enhance Fastsocket.
- Fastsocket is also supported by Intel with extensive assistance.
- Zeuux Community is providing open source consulting for Fastsocket.
The source code is available at https://github.com/fastos/fastsocket.git. Clone the repository by:
[root@localhost ~]# git clone https://github.com/fastos/fastsocket.git
Here is a brief introduction to the directories in the repository.
- kernel - source code of the Fastsocket customized kernel
- module - source code of the Fastsocket kernel module
- library - source code of user-level library to support Fastsocket
- scripts - scripts to set NIC and system environment(NOT compulsory for Fastsocket)
- demo - source code of a demo server to demonstrate performance of Fastsocket
The following commands will build and install the kernel after Fastsocket repository is downloaded from git. You can customize the config file if you are sure you will not miss some important component. Fastsocket can be built smoothly on 64-bit CentOS-6.X systems. Problems may arise on 32-bit systems and CentOS-7 systems.
[root@localhost ~]# cd fastsocket/kernel
[root@localhost kernel]# make defconfig
[root@localhost kernel]# make
[root@localhost kernel]# make modules_install
[root@localhost kernel]# make install
Enter the library directory and make the library:
[root@localhost fastsocket]# cd library
[root@localhost library]# make
After that, libfsocket.so is created in the same directory.
When the installation is done, remember to modify grub file to switch to the Fastsocket kernel and reboot the system.
After booting into the kernel with Fastsocket, load the Fastsocket module with default parameters:
[root@localhost ~]# modprobe fastsocket
For more detailed information of modules parameters, please refer to Module.
Two ways to check if the module is loaded successfully.
-
Check lsmod:
[root@localhost ~]# lsmod | grep fastsocket fastsocket 23145 0 -
Check dmesg:
[root@localhost ~]# dmesg | tail Fastsocket: Load Module Fastsocket: Enable Listen Spawn[Mode-2] Fastsocket: Enable Recieve Flow Deliver Fastsocket: Enable Fast Epoll
Run nic.sh provided in the scripts directory of the repository to take care of remaining configuration.
[root@localhost ~]# cd fastsocket
[root@localhost fastsocket]# scripts/nic.sh -i eth0
eth0 is the interface to be used and should be changed according to your system configuration. The script will automatically check system and NIC parameters, then configures various features.
If you are interested in how nic.sh works, please refer to Scripts.
Generally, scenarios meeting the following conditions will benefit the most from Fastsocket (V1.0):
- The machine has no less than 8 CPU cores.
- Large portion of the CPU cycles is spent in network softirq and socket related system calls.
- Short TCP connections are heavily used.
- Application uses non-blocking IO over epoll as the IO framework.
- Application uses multiple processes to accept connections individually.
Meanwhile, we are developing Fastsocket to improve the network stack performance in more general scenarios. You can refer to New Features.
Fastsocket is enabled by preloading a shared library named libfsocket.so when launching an application. For example, ngnix can be started with Fastsocket by:
[root@localhost fastsocket]# cd library
[root@localhost library]# LD_PRELOAD=./libfsocket.so nginx
Without the preloaded library, applications can run as if they are on the original kernel, which provides a super quick rollback in case there is a need.
[root@localhost ~]# nginx
For more information about the library, please refer to Library.
Here we list a few applications that are working fine with Fastsocket:
- haproxy
- nginx (Do disable accept mutex)
- lighttpd
We are also using Fastsocket on the load generators in our benchmark tests. This is very helpful since Fastsocket greatly increases the maximum work load that could be generated from a single machine, which saves machines and operations. These load generators are:
- ab
- http_load
We provide a demo server in the demo directory of the repository. The demo server does nothing but read/write messages from/to network sockets and is purely used to study and benchmark the performance of network stack of Linux kernel. When the demo server is running, it has little user CPU consumption, which makes it a perfect network application to observe the network stack performance.
Moreover, it is also used to demonstrate the scalability and performance improvement of Fastsocket over the base Linux kernel.
For more information about the demo server, please refer to Demo.
Some important configurations:
- Worker number is set to the number of CPU cores.
- HTTP Keep-alive is disabled on Nginx for a short connection test.
- Http_load fetches a 64 bytes static file from Nginx with a concurrency of 500 multiplied by the number of cores.
- We enable memory cache for that static file in order to rule out any disk affection.
- accept mutex is disabled. The default value of nginx is on.
Note: YOU'D BETTER DO DISABLE accept_mutex! With default Fastsocket module parameters, Fastsocket has partioned listen socket, therefore, there is no need to force user to accept connections one by one. If some cpu had no chance to receive packet especially the tcp syn packet by RPS or others, the nginx would fail to accept the new conn forever with accept_mutex enabled. So you should make sure the new request could be passed to every CPU, if you want load blance the accept with accept mutex of nginx.
From the figure below, Fastsocket on 24 CPU cores achieves 475K connection per second (cps), with a speed up of 21X. The throughput of base CentOS-6.5 kernel increases non-linearly up to 12 CPU cores and drops dramatically to 159K cps with 24 CPU cores. The latest 3.13 kernel doubles the throughput to 283K cps when using 24 CPU cores compared with the base CentOS-6.5 kernel. However, it has not completely solved the scalability bottlenecks, preventing it from scaling beyond 12 CPU cores.
Some important configurations:
- Worker number is set to the number of CPU cores.
- RFD(Receive Flow Deliver) in Fastsocket is enabled.
- HTTP Keep-alive is disabled on HAProxy for a short connection test.
- A client runs http_load with a concurrency of 500 multiplied by number of cores.
- A backend server responds each incoming HTTP request with a 64 bytes message.
As shown in the same figure, Fastsocket presents an excellent scalability performance, which is very similar to the previous Nginx case. Fastsocket outperforms Linux 3.13 by 139K cps and base CentOS-6.5 kernel by 370K cps when using 24 CPU cores, though the one core throughputs are very close among all the three kernels.
As mentioned before, Fastsocket has already been deployed in the SINA production environment. One typical scenario is using Fastsocket with HAProxy to provide HTTP load balance service to WEIBO and other SINA productions.
In the figure below, it is the CPU utilization of an 8-core servers within 24 hours. Figure (a) shows the CPU utilization before deploying Fastsocket and figure (b) shows the CPU utilization after deploying Fastsocket.
We can see from the figure, what happened after Fastsocket is used:
- The load of each CPU core is perfect balanced.
- The average CPU utilization of all CPU cores is reduced by 10%.
- As a result, the effective capacity of the HAProxy server is increased by 85%.
Moreover, since the server is an old 8-core machine, we expect Fastsocket would make more performance improvement when Fastsocket is deployed on a machine with more CPU cores (It is already observed on a 12-core machine after updating Fastsocket).
We are now improving network stack efficiency in the case of long TCP connection. Four more features are introduced:
- Direct-TCP: Skip the route process when receiving packets if these packets belong to upper TCP sockets.
- Skb-Pool: Get skb from per-core pre-allocated skb pool instead of kernel slab.
- Receive-CPU-Select: Steer a packet to a CPU core where application is waiting for it. The idea is similar with RFS from Google, however, it is lighter and more accurate.
- RPS-Framework: We extend the idea of RPS that is to redispatch the receiving packets before they entering the network stack. We build a framework where developers can implement their own packets-redispatching rules in out-of-tree module and hook into the RPS framework.
We evaluated our new work on redis which is a typical and popular key-value cache application.
Some important configurations:
- Redis works in persistent TCP connection mode.
- Multiple Redis instances are set up.
- Each Redis instance listens on a different port and binds to a different CPU core.
The 8-redis-instance test shows:
- With commodity NIC supporting RSS, Fastsocket improves the throughput by more than 20%.
- WIth advanced NIC supporting Flow-Director(Intel 82599), a 45% improvement can be reached.
Notes:
- These new features are in the experimental stage, neither well tuned for performance, nor proved stable by long-time production environment running.
- There new features are complementary to the features in V1.0, therefore, Nginx and HAProxy performance can be further increased by Fastsocket with these new features.
Mailing-list: fastsocket@librelist.com
Google Group: fastsocket-dev (https://groups.google.com/forum/#!forum/fastsocket-dev)
Sending a mail to the address above will subcribe to the mailing list. The subject and message do not matter.