在阿里云ECS上自建LVS负载均衡

我们需要4台阿里云的ECS机器,这里选用网络增强型4核16G,并开通HAVIP功能,点这里申请。

我们假设如下内网IP环境:

  • LVS服务器,172.20.3.59,绑定VIP172.20.3.10
  • 3台Nginx服务器,172.20.3.60 ~ 62

1 配置Ngxin服务器

在60 ~ 62上分别执行

apt-get install -y nginx

修改文件

vim /var/www/html/index.nginx-debian.html

<h1>server 2</h1>
<h1>server 3</h1>
<h1>server 4</h1>

2 配置LVS

以下在59这台机器上操作

modprobe ip_vs
lsmod | grep ip_vs
ip_vs                 155648  0
nf_conntrack          139264  1 ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
libcrc32c              16384  2 nf_conntrack,ip_vs

apt-get update && apt-get install -y ipvsadm

# 必须开启
sysctl -w net.ipv4.ip_forward=1
# 可选优化
sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.somaxconn=32768

# 清理
ipvsadm -C
# 这里用性能最差的也是最简单的full nat模式,lc是最小连接,也可以用rr
ipvsadm -A -t 172.20.3.10:80 -s lc 
ipvsadm -a -t 172.20.3.10:80 -r 172.20.3.60:80 -m
ipvsadm -a -t 172.20.3.10:80 -r 172.20.3.61:80 -m
ipvsadm -a -t 172.20.3.10:80 -r 172.20.3.62:80 -m
ipvsadm -Ln

接着测试一下:

curl "172.20.3.10" | grep "server"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3372  100  3372    0     0  3292k      0 --:--:-- --:--:-- --:--:-- 3292k
 <h1> server 2</h1>
root@h001:~# curl "172.20.3.10" | grep "<h1>"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3372  100  3372    0     0  3292k      0 --:--:-- --:--:-- --:--:-- 3292k
 <h1> server 3</h1>

已经随机,负载均衡了!

3 性能测试

压单机,8.7w

./wrk -t 500 -c 1000 -d 15s http://172.20.3.60
Running 15s test @ http://172.20.3.60
  500 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   100.31ms  172.58ms   1.67s    82.94%
    Req/Sec   252.12    145.94     1.89k    54.74%
  1314168 requests in 15.10s, 1.05GB read
  Socket errors: connect 0, read 0, write 0, timeout 26
Requests/sec:  87016.94
Transfer/sec:     71.28MB

压lvs,25万

./wrk -t 500 -c 1000 -d 15s http://172.20.3.10
Running 15s test @ http://172.20.3.10
  500 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.79ms    5.11ms  68.18ms   81.08%
    Req/Sec   346.12    248.99     8.24k    91.41%
  3842848 requests in 15.10s, 3.07GB read
Requests/sec: 254473.39
Transfer/sec:    208.45MB

基本是线性的,而且毫无压力,尚未到lvs上限!

4 性能优化

在/etc/modprobe.d/目录下添加文件ip_vs.conf,内容为:

options ip_vs conn_tab_bits=20

修改超时时间

ipvsadm --set  900 60 300

禁用ARP

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

 

Leave a Reply

Your email address will not be published.