通過我們會用Nginx的upstream做基于http/https端口的7層負載均衡，由于Nginx老版本不支援tcp協定，是以基于tcp/udp端口的四層負載均衡一般用LVS或Haproxy來做。至于4層負載均衡和7層負載均衡的差別，可以參考:http://www.cnblogs.com/kevingrace/p/6137881.html。然而Nginx從1.9.0版本開始，新增加了一個stream子產品，用來實作四層協定的轉發、代理或者負載均衡等，鑒于Nginx在7層負載均衡和web service上的成功，和Nginx良好的架構，stream子產品前景一片光明。
Nginx的stream子產品預設不會自帶安裝，需要編譯安裝的時候手動添加上這個子產品。廢話不多說了，下面介紹下一個自己使用stream做四層負載均衡的案例：
在此之前已經使用Ningx+Keepalived（主從模式）做了7層負載均衡的LB環境，之前編譯的時候，沒有加上這個stream子產品，是以需要後續手動添加該子產品。
由于Nginx的LB已經有業務跑在上面，可以選擇平滑添加stream子產品，并不會對線上業務造成多大影響。步驟如下：
1）現在LB的slave從機上進行平滑添加，然後再将vip切換到從機上，随即在對master主機進行平滑添加該子產品。
2）平滑添加即是重新configure編譯的時候加上–with-stream，接着make。
3）千萬注意，make之後，不要make install，否則會覆寫掉之前的配置！！！
由于本人的LB環境更新了openssl版本，再添加–with-stream重新編譯的時候報了錯誤，具體可參考：
http://www.cnblogs.com/kevingrace/p/8058535.html
檢查下，發現nginx沒有安裝stream子產品
[[email protected] ~]# /data/nginx/sbin/nginx -V
nginx version: nginx/1.12.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
built with OpenSSL 1.1.0g 2 Nov 2017
TLS SNI support enabled
configure arguments: --prefix=/data/nginx --user=www --group=www --with-http_ssl_module --with-http_flv_module
–with-http_stub_status_module --with-http_gzip_static_module --with-pcre
操作之前，一定要備份一下之前的nginx安裝目錄，防止操作失敗進行復原！
[[email protected] ~]# cp -r /data/nginx /mnt/nginx.bak
之前的編譯指令是：
[[email protected] vhosts]# cd /data/software/nginx-1.12.2
[[email protected] nginx-1.12.2]# ./configure --prefix=/data/nginx --user=www --group=www --with-http_ssl_module
–with-http_flv_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre
現在需要手動添加stream，編譯指令如下：
[[email protected] vhosts]# cd /data/software/nginx-1.12.2
[[email protected] nginx-1.12.2]# ./configure --prefix=/data/nginx --user=www --group=www --with-http_ssl_module
–with-http_flv_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream
[[email protected] nginx-1.12.2]# make
[[email protected] nginx-1.12.2]# cp -f /data/software/nginx-1.12.2/objs/nginx /data/nginx/sbin/nginx
[[email protected] nginx-1.12.2]# pkill -9 nginx
[[email protected] nginx-1.12.2]# /data/nginx/sbin/nginx
檢查下，發現nginx已經安裝了stream子產品了
[[email protected] nginx-1.12.2]# /data/nginx/sbin/nginx -V
nginx version: nginx/1.12.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC)
built with OpenSSL 1.1.0g 2 Nov 2017
TLS SNI support enabled
configure arguments: --prefix=/data/nginx --user=www --group=www --with-http_ssl_module --with-http_flv_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream --with-openssl=/usr/local/ssl
======================================================================================================================
stream的4層負載均衡和upstream的7層負載均衡可以共同配置在nginx中，stream子產品用法和http子產品差不多，關鍵的是文法幾乎一緻。
具體如下：
[[email protected] ~]# cat /data/nginx/conf/nginx.conf
user www;
worker_processes 8;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 65535;
}
stream {
upstream kk5 {
server 10.0.58.22:5100;
server 10.0.58.23:5100;
}
upstream kk5http {
server 10.0.58.22:8000;
server 10.0.58.23:8000;
}
upstream kk5https {
server 10.0.58.22:8443;
server 10.0.58.23:8443;
}
server {
listen 5100;
proxy_connect_timeout 1s;
proxy_pass kk5;
}
server {
listen 8000;
proxy_connect_timeout 1s;
proxy_pass kk5http;
}
server {
listen 8443;
proxy_connect_timeout 1s;
proxy_pass kk5https;
}
}
http {
include mime.types;
default_type application/octet-stream;
charset utf-8;
######
## set access log format
######
log_format main '$remote_addr $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'$http_user_agent $http_x_forwarded_for $request_time $upstream_response_time $upstream_addr $upstream_status';
#######
## http setting
#######
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
proxy_cache_path /var/www/cache levels=1:2 keys_zone=mycache:20m max_size=2048m inactive=60m;
proxy_temp_path /var/www/cache/tmp;
fastcgi_connect_timeout 3000;
fastcgi_send_timeout 3000;
fastcgi_read_timeout 3000;
fastcgi_buffer_size 256k;
fastcgi_buffers 8 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;
#
client_header_timeout 600s;
client_body_timeout 600s;
client_max_body_size 50m;
client_max_body_size 100m;
client_body_buffer_size 256k;
gzip on;
gzip_min_length 1k;
gzip_buffers 4 16k;
gzip_http_version 1.1;
gzip_comp_level 9;
gzip_types text/plain application/x-javascript text/css application/xml text/javascript application/x-httpd-php;
gzip_vary on;
## includes vhosts
include vhosts/*.conf;
}
[[email protected] ~]# cd /data/nginx/conf/vhosts/
[[email protected] vhosts]# ls
-rw-r-xr-- 1 root root 889 12月 26 15:18 bpm.kevin.com.conf
-rw-r-xr-- 1 root root 724 12月 26 14:38 mobi.kevin.com.conf
[[email protected] vhosts]# cat bpm.kevin.com.conf
upstream os-8080 {
ip_hash;
server 10.0.58.20:8080 max_fails=3 fail_timeout=15s;
server 10.0.58.21:8080 max_fails=3 fail_timeout=15s;
}
server {
listen 80;
server_name bpm.kevin.com;
access_log /data/nginx/logs/bpm.kevin.com-access.log main;
error_log /data/nginx/logs/bpm.kevin.com-error.log;
location / {
proxy_pass http://os-8080;
proxy_set_header Host $host;
proxy_redirect http://os-8080/ http://bpm.kevin.com/;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
[[email protected] vhosts]# cat mobi.kevin.com.conf
upstream mobi_cluster{
server 10.0.54.20:8080;
}
server {
listen 80;
server_name mobi.kevin.com;
access_log /data/nginx/logs/mobi.kevin.com-access.log main;
error_log /data/nginx/logs/mobi.kevin.com-error.log;
location / {
proxy_pass http://mobi_cluster;
proxy_set_header Host $host;
proxy_redirect http://mobi_cluster/ http://mobi.kevin.com/;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
關閉防火牆，否則要依次打開如上配置中的端口！
[[email protected] vhosts]# /data/nginx/sbin/nginx -s reload
重新開機nginx後，發現http端口80、8080、8000、8443都起來了（lsof指令可以檢視到），而tcp/udp端口5100沒有起來，這是正常的。
==順便總結下Nginx的四大子產品——proxy、headers、upstream、stream子產品=
一、ngx_http_proxy_module子產品
1）proxy_pass URL;
Context: location, if in location, limit_except
注意：proxy_pass後面的路徑不帶uri時，其會将location的uri傳遞給後端主機
server {
…
server_name HOSTNAME;
location /uri/ {
proxy http://hos[:port];
}
…
}
http://HOSTNAME/uri –> http://host/uri
proxy_pass後面的路徑是一個uri時，其會将location的uri替換為proxy_pass的uri
server {
…
server_name HOSTNAME;
location /uri/ {
proxy http://host/new_uri/;
}
…
}
http://HOSTNAME/uri/ –> http://host/new_uri/
如果location定義其uri時使用了正規表達式的模式，則proxy_pass之後必須不能使用uri; 使用者請求時傳遞的uri将直接附加代理到的服務的之後
server {
…
server_name HOSTNAME;
location |* /uri/ {
proxy http://host;
}
…
}
http://HOSTNAME/uri/ –> http://host/uri/
2）proxy_set_header field value;
設定發往後端主機的請求封包的請求首部的值；Context: http, server, location
proxy_set_header X-Real-IP $remote_addr;
$remote_addr：記錄的是上一台主機的IP，而上一台主機有可能也是代理伺服器
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
$proxy_add_x_forwarded_for：記錄的是源IP位址
在http用戶端還有修改/etc/httpd/conf/httpd.conf檔案
LogFormat “%{X-Real-IP}i %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combined
通過上述方法則可以在後端主機上記錄真實的httpd資源請求者，而不再是隻記錄前端代理伺服器的IP位址
3）proxy_cache_path
定義可用于proxy功能的緩存；Context: http
proxy_cache_path path [levels=levels] [use_temp_path=on|off] keys_zone=name:size [inactive=time]
[max_size=size] [manager_files=number] [manager_sleep=time] [manager_threshold=time] [loader_files=number]
[loader_sleep=time] [loader_threshold=time] [purger=on|off] [purger_files=number] [purger_sleep=time] [purger_threshold=time];
proxy_cache_path /var/cache/nginx/proxy_cache levels=1:2:1 keys_zone=gmtest:20M max_size=1G;
4）proxy_cache zone | off;
指明要調用的緩存，或關閉緩存機制；Context: http, server, location
proxy_cache gmtest;
5）proxy_cache_key string;
緩存中用于“鍵”的内容；
預設值：proxy_cache_key s c h e m e scheme schemeproxy_host$request_uri;
建議定義成方法和url
6）proxy_cache_valid [code …] time;
定義對特定響應碼的響應内容的緩存時長；
定義在http{…}中；
proxy_cache_path /var/cache/nginx/proxy_cache levels=1:1:1 keys_zone=gmtest:20m max_size=1g;
定義在需要調用緩存功能的配置段，例如server{…}，或者location中；
proxy_cache gmtest;
proxy_cache_key $request_uri;
proxy_cache_valid 200 302 301 1h;
proxy_cache_valid any 1m;
7）proxy_cache_use_stale
proxy_cache_use_stale error | timeout | invalid_header | updating | http_500 | http_502 | http_503 | http_504 | http_403
| http_404 | off …;
Determines in which cases a stale cached response can be used when an error occurs during communication with the proxied server.
後端伺服器的故障在那種情況下，就使用緩存的功能對客戶的進行傳回
8）proxy_cache_methods GET | HEAD | POST …;
If the client request method is listed in this directive then the response will be cached. “GET” and “HEAD” methods are always
added to the list, though it is recommended to specify them explicitly.
預設方法就是GET HEAD方法
9）proxy_hide_header field;
By default, nginx does not pass the header fields “Date”, “Server”, “X-Pad”, and “X-Accel-…” from the response of a proxied server
to a client. The proxy_hide_header directive sets additional fields that will not be passed.
10）proxy_connect_timeout time;
Defines a timeout for establishing a connection with a proxied server. It should be noted that this timeout cannot usually exceed 75
seconds.
預設為60s
11）buffer相關的配置
a：proxy_buffer_size size;
Sets the size of the buffer used for reading the first part of the response received from the proxied server. This part usually contains
a small response header. By default, the buffer size is equal to one memory page.
預設為4k|8k
b：proxy_buffering on | off;
Enables or disables buffering of responses from the proxied server.
預設為on
c：proxy_buffers number size;
Sets the number and size of the buffers used for reading a response from the proxied server, for a single connection. By default, the buffer size is equal to one memory page.
預設為8 4k|8k
d：proxy_busy_buffers_size size;
When buffering of responses from the proxied server is enabled, limits the total size of buffers that can be busy sending a response to the client while the response is not yet fully read.
預設為8k|16k
二、ngx_http_headers_module子產品
The ngx_http_headers_module module allows adding the “Expires” and “Cache-Control” header fields, and arbitrary fields,
to a response header.
向由代理伺服器響應給用戶端的響應封包添加自定義首部，或修改指定首部的值；
1）add_header name value [always];
添加自定義首部；
add_header X-Via $server_addr;
經由的代理伺服器位址
add_header X-Accel $server_name;
2）expires [modified] time;
expires epoch | max | off;
用于定義Expire或Cache-Control首部的值；
可以把伺服器定義的緩存時長修改了；
三、ngx_http_upstream_module子產品
The ngx_http_upstream_module module is used to define groups of servers that can be referenced by the proxy_pass, fastcgi_pass,
uwsgi_pass, scgi_pass, and memcached_pass directives.
1）upstream name { … }
定義後端伺服器組，會引入一個新的上下文；Context: http
upstream httpdsrvs {
server …
server…
…
}
2）server address [parameters];
在upstream上下文中server成員，以及相關的參數；Context: upstream
address的表示格式：
unix:/PATH/TO/SOME_SOCK_FILE
IP[:PORT]
HOSTNAME[:PORT]
parameters：
weight=number
權重，預設為1；預設算法是wrr
max_fails=number
失敗嘗試最大次數；超出此處指定的次數時，server将被标記為不可用
fail_timeout=time
設定将伺服器标記為不可用狀态的逾時時長
max_conns
目前的伺服器的最大并發連接配接數
backup
将伺服器标記為“備用”，即所有伺服器均不可用時此伺服器才啟用
down
标記為“不可用”
先在nginx前端配置down，然後在下架後端伺服器，上架新的web程式，然後上架，在修改配置檔案立馬的down
3）least_conn;
最少連接配接排程算法，當server擁有不同的權重時其為wlc
要在後端伺服器是長連接配接時，效果才好，比如mysql
4）ip_hash;
源位址hash排程方法
5）hash key [consistent];
基于指定的key的hash表來實作對請求的排程，此處的key可以直接文本、變量或二者的組合
作用：将請求分類，同一類請求将發往同一個upstream server
If the consistent parameter is specified the ketama consistent hashing method will be used instead.
示例：
hash $request_uri consistent;
hash $remote_addr;
hash $cookie_name; 對同一浏覽器的請求，發往同一個upstream server
6）keepalive connections;
為每個worker程序保留的空閑的長連接配接數量
nginx的其它的二次發行版：
tengine
OpenResty
1.9版本之後可以反代tcp／udp的協定，基于stream子產品，工作與傳輸層
四、ngx_stream_core_module子產品
模拟反代基于tcp或udp的服務連接配接，即工作于傳輸層的反代或排程器
1）stream { … }
定義stream相關的服務；Context: main
stream {
upstream sshsrvs {
server 192.168.22.2:22;
server 192.168.22.3:22;
least_conn;
}
server {
listen 10.1.0.6:22022;
proxy_pass sshsrvs;
}
}
stream子產品中管的upstream子產品的用法同上
2）listen
listen address:port [ssl] [udp] [proxy_protocol] [backlog=number] [bind] [ipv6only=on|off] [reuseport]
[so_keepalive=on|off|[keepidle]:[keepintvl]:[keepcnt]];
![修改or隐藏nginx版本号[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)