FastDFS深入應用詳解及分布式叢集配置

起始

• 從應用層面詳解fastdfs各元件

• fastdfs的多伺服器場景使用及部署配置說明

相關的文章

1、單體安裝教程 https://blog.csdn.net/suoyanming/article/details/88797360

2、開源中國fastdfs首頁 p/fastdfs

3、github首頁（不确定是否是原作者維護） happyfish100/fastdfs

4、對fastdfs-nginx-module 實作原理講的非常清楚 https://www.cnblogs.com/littleatp/p/4361318.html     

一、FastDFS

1、FastDFS是一個開源的輕量級分布式檔案系統，它對檔案進行管理，功能包括：檔案存儲、檔案同步、檔案通路（檔案上傳、檔案下載下傳）等，解決了大容量存儲和負載均衡的問題。特别适合以檔案為載體的線上服務，如相冊網站、視訊網站等等。

2、FastDFS為網際網路量身定制，充分考慮了備援備份、負載均衡、線性擴容等機制，并注重高可用、高性能等名額，使用FastDFS很容易搭建一套高性能的檔案伺服器叢集提供檔案上傳、下載下傳等服務。

二、深入認識FastDFS

•      任何一個中間件的應用，都必須深入了解該中間件内部各元件的承擔的功能角色、運作機制，能深入了解各元件的實作原理更好。這樣才能靈活應對實際應用場景、多變的業務需求、生産環境應急等問題，快速實施架構調整。
•      我們一直在使用FastDFS作為圖檔檔案資料庫，部署架構為單體（即：一個tracker、一個storage、一個group）,由于本次用于部署fastdfs的伺服器硬碟空間報警，當務之急必須更改fastdfs部署架構，擴充存儲。
•      下面從項目總體情況、tracker 、storage、fastdfs-nginx-module 、group 元件詳細說明其功能角色及運作機制

1、項目總體情況

•        fastdfs是開源的項目
•        通過github源碼可看出，該項目是基于C語言開發的
•        fastdfs是基于作業系統OS的檔案管理系統功能之上進行分布式檔案管理（Linux、FreeBSD等），通過看檔案在硬碟的儲存方式也可以得出
•        提供C、Java和PHP API接口

2、tracker跟蹤器

•  主要做排程工作， 起負載均衡的作用

•  在記憶體中記錄叢集中所有存儲組group和存儲伺服器storage 的狀态資訊, 是用戶端和資料伺服器互動的樞紐

• tracker的核心工作内容：

          （1） 記錄叢集中有多少個group（group1\group2....）

            (2)  每個group 分布在那個幾個storage上，以及storage所在機器的ip，端口等資訊，group之間的同步由tracker 和storage一起完成（後面細講）

          （3）如果同一個group 存在多個storage, 而這些storage又被分布在一台或多台機器上，那麼對該group上傳或讀取檔案具體落到那個機器上（即那個storage）？（有點繞）

                   tracker完美的解決了這個問題，即對分布式部署架構下：多group、多storage的上傳和下載下傳做負載均衡政策，通過配置tracker.conf可實作具體負載均衡政策

            (4)  tracker 可部署多台，多個tracker在伺服器記憶體中記錄的資訊是一樣的，通過nginx對tracker做負載均衡，以提高并發性能及容災能力

          （5）tracker 不去主動讀取storage的相關資訊，而是由storage主動推送給tracker （這也是為什麼必須先啟動tracker的原因） 

          （6）以下圖檔摘自網上 ： 上傳檔案過程 、下載下傳檔案過程，通過圖檔可以看到，tracker的核心工作是為用戶端找到一個storage, client用戶端和storage進行上傳下載下傳通信。

• tracker.conf  (在分布式部署架構下，通過tracker負載均衡給client端傳回特定storage資訊，而負載均衡的政策配置主要在tracker.conf中)

•  1）核心參數配置說明

（1）disabled=false

         #配置檔案是否失效

         # is this config file disabled

        # false for enabled

        # true for disabled disabled=false

        # is this config file disabled # false for enabled # true for disabled

（2）port=22122

           #服務端口

           # the tracker server port

（3） base_path=/data/fastdfs/tracker

             # 存放track 資料及日志檔案目錄 

             # the base path to store data and log files

（4）work_threads=4

          #時線程數：一般和cpu的個數設為同一個值         

          # work thread count, should <= max_connections

          # default value is 4

           # since V2.00

（4）（重要） store_lookup=1 

             #  上傳檔案選擇哪個一個group 的 政策：0：輪詢；1:指定組  ； 2： 負載均衡，選擇剩餘存儲空間最大的組group 上傳檔案

              # the method of selecting group to upload files

             # 0: round robin

             # 1: specify group

            # 2: load balance, select the max free space group to upload file

（5）（重要） store_group=group2

        # 當  store_lookup=1  時，該配置有效，指定存儲的組名

         # which group to upload file

         # when store_lookup set to 1, must set store_group to the group name

（6）（重要） store_server=0

       # 應用場景： 存在多個相同的組，例如group1 ， 在多個storage 伺服器上 例如：192.168.0.171 、，

                            當上傳檔案時優先選擇那個storage的政策配置：~ 0：輪詢 ；1：按ip升序排序後選擇第一個ip，即最小的那個ip (192.168.0.164)；2：按優先級排列的第一個伺服器順序，數字越小優先級越 高，storage伺服器的優先在storage.conf中配置 upload_priority 參數

    # which storage server to upload file

       # 0: round robin (default)

        # 1: the first server order by ip address

        # 2: the first server order by priority (the minimal)

（7）（重要）  store_path=0

       # 應用場景：選擇具體一個組的那一條存儲路徑（一個group有多條存儲路徑，一般一個伺服器有兩塊大硬碟挂載到了兩個路徑下，專門用來存放檔案），~0：輪詢，2：負載均衡，選擇剩餘空間最大的路徑 

         （邏輯~重要） 通過上面的配置參數确定了3件事的基礎上，該配置才會起作用：

        （1）确定了要存儲在那個group上，例如group1；（2）确定上傳檔案要儲存在那一台storage 中的group1，假如是192.168.0.171；（3） 此時如果 192.168.0.171 上的storage server中group1 有兩個存儲路徑，即store_path0,store_path1,(對應的檔案路徑即M00,M01)

  # which path(means disk or mount point) of the storage server to upload file

       # 0: round robin

      # 2: load balance, select the max free space path to upload file

（8）download_server=0

        # 下載下傳檔案時存儲伺服器的選擇政策； 應用場景：要下載下傳的檔案所在group 存在多個storage 伺服器上， ~0：輪詢  ；1：目前檔案上載到的源存儲伺服器

        # which storage server to download file

        # 0: round robin (default)

       # 1: the source storage server which the current file uploaded to

（9）reserved_storage_space = 10%

    # 給系統或其他應用程式預留存儲空間設定 

   #（重要）場景：某一個group所在某一個storage伺服器（可能存在多個伺服器上）剩餘的存儲空間小于等于這個閥值時，則檔案不能被儲存，即使該group的其他storage伺服器還有很大的存儲空間

   # reserved storage space for system or other applications.

   # if the free(available) space of any stoarge server in

   # a group <= reserved_storage_space,

   # no file can be uploaded to this group.

   # bytes unit can be one of follows:

   ### G or g for gigabyte(GB)

   ### M or m for megabyte(MB)

   ### K or k for kilobyte(KB)

   ### no unit for byte(B)

   ### XX.XX% as ratio such as reserved_storage_space = 10%

3、storage 存儲伺服器

• storage 定期向tracker發送心跳，報告自己的狀态，tracker會将同組的 storage server資訊傳回給storage （該部分邏輯後面再細講） 

• tracker不負責具體的檔案上傳、下載下傳實作，這些都是有storage完成的

• storage儲存檔案和檔案的屬性

• storage server是基于作業系統的檔案管理系統進行檔案管理的（上面有提到）

• group之間檔案同步由storage server 和tacker server一起完成的（該部分邏輯後面再細講） 

• storage server的狀态（7個）

• FDFS_STORAGE_STATUS_INIT :初始化，尚未得到同步已有資料的源伺服器

• FDFS_STORAGE_STATUS_WAIT_SYNC :等待同步，已得到同步已有資料的源伺服器

• FDFS_STORAGE_STATUS_SYNCING :同步中

• FDFS_STORAGE_STATUS_DELETED :已删除，該伺服器從本組中摘除（注：本狀态的功能尚未實作）

• FDFS_STORAGE_STATUS_OFFLINE :離線

• FDFS_STORAGE_STATUS_ONLINE :線上，尚不能提供服務

• FDFS_STORAGE_STATUS_ACTIVE :線上，可以提供服務

storage.conf 核心參數配置

（1）port=23000

      # storage 服務端口

     # the storage server port

（2）base_path=/usr/local/fastdfs/fdfs_storage

      #存放storage 服務的資料和日志

      # the base path to store data and log files

（3）store_path0=/usr/local/fastdfs/fdfs_storage

      # 存儲路徑配置，可以配置多個，對應的  store_path_count=1 參數需要累加

      # store_path#, based 0, if store_path0 not exists, it's value is base_path

      # the paths must be exist

      #store_path1=/home/yuqing/fastdfs2

（4）tracker_server=192.168.0.171:22122

     #tracker 服務的 ip和端口， ip替換為域名也可以，可以配置多個 

     # tracker_server can ocur more than once, and tracker_server format is

     #  "host:port", host can be hostname or ip address

（5）file_distribute_path_mode=0

   # 分布式存儲檔案政策： 當storage下有多個存儲路徑時，該配置起作用  ~  # 0: 輪詢     # 1: 根據檔案名hash結果随機存儲

   # the mode of the files distributed to the data path

   # 0: round robin(default)

   # 1: random, distributted by hash code  

（6）upload_priority=10 （在tracker.conf 中有提到）

# 上傳檔案事，同組内的storage 伺服器優先級設定，且當 tracker.conf 中store_server= 2時 起作用，值越小，優先級越高。

# the priority as a source server for uploading file.

# the lower this value, the higher its uploading priority.

# default value is 10

4、group

•  group 分組是fastdfs應對大流量應用系統中處理高并發、高容災的經典設計，并且group還起到了應用隔離的功能
•  一個group可以存在多個storage中（在storage中也可以提到）
• 根據client端的請求配置設定到不同的group，檔案系統具備直接的負載均衡；
• group内有storage服務節點壞掉時，需從其他group内恢複資料  

5、 fastdfs-nginx-module 

• fastdfs 中storage、tracker 均提供的http服務，可以直接下載下傳檔案，但考慮到性能及負載實作難易度的問題，一般都用web伺服器來下載下傳檔案，例如nginx、apache
• fastdfs-nginx-module 就是fastdfs基于ngnix實作檔案http傳輸的元件，以nginx module的方式添加到nginx 程式中
• 每個storage 均需安裝  fastdfs-nginx-module 、Nginx ，目前storage找不到檔案時，向源storage主機發起redirect重定向或proxy轉發代理動作
• fastdfs-nginx-module 安裝後目錄結構如下圖

      說明及圖檔 摘自：https://www.cnblogs.com/littleatp/p/4361318.html        

          (1)ngx_http_fastdfs_module.c    ~ nginx 子產品接口實作檔案，用于向nginx 接入fastdfs-module核心子產品邏輯

       （2）common.c   ~ fastdfs-module核心子產品，實作了初始化、檔案下載下傳的主要邏輯

       （3）config     ~ 編譯子產品所用的配置，裡面定義了一些重要的常量，調用fastdfs基礎元件功能，以及擴充配置檔案路徑、檔案下載下傳chunk大小

（4）mod_fastdfs.conf  ~擴充配置檔案的demo，一般會将該檔案拷貝到config指定的目錄下 例如：/etc/fdfs
           

• 初始化： nginx啟動時，  fastdfs-nginx-module 要完成初始化如下圖 ，我們一般在mod_fastdfs.conf配置參數，如下圖

（重要）： fastdfs-nginx-module 初始化的過程要加載mod_fastdfs.conf參數，如果本機器下存在多個storage,且有多個group（group1、group2）,則 mod_fastdfs.conf 配置需做如下變動

（1）組名：group_name=group1/group2     多個用/區分開

（2）設定組個數：group_count = 4

（4）設定各group資訊：

[group1]

group_name=group1

storage_server_port=23000

store_path_count=1

store_path0=/usr/local/fastdfs/storage

[group2]

group_name=group2

storage_server_port=23010

store_path_count=1

store_path0=/usr/local/fastdfs/storage

（重要）通過nginx 從fastdfs下載下傳檔案，詳細說明可參考https://www.cnblogs.com/littleatp/p/4361318.html     

6、各元件運作機制總結(重要)

•         一個group 對應多個 storage (1:N)
•        一個storage對應一個group   (1:1)
•        一個tracker對應多個storage(1:N)
•        一個storage對應多個tracker(1:N)  , tracker 和storage的關系是多對多（N:M）
•        一個storage下有多個存儲路徑 store_path(1:N)

7、部署架構彙總

1）單體部署: 單group\單storage\單tracker

2）單伺服器多storage部署（在實際生産環境中沒有意義）

 多group\多storage\單tracker

3）多伺服器多group且group不互備，單tracker（我們項目本次硬碟擴充部署架構）

     由于目前伺服器資源緊缺暫不做group互備，後面需要做group互備

•   部署步驟及參數配置

  （1）兩台伺服器分别為192.168.0.171、192.168.0.164， 171伺服器擔任的功能角色更多一些: 檔案下載下傳請求 nginx同一入口（分發到storage1、stroage2）、tracker server  、storage1 - group0（fastdfs-nginx-module）。

              164伺服器主要負責storage1 -group2 的存儲、下載下傳功能，沒有tacker server，直接連接配接171伺服器的tracker,需要安裝nginx 、 fastdfs-nginx-module

         （2) 171、 164 都需要安裝 fastdfs 、fastdfs-nginx-module、 nginx  安裝步驟 與  知識庫文檔  Centos7 上安裝 FastDFS 一緻 ，但注意一點164伺服器不用啟動及配置tracker

        （ 3） 171 tracker.conf 配置

      171 tracker.conf 核心參數配置說明，其他參數見附件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

# is this config file disabled # false for enabled # true for disabled disabled= false # the tracker server port port= 22122 # the base path to store data and log files base_path=/data/fastdfs/tracker # the method of selecting group to upload files # : round robin # 1 : specify group # 2 : load balance, select the max free space group to upload file store_lookup= 1 # which group to upload file # when store_lookup set to 1 , must set store_group to the group name store_group=group2 # which storage server to upload file # : round robin ( default ) # 1 : the first server order by ip address # 2 : the first server order by priority (the minimal) store_server= # which path(means disk or mount point) of the storage server to upload file # : round robin # 2 : load balance, select the max free space path to upload file store_path= # which storage server to download file # : round robin ( default ) # 1 : the source storage server which the current file uploaded to download_server= # reserved storage space for system or other applications. # if the free(available) space of any stoarge server in # a group <= reserved_storage_space, # no file can be uploaded to this group. # bytes unit can be one of follows: ### G or g for gigabyte(GB) ### M or m for megabyte(MB) ### K or k for kilobyte(KB) ### no unit for byte(B) ### XX.XX% as ratio such as reserved_storage_space = 10 % reserved_storage_space = 10 %

（4）171 storage.conf 配置

171 storage 核心參數配置，其他參數見附件

# the name of the group this storage server belongs to # # comment or remove this item for fetching from tracker server, # in this case , use_storage_id must set to true in tracker.conf, # and storage_ids.conf must be configed correctly. group_name=group0 # the storage server port port= 23000 # the base path to store data and log files base_path=/data/fastdfs/storage # path(disk or mount point) count, default value is 1 store_path_count= 1 # store_path#, based , if store_path0 not exists, it's value is base_path # the paths must be exist store_path0=/data/fastdfs/storage #store_path1=/home/yuqing/fastdfs2 # tracker_server can ocur more than once, and tracker_server format is #  "host:port" , host can be hostname or ip address tracker_server= 192.168 . 0.171 : 22122 # the priority as a source server for uploading file. # the lower this value, the higher its uploading priority. # default value is 10 upload_priority= 10

（5）171 fastdfs_nginx_module 配置參數 （mod_fastdfs.conf）

171 mod_fastdfs.conf 核心參數配置，其他參數見附件

# the base path to store log files base_path=/data/fastdfs/storage # if load FastDFS parameters from tracker server # since V1. 12 # default value is false load_fdfs_parameters_from_tracker= true # FastDFS tracker_server can ocur more than once, and tracker_server format is #  "host:port" , host can be hostname or ip address # valid only when load_fdfs_parameters_from_tracker is true tracker_server= 192.168 . 0.171 : 22122 # the port of the local storage server # the default value is 23000 storage_server_port= 23000 # the group name of the local storage server group_name=group0 # if the url / uri including the group name # set to false when uri like /M00/ 00 / 00 /xxx # set to true when uri like ${group_name}/M00/ 00 / 00 /xxx, such as group1/M00/xxx # default value is false url_have_group_name = true # path(disk or mount point) count, default value is 1 # must same as storage.conf store_path_count= 1 # store_path#, based , if store_path0 not exists, it's value is base_path # the paths must be exist # must same as storage.conf store_path0=/data/fastdfs/storage #store_path1=/home/yuqing/fastdfs1 # set the group count # set to none zero to support multi-group # set to for single group only # groups settings section as [group1], [group2], ..., [groupN] # default value is # since v1. 14 group_count = # group settings for group # 1 # since v1. 14 # when support multi-group, uncomment following section #[group1] #group_name=group1 #storage_server_port= 23000 #store_path_count= 2 #store_path0=/home/yuqing/fastdfs #store_path1=/home/yuqing/fastdfs1 # group settings for group # 2 # since v1. 14 # when support multi-group, uncomment following section as neccessary #[group2] #group_name=group2 #storage_server_port= 23000 #store_path_count= 1 #store_path0=/home/yuqing/fastdfs

（6）171 nginx 參數配置 nginx.conf 

171 nginx.conf 核心參數配置，詳見附件

events { worker_connections  1024 ; } http {     include mime.types;     default_type  application/octet-stream; sendfile        on;     #tcp_nopush     on; #keepalive_timeout  ;     keepalive_timeout  65 ; #gzip  on; # 192.168 . 0.164 storage group2  upstream fdfs_group2_164 {  server 192.168 . 0.164 : 8288 weight= 1 max_fails= 2 fail_timeout=30s;  } server {         listen       8070 ;         server_name  localhost, 192.168 . 0.171 ; #charset koi8-r; #access_log  logs/host.access.log  main; location / {             root   html;             max_ranges 1 ;             index  index.html index.htm;         }  location /group0/M00{             root /data/fastdfs/storage/data;             ngx_fastdfs_module;             if ($request_method = 'OPTIONS' ) {                 add_header 'Access-Control-Allow-Origin' '*' ;                 add_header 'Access-Control-Allow-Headers' 'Range' ;                add_header 'Content-Type' 'text/plain charset=UTF-8' ;                 add_header 'Content-Length' ;                 return 204 ;              }             if ($request_method = 'POST' ) {                 add_header 'Access-Control-Allow-Origin' '*' ;                 add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' ;                 add_header 'Access-Control-Allow-Headers' 'Range' ;              }             if ($request_method = 'GET' ) {                 add_header 'Access-Control-Allow-Origin' '*' ;                 add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' ;                 add_header 'Access-Control-Allow-Headers' 'Range' ;                 add_header 'Access-Control-Expose-Headers' 'Accept-Ranges, Content-Encoding, Content-Length, Content-Range' ;              }       if ($arg_attname ~* \.(doc|docx|txt|pdf|zip|rar|txt|jpg|png|gif|bmp)$) { add_header "Content-Disposition" "attachment;filename=$arg_attname" ; }         }  location ~* /group2/(M00|M01) {   proxy_next_upstream http_502 http_504 error timeout invalid_header;  proxy_pass http: //fdfs_group2_164; expires 30d;  } error_page   500 502 503 504 /50x.html;         location = /50x.html {             root   html;         } }

（7）164 storage參數配置

164 storage.con 核心參數配置，其他參數見附件

# the name of the group this storage server belongs to # # comment or remove this item for fetching from tracker server, # in this case , use_storage_id must set to true in tracker.conf, # and storage_ids.conf must be configed correctly. group_name=group2 # the storage server port port= 23000 # the base path to store data and log files base_path=/usr/local/fastdfs/fdfs_storage # path(disk or mount point) count, default value is 1 store_path_count= 1 # store_path#, based , if store_path0 not exists, it's value is base_path # the paths must be exist store_path0=/usr/local/fastdfs/fdfs_storage #store_path1=/home/yuqing/fastdfs2 # tracker_server can ocur more than once, and tracker_server format is #  "host:port" , host can be hostname or ip address tracker_server= 192.168 . 0.171 : 22122 # the priority as a source server for uploading file. # the lower this value, the higher its uploading priority. # default value is 10 upload_priority= 10

（8） 164 fastdfs_nginx_module 參數配置 （mod_fastdfs.conf）

164 mod_fastdfs.conf 核心參數配置，其他參數見附件

# the base path to store log files base_path=/usr/local/fastdfs/ # if load FastDFS parameters from tracker server # since V1. 12 # default value is false load_fdfs_parameters_from_tracker= true # FastDFS tracker_server can ocur more than once, and tracker_server format is #  "host:port" , host can be hostname or ip address # valid only when load_fdfs_parameters_from_tracker is true tracker_server= 192.168 . 0.171 : 22122 # the port of the local storage server # the default value is 23000 storage_server_port= 23000 # the group name of the local storage server group_name=group2 # if the url / uri including the group name # set to false when uri like /M00/ 00 / 00 /xxx # set to true when uri like ${group_name}/M00/ 00 / 00 /xxx, such as group1/M00/xxx # default value is false url_have_group_name = true # path(disk or mount point) count, default value is 1 # must same as storage.conf store_path_count= 1 # store_path#, based , if store_path0 not exists, it's value is base_path # the paths must be exist # must same as storage.conf store_path0=/usr/local/fastdfs/fdfs_storage #store_path1=/home/yuqing/fastdfs1 # set the group count # set to none zero to support multi-group # set to for single group only # groups settings section as [group1], [group2], ..., [groupN] # default value is # since v1. 14 group_count =

（9）164 nginx參數配置，nginx.conf

164 nginx.conf 參數配置，詳見附件

user  www  www; worker_processes  12 ; error_log  / var /log/nginx/error.log; events { worker_connections  1024 ; } http { include mime.types; default_type  application/octet-stream; client_max_body_size 10m; sendfile        on; server { listen       8288 ; server_name  192.168 . 0.164 ,localhost; location /group2/M00/{ root /usr/local/fastdfs/fdfs_storage/data; ngx_fastdfs_module; if ($request_method = 'OPTIONS' ) { add_header 'Access-Control-Allow-Origin' '*' ; add_header 'Access-Control-Allow-Headers' 'Range' ; add_header 'Content-Type' 'text/plain charset=UTF-8' ; add_header 'Content-Length' ; return 204 ; } if ($request_method = 'POST' ) { add_header 'Access-Control-Allow-Origin' '*' ; add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' ; add_header 'Access-Control-Allow-Headers' 'Range' ; } if ($request_method = 'GET' ) { add_header 'Access-Control-Allow-Origin' '*' ; add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS' ; add_header 'Access-Control-Allow-Headers' 'Range' ; add_header 'Access-Control-Expose-Headers' 'Accept-Ranges, Content-Encoding, Content-Length, Content-Range' ; } if ($arg_attname ~* \.(doc|docx|txt|pdf|zip|rar|txt|jpg|png|gif|bmp)$) { add_header "Content-Disposition" "attachment;filename=$arg_attname" ; } }   } }

4）真正分布式叢集部署：多伺服器多group且group間互備，多tracker