大資料項目實施筆記

使用者和綜合分析系統

項目背景

近年來，伴随着網際網路金融的風生水起；國家出台相關檔案，要求加大網際網路交易風險防控力度；鼓勵通過大資料分析、使用者行為模組化等手段建立和完善交易風險檢測模型。但是目前大資料風控還存在時效性差，準确性不高等問題。綜合使用者分析平台包含 綜合資料分析|登陸風險|注冊風險|交易風險|活動風險分析等子產品。以下是個各個子系統之間的關系。

• 業務系統：通常指的是APP+背景或Web端（服務目标使用者），是業務子產品的載體，大資料計算的資料來源。
• 風控系統（流計算）：為業務系統提供支援（服務目标是背景服務），根據業務系統搜集來得資料資料或者在頁面埋藏的一些

  埋點

  去擷取使用者使用環境或者使用者操作行為。
• 懲罰系統：根據風險評估系統計算出的

  報告

  、根據報告配合具體的懲罰業務對事件産生者進行控制或者懲罰。例如：增加驗證碼、禁止使用者登入、注冊或者下單。
• 綜合分析: 輔助風控系統，根據分控系統的表現，判斷的風控的政策或者參數設定是否存在問題。比如：評估因子觸發率極低或者極高等，該系統的可以有效的幫助的營運和分析人員完成新政策的發現以及營銷手段改良。

平台架構

系統模型

實時業務

注冊風險

檢測出一些不正常的注冊行為，并産生注冊評估報告。

• 短時間内同一個裝置在同一個網絡下注冊資料超過N個
• 如果注冊時間少于m分鐘,且産生多次注冊行為，認定注冊有異常.
• 同一個裝置或者IP，規定時間内多次調用

  短信

  |

  郵件服務

  超過N次 ，注冊風險

√登入風險

檢查出非使用者登入，或者賬号被盜竊等潛在風險挖掘。

• 異地登入認定有風險（不在使用者經常登陸地）
• 登入的位移速度超過正常值，認定有風險
• 更換裝置，認定有風險
• 登入時段-習慣，出現了異常，存在風險
• 每天的累計登入次數超過上限，認定有風險
• 密碼輸入錯誤或者輸入差異性較大，認定風險
• 如果使用者的輸入特征（輸入每個控件所需時長 ms）發生變化

交易風險

輔助系統檢測使用者支付環境，地區、網絡、次數

• 使用者交易習慣，出現在異常時段認定有異常
• 檢測使用者的網絡環境，如果不是經常使用WI-FI
• 短時間内産生多次交易，存在風險
• 支付金額過大，認定有風險
• 陌生地區支付，認定有風險
• 交易裝置發生變化

活動風險

活動風險涉及活動時間、獎勵兌換/兌換率

• 活動時間提前結束，認定有風險。
• 活動短時間内兌換率過高。
• 一個賬号多次兌換認定有風險。

離線業務

輔助流計算或者企業營運部更好經營業務。

分析平台使用者成分（性别、裝置、年齡段、地域）

應用使用習慣（時間、類别|頻道）

頻道或者類别點選排行榜，頁面停留時長、跳出率

風控系統中各類評估因子在每個應用的應用表現

Java業務闆塊 (階段1）

工作計劃表

任務計劃	工時安排	備注
SpringBoot快速建構Web-Model後端 Restful 接口釋出	1天-編碼
RestTemplate、WebClient/Postman 單元測試	1天-講解
基于EasyUI對接Restful接口完成前後端對接	3天-70%講解、30%實戰
jQuery插件定制和和埋點設計	0.5天-100%講解
jQuery自定義插件和和埋點設計	1.5天-70%講解、30%實戰
Echars/HighCharts基礎報表可視化展示（靜态）	1.5天-30%講解、70%實戰
對接SpringCloud完成微服務的開發與部署	3天- 70%講解 30%實戰

Notes:輔線，不會涉及Java中過多複雜業務場景，僅僅作為JavaWeb業務基礎，用于展示大資料可視化平台，以及了解什麼是微服務開發。

SpringBoot快速建構Web-User-Model後端 Restful 接口釋出

<groupId>com.baizhi</groupId>
<artifactId>UserModel</artifactId>
<version>1.0-SNAPSHOT</version>
           

1、要求SpringBoot版本

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.1.5.RELEASE</version>
</parent>
           

2、給出以下腳本

DROP TABLE IF EXISTS t_user;
set character_set_results=utf8;
set character_set_client=utf8;
CREATE TABLE t_user  (
  id int primary key AUTO_INCREMENT,
  name varchar(32) unique ,
  password varchar(128) ,
  sex tinyint(1) ,
  photo varchar(255) ,
  birthDay date,
  email varchar(128)
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8;
           

3、實體類

public class User implements Serializable {
    private Integer id;
    private String name;
    private boolean sex;
    private String password;
    private Date birthDay;
    private String photo;
    private String email;
  ...
}
           

4、DAO接口

public interface IUserDAO {
     void saveUser(User user);
     User queryUserByNameAndPassword(User user);
     User queryUserById(Integer id);
     void deleteByUserId(Integer id);
     List<User> queryUserByPage(
             @Param(value = "pageNow") Integer pageNow,
             @Param(value = "pageSize") Integer pageSize,
             @Param(value = "column") String column,
             @Param(value = "value") Object value);

     int queryCount(
             @Param(value = "column") String column,
             @Param(value = "value") Object value);

    void updateUser(User user);
}
           

5、Service接口

public interface IUserService {
    /**
     * 儲存使用者
     * @param user
     */
    void saveUser(User user);

    /**
     * 根據密碼和使用者名查詢使用者
     * @param user
     * @return
     */
    User queryUserByNameAndPassword(User user);
    /***
     *
     * @param pageNow
     * @param pageSize
     * @param column 模糊查詢列
     * @param value  模糊值
     * @return
     */
    List<User> queryUserByPage(Integer pageNow, Integer pageSize,
                               String column, Object value);

    /**
     * 查詢使用者總記錄
     * @param column
     * @param value
     * @return
     */
    int queryUserCount(String column, Object value);

    /**
     * 根據ID查詢使用者資訊
     * @param id
     * @return
     */
    User queryUserById(Integer id);

    /**
     * 根據IDS删除使用者
     * @param ids
     */
    void deleteByUserIds(Integer[] ids);

    /**
     * 更新使用者資訊
     * @param user
     */
    void updateUser(User user);
}
           

6、控制層通路接口

@RestController
@RequestMapping(value = "/formUserManager")
public class FormUserController {
  
    @PostMapping(value = "/registerUser")
    public User registerUser(User user,
                             @RequestParam(value = "multipartFile",required = false) MultipartFile multipartFile) throws IOException 

    @PostMapping(value = "/userLogin")
    public User userLogin(User user)


    @PostMapping(value = "/addUser")
    public User addUser(User user,
                        @RequestParam(value = "multipartFile",required = false) MultipartFile multipartFile) throws IOException 

    @PutMapping(value = "/updateUser")
    public void updateUser(User user,
                           @RequestParam(value = "multipartFile",required = false) MultipartFile multipartFile) throws IOException 

    @DeleteMapping(value = "/deleteUserByIds")
    public void delteUserByIds(@RequestParam(value = "ids") Integer[] ids)

    @GetMapping(value = "/queryUserByPage")
    public List<User> queryUserByPage(@RequestParam(value = "page",defaultValue = "1") Integer pageNow,
                                              @RequestParam(value = "rows",defaultValue = "10") Integer pageSize,
                                              @RequestParam(value = "column",required = false) String column,
                                              @RequestParam(value = "value",required = false) String value)

    @GetMapping(value = "/queryUserCount")
    public Integer queryUserCount(    @RequestParam(value = "column",required = false) String column,
                                      @RequestParam(value = "value",required = false) String value)
    
      @GetMapping(value = "/queryUserById")
    public User queryUserById(@RequestParam(value = "id") Integer id)

}
           

建構項目

項目基本結構

配置⽂件清單

• maven pom依賴

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.baizhi</groupId>
    <artifactId>UserModel</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>war</packaging>

    <name>UserModel Maven Webapp</name>
    <!-- FIXME change it to the project's website -->
    <url>http://www.example.com</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>

    <!-- 仲裁中心 -->
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.5.RELEASE</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <!-- web啟動器 -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <!-- msq -->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.12</version>
        </dependency>
        <!-- 阿裡連接配接池 -->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid</artifactId>
            <version>1.0.31</version>
        </dependency>
        <!-- mybatis -->
        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>1.3.3</version>
        </dependency>
        <!-- 使内嵌的tomcat支援解析jsp -->
        <dependency>
            <groupId>org.apache.tomcat.embed</groupId>
            <artifactId>tomcat-embed-jasper</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-tomcat</artifactId>
            <scope>provided</scope>
        </dependency>
        <!-- jstl -->
        <dependency>
            <groupId>jstl</groupId>
            <artifactId>jstl</artifactId>
            <version>1.2</version>
        </dependency>
        <!-- 測試 -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <!-- aop -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-tx</artifactId>
            <version>5.1.5.RELEASE</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.mybatis</groupId>
            <artifactId>mybatis-spring</artifactId>
            <version>1.3.2</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.mybatis</groupId>
            <artifactId>mybatis</artifactId>
            <version>3.4.6</version>
            <scope>compile</scope>
        </dependency>
        <!-- 檔案支援 -->
        <dependency>
            <groupId>commons-fileupload</groupId>
            <artifactId>commons-fileupload</artifactId>
            <version>1.3</version>
        </dependency>
        <!--     lombok相關   -->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.4</version>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.47</version>
        </dependency>
        <!--Spring Redis RedisAutoConfiguration-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>3.2.0</version>
        </dependency>
    </dependencies>

    <build>
        <finalName>UserModel</finalName>
        <plugins>
            <!--    springbootjar包支援    -->
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>1.4.2.RELEASE</version>
            </plugin>
        </plugins>
    </build>
</project>
           

• application.yml

# 通路接口配置
server:
  port: 8989
  servlet:
    context-path: /UserModel
spring:
  # 資料源配置
  datasource:
    type: com.alibaba.druid.pool.DruidDataSource
    username: root
    password: root
    driver-class-name: com.mysql.jdbc.Driver
    url: jdbc:mysql://CentOS:3306/usermodel?useSSL=false&characterEncoding=UTF8&serverTimezone=GMT
  # 上傳檔案控制
  servlet:
    multipart:
      max-request-size: 50MB
      max-file-size: 50MB
      enabled: true
  # 亂碼解決
  http:
    encoding:
      charset: utf-8
  # 連接配接redis
  redis:
    host: CentOS
    port: 6379

# Mybatis配置
mybatis:
  mapper-locations: classpath:mapper/*.xml
  type-aliases-package: com.baizhi.entity
  # //在resource目錄下建立config檔案夾
  config-location: classpath:config/mybatis.xml
  # 開啟Mybatis批處理模式
logging:
  level:
    root: info
           

• UserDAO.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.baizhi.dao.IUserDAO">
    <cache type="com.baizhi.cache.UserDefineRedisCache"></cache>
    <sql id="all">
        id, name, password, sex, photo, birthDay, email
    </sql>
    <resultMap type="com.baizhi.entity.User" id="TUserMap">
        <result property="id" column="id" jdbcType="INTEGER"/>
        <result property="name" column="name" jdbcType="VARCHAR"/>
        <result property="password" column="password" jdbcType="VARCHAR"/>
        <result property="sex" column="sex" jdbcType="OTHER"/>
        <result property="photo" column="photo" jdbcType="VARCHAR"/>
        <result property="birthDay" column="birthDay" jdbcType="OTHER"/>
        <result property="email" column="email" jdbcType="VARCHAR"/>
    </resultMap>

    <!--查詢單個-->
    <select id="queryUserById" resultMap="TUserMap">
        select
        <include refid="all"></include>
        from usermodel.t_user
        where id = #{id}
    </select>

    <!--查詢指定行資料-->
    <select id="queryUserByPage" resultMap="TUserMap">
        select <include refid="all"></include> from usermodel.t_user
        <where>
            <if test="column != null and column != '' and value != ''">
                ${column} like "%"#{value}"%"
            </if>
        </where>
        limit #{pageNow},#{pageSize}
    </select>

    <!--通過姓名密碼篩選條件查詢-->
    <select id="queryUserByNameAndPassword" resultMap="TUserMap">
        select
        <include refid="all"></include>
        from usermodel.t_user
        <where>
            <if test="name != null and name != ''">
                and name = #{name}
            </if>
            <if test="password != null and password != ''">
                and password = #{password}
            </if>
        </where>
    </select>

    <!--添加使用者-->
    <insert id="saveUser" keyProperty="id" useGeneratedKeys="true">
        insert into usermodel.t_user(name, password, sex, photo, birthDay, email)
        values (#{name}, #{password}, #{sex}, #{photo}, #{birthDay}, #{email})
    </insert>

    <!--修改資料-->
    <update id="updateUser">
        update usermodel.t_user
        <set>
            <if test="name != null and name != ''">
                name = #{name},
            </if>
            <if test="password != null and password != ''">
                password = #{password},
            </if>
            <if test="sex != null">
                sex = #{sex},
            </if>
            <if test="photo != null and photo != ''">
                photo = #{photo},
            </if>
            <if test="birthDay != null">
                birthDay = #{birthDay},
            </if>
            <if test="email != null and email != ''">
                email = #{email},
            </if>
        </set>
        where id = #{id}
    </update>

    <!--通過主鍵删除-->
    <delete id="deleteByUserId">
        delete from usermodel.t_user where id = #{id}
    </delete>
    <!--  查詢總行數  -->
    <select id="queryCount" resultType="Integer">
        select count(*) from t_user
        <where>
            <if test="column != null and column != '' and value != ''">
                ${column} like "%"#{value}"%"
            </if>
        </where>
    </select>

</mapper>
           

• logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds" debug="false">
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%p %c#%M %d{yyyy-MM-dd HH:mm:ss} %m%n</pattern>
            <charset>UTF-8</charset>
        </encoder>
    </appender>
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        　
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            　　　　　　　　　
            <fileNamePattern>logs/userLoginFile-%d{yyyyMMdd}.log</fileNamePattern>
            　　　　　　　　　
            <maxHistory>30</maxHistory>
            　　　　　　
        </rollingPolicy>
        　　　　　　
        <encoder>
            　　　　　　　　　
            <pattern>%p %c#%M %d{yyyy-MM-dd HH:mm:ss} %m%n</pattern>
            <charset>UTF-8</charset>
            　　　　　　
        </encoder>
    </appender>
    <!-- 控制台輸出⽇志級别 -->
    <root level="ERROR">
        <appender-ref ref="STDOUT"/>
    </root>
    <logger name="org.springframework.jdbc" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger name="com.baizhi.dao" level="TRACE" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>
    <logger name="com.baizhi.cache" level="DEBUG" additivity="false">
        <appender-ref ref="STDOUT"/>
    </logger>
</configuration>
           

• t_user.sql

DROP TABLE IF EXISTS t_user;
set character_set_results=utf8;
set character_set_client=utf8;
CREATE TABLE t_user (
 id int primary key AUTO_INCREMENT,
 name varchar(32) unique ,
 password varchar(128) ,
 sex tinyint(1) ,
 photo varchar(255) ,
 birthDay date,
 email varchar(128)
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8;
           

項⽬代碼清單

• User實體類

/**
 * (TUser)實體類
 *
 * @author makejava
 * @since 2020-03-16 16:54:52
 */
@Data
@AllArgsConstructor
@NoArgsConstructor
public class User implements Serializable {
    private static final long serialVersionUID = 250284593728595695L;
    private Integer id;
    private String name;
    private boolean sex;
    private String password;
    @DateTimeFormat(pattern = "yyyy-MM-dd")
    @JsonFormat(pattern = "yyyy-MM-dd", timezone = "GMT+8")
    private Date birthDay;
    private String photo;
    private String email;

}
           

• Dao接口

/**
 * (TUser)DAO接口
 *
 * @author makejava
 * @since 2020-03-16 16:54:52
 */
public interface IUserDAO {
    /**
     * 添加使用者資訊
     *
     * @param user
     */
    void saveUser(User user);

    /**
     * 根據使用者名稱及密碼查詢使用者
     *
     * @param user
     * @return
     */
    User queryUserByNameAndPassword(User user);

    /**
     * 根基使用者ID查詢
     *
     * @param id
     * @return
     */
    User queryUserById(Integer id);

    /**
     * 删除使用者
     *
     * @param id
     */
    void deleteByUserId(Integer id);

    /**
     * 分頁查詢使用者
     *
     * @param pageNow
     * @param pageSize
     * @param column
     * @param value
     * @return
     */
    List<User> queryUserByPage(
            @Param(value = "pageNow") Integer pageNow,
            @Param(value = "pageSize") Integer pageSize,
            @Param(value = "column") String column,
            @Param(value = "value") Object value);

    /**
     * 查詢總行數
     *
     * @param column
     * @param value
     * @return
     */
    int queryCount(
            @Param(value = "column") String column,
            @Param(value = "value") Object value);

    /**
     * 修改使用者資訊
     *
     * @param user
     */
    void updateUser(User user);
}
           

• Service接口

/**
 * (TUser)表服務接口
 *
 * @author makejava
 * @since 2020-03-16 16:54:52
 */
public interface IUserService {
    /**
     * 添加使用者
     *
     * @param user
     */
    void saveUser(User user);

    /**
     * 根據密碼和使用者名查詢使用者
     *
     * @param user
     * @return
     */
    User queryUserByNameAndPassword(User user);

    /***
     *
     * @param pageNow
     * @param pageSize
     * @param column 模糊查詢列
     * @param value  模糊值
     * @return
     */
    List<User> queryUserByPage(Integer pageNow, Integer pageSize,
                               String column, Object value);

    /**
     * 查詢使用者總記錄
     *
     * @param column
     * @param value
     * @return
     */
    int queryUserCount(String column, Object value);

    /**
     * 根據ID查詢使用者資訊
     *
     * @param id
     * @return
     */
    User queryUserById(Integer id);

    /**
     * 根據IDS删除使用者
     *
     * @param ids
     */
    void deleteByUserIds(Integer[] ids);

    /**
     * 更新使用者資訊
     *
     * @param user
     */
    void updateUser(User user);
}
           

• Service實作類

/**
 * (TUser)表服務實作類
 *
 * @author makejava
 * @since 2020-03-16 16:54:52
 */
@Service("tUserService")
@Transactional
public class IUserServiceImpl implements IUserService {

    @Resource
    private IUserDAO iUserDAO;

    /**
     * 添加使用者
     *
     * @param user
     */
    @Override
    public void saveUser(User user) {
        iUserDAO.saveUser(user);
    }

    /**
     * 根據條件查詢
     *
     * @param user
     * @return
     */
    @Override
    @Transactional(propagation = Propagation.SUPPORTS, readOnly = true)
    public User queryUserByNameAndPassword(User user) {
        User user1 = iUserDAO.queryUserByNameAndPassword(user);
        return user1;
    }

    /**
     * 分頁查詢
     *
     * @param pageNow
     * @param pageSize
     * @param column   模糊查詢列
     * @param value    模糊值
     * @return
     */
    @Override
    @Transactional(propagation = Propagation.SUPPORTS, readOnly = true)
    public List<User> queryUserByPage(Integer pageNow, Integer pageSize, String column, Object value) {
        int i = pageNow - 1;
        int i1 = i * pageSize;
        List<User> users = iUserDAO.queryUserByPage(i1, pageSize, column, value);
        return users;
    }

    /**
     * 統計總行數
     *
     * @param column
     * @param value
     * @return
     */
    @Override
    @Transactional(propagation = Propagation.SUPPORTS, readOnly = true)
    public int queryUserCount(String column, Object value) {
        int count = iUserDAO.queryCount(column, value);
        return count;
    }

    /**
     * 根據ID查詢使用者
     *
     * @param id
     * @return
     */
    @Override
    @Transactional(propagation = Propagation.SUPPORTS, readOnly = true)
    public User queryUserById(Integer id) {
        User user = iUserDAO.queryUserById(id);
        return user;
    }

    /**
     * 删除使用者
     *
     * @param ids
     */
    @Override
    public void deleteByUserIds(Integer[] ids) {
        for (Integer id : ids) {
            iUserDAO.deleteByUserId(id);
        }

    }

    /**
     * 修改使用者資訊
     *
     * @param user
     */
    @Override
    public void updateUser(User user) {
        iUserDAO.updateUser(user);
    }
}
           

• Application-springboot入口類

package com.baizhi;


import org.mybatis.spring.annotation.MapperScan;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
import org.springframework.context.annotation.Bean;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;

@SpringBootApplication
@MapperScan(basePackages = "com.baizhi.dao")
public class Application extends SpringBootServletInitializer {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    public SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
        return builder.sources(Application.class);
    }

    @Bean
    public RedisTemplate redisTemplate() {
        return new RedisTemplate();
    }

    @Bean
    public RedisTemplate redisTemplate(RedisConnectionFactory redisConnectionFactory) {
        RedisTemplate redisTemplate = new RedisTemplate();
        redisTemplate.setConnectionFactory(redisConnectionFactory);
        redisTemplate.setValueSerializer(new GenericJackson2JsonRedisSerializer());
        return redisTemplate;
    }
}
           

內建Junit測試

• 建立測試配置類
• Dao測試

/**
 * UserDao測試類
 */
@RunWith(value = SpringRunner.class)
@SpringBootTest
@Slf4j
public class UserDaoTest {
    @Resource
    private IUserDAO iUserDAO;


    @Test
    public void queryById() {
        System.out.println(iUserDAO.queryUserById(3));
    }

    @Test
    public void queryAll() {
        List<User> users = iUserDAO.queryUserByPage(0, 3, "name", "z");
        for (User user : users) {
            System.out.println(user);
        }
    }

    @Test
    public void addUser() {
        User user = new User();
        user.setName("張三");
        user.setPassword("123123");
        Date date = new Date();
        user.setBirthDay(date);
        iUserDAO.saveUser(user);

    }

    @Test
    public void twoCacheTest() {
        User user1 = iUserDAO.queryUserById(1);
        log.info("user1:{}", JSON.toJSONString(user1));
        log.info("第一次查詢");
        User user2 = iUserDAO.queryUserById(1);
        log.info("user2:{}", JSON.toJSONString(user2));
        log.info("第二次查詢");
        User user3 = iUserDAO.queryUserById(1);
        log.info("user3:{}", JSON.toJSONString(user3));
        log.info("第三次查詢");
        user1.setName("test1");
        iUserDAO.updateUser(user1);
        User user4 = iUserDAO.queryUserById(1);
        log.info("user4:{}", JSON.toJSONString(user4));
        log.info("第四次查詢");
    }
}
           

• Service測試

package com.baizhi.service;

import com.baizhi.entity.User;
import com.baizhi.service.impl.IUserServiceImpl;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import java.util.Date;
import java.util.List;

import static org.junit.Assert.*;

/**
 * UserService測試類
 */
@RunWith(value = SpringRunner.class)
@SpringBootTest
@Slf4j
public class UserServiceTest {
    @Autowired
    private IUserService userService;

    @Test
    public void saveUserTest() {
        User user = new User(null, "趙⼩六", true, "123456", new Date(), "aa.png", "qq.com");
        userService.saveUser(user);
        assertNotNull("⽤戶ID不為空", user.getId());
    }

    @Test
    public void queryUserByNameAndPasswordTests() {
        User loginUser = new User();
        loginUser.setName("趙⼩六");
        loginUser.setPassword("123456");
        User queryUser = userService.queryUserByNameAndPassword(loginUser);
        assertNotNull("⽤戶ID不為空", queryUser.getId());
    }

    @Test
    public void queryUserByPageTests() {
        Integer pageNow = 1;
        Integer pageSize = 10;
        String column = "name";
        String value = "⼩";
        List<User> userList = userService.queryUserByPage(pageNow, pageSize, column,
                value);
        assertFalse(userList.isEmpty());
    }

    @Test
    public void queryUserCountTest() {
        String column = "name";
        String value = "⼩";
        Integer count = userService.queryUserCount(column, value);
        assertTrue(count != 0);
    }

    @Test
    public void queryUserById() {
        Integer id = 2;
        User u = userService.queryUserById(id);
        assertNotNull(u.getName());
    }

    @Test
    public void deleteByUserIdsTests() {
        userService.deleteByUserIds(new Integer[]{1, 4, 3});
    }

    @Test
    public void updateUserTests() {
        Integer id = 2;
        User u = userService.queryUserById(id);
        assertNotNull(u.getName());
        u.setSex(false);
        userService.updateUser(u);
        User newUser = userService.queryUserById(2);
        assertEquals("⽤戶sex", false, newUser.getSex());
    }
}
           

Notes:這⾥使⽤Junit測試的Assert(斷⾔測試)，需要額外的導⼊ import static org.junit.Assert.*;

RestControler釋出與測試

RestControler

• form表單參數接收

/**
 * (TUser)表控制層
 *
 * @author makejava
 * @since 2020-03-16 16:54:52
 */
@RestController
@RequestMapping(value = "/formUserManager")
public class FormUserController {
    private static final Logger LOGGER = LoggerFactory.getLogger(FormUserController.class);
    @Autowired
    private IUserServiceImpl tUserService;

    /**
     * 使用者登陸
     *
     * @param user
     * @return
     */
    @PostMapping(value = "/userLogin")
    public User userLogin(User user) {
        return tUserService.queryUserByNameAndPassword(user);
    }

    /**
     * 添加使用者
     *
     * @param user
     * @param multipartFile
     * @return
     * @throws IOException
     */
    @PostMapping(value = "/addUser")
    public User addUser(User user, @RequestParam(value = "multipartFile", required = false) MultipartFile multipartFile) throws IOException {
        if (multipartFile != null) {
            String filename = multipartFile.getOriginalFilename();
            String suffix = filename.substring(filename.lastIndexOf("."));
            File tempFile = File.createTempFile(filename.substring(0, filename.lastIndexOf(".")), suffix);
            System.out.println(tempFile.getName());
            tempFile.delete();
        }
        tUserService.saveUser(user);
        return user;
    }

    /**
     * 修改使用者資訊
     *
     * @param user
     * @param multipartFile
     * @throws IOException
     */
    @PutMapping(value = "/updateUser")
    public void updateUser(User user, @RequestParam(value = "multipartFile", required = false) MultipartFile multipartFile) throws IOException {
        if (multipartFile != null) {
            String filename = multipartFile.getOriginalFilename();
            String suffix = filename.substring(filename.lastIndexOf("."));
            File tempFile = File.createTempFile(filename.substring(0, filename.lastIndexOf(".")), suffix);
            System.out.println(tempFile.getName());
            tempFile.delete();
        }
        //更新使用者資訊
        tUserService.updateUser(user);
    }

    /**
     * 删除使用者
     *
     * @param ids
     */
    @DeleteMapping(value = "/deleteUserByIds")
    public void delteUserByIds(@RequestParam(value = "ids") Integer[] ids) {
        tUserService.deleteByUserIds(ids);
    }

    /**
     * 模糊查詢
     *
     * @param pageNow
     * @param pageSize
     * @param column
     * @param value
     * @return
     */
    @GetMapping(value = "/queryUserByPage")
    public List<User> queryUserByPage(@RequestParam(value = "page", defaultValue = "1") Integer pageNow,
                                      @RequestParam(value = " ", defaultValue = "10") Integer pageSize,
                                      @RequestParam(value = "column", required = false) String column,
                                      @RequestParam(value = "value", required = false) String value) {
        HashMap<String, Object> map = new HashMap<>();
        map.put("total", tUserService.queryUserCount(column, value));
        map.put("rows", tUserService.queryUserByPage(pageNow, pageSize, column, value));
        return tUserService.queryUserByPage(pageNow, pageSize, column, value);
    }

    /**
     * 統計總人數
     *
     * @param column
     * @param value
     * @return
     */
    @GetMapping(value = "/queryUserCount")
    public Integer queryUserCount(@RequestParam(value = "column", required = false) String column,
                                  @RequestParam(value = "value", required = false) String value) {
        return tUserService.queryUserCount(column, value);

    }

    /**
     * 根據ID查詢使用者
     *
     * @param id
     * @return
     */
    @GetMapping(value = "/queryUserById")
    public User queryUserById(@RequestParam(value = "id") Integer id) {
        //從資料庫中查詢
        return tUserService.queryUserById(id);

    }

}
           

• json格式資料接收

/**
 * (TUser)表控制層-JSON方式送出
 *
 * @author makejava
 * @since 2020-03-16 16:54:52
 */
@RestController
@RequestMapping(value = "/restUserManager")
public class RestUserController {
    private static Logger logger = LoggerFactory.getLogger(RestUserController.class);

    @Autowired
    private IUserServiceImpl iUserService;

    /**
     * 使用者登陸
     *
     * @param user
     * @return
     */
    @PostMapping(value = "userLogin")
    public User userLogin(@RequestBody User user) {
        return iUserService.queryUserByNameAndPassword(user);
    }

    /**
     * 注冊使用者
     *
     * @param user
     * @param multipartFile
     * @return
     * @throws IOException
     */
    @PostMapping(value = "addUser")
    public User addUser(@RequestPart(value = "user") User user, @RequestParam(value = "multipartFile", required = false) MultipartFile multipartFile) throws IOException {
        //判斷上傳的檔案是否為空
        if (multipartFile != null) {
            //擷取檔案的原始名
            String filename = multipartFile.getOriginalFilename();
            //擷取檔案類型
            String substring = filename.substring(filename.lastIndexOf("."));
            //建立臨時檔案
            File tempFile = File.createTempFile(filename.substring(0, filename.lastIndexOf(".")), substring);
            //列印臨時檔案名
            System.out.println(tempFile.getName());
            //删除臨時檔案名
            tempFile.delete();
        }
        iUserService.saveUser(user);
        return user;
    }

    public void updataUser(@RequestPart(value = "user") User user, @RequestParam(value = "multipartFile", required = false) MultipartFile multipartFile) throws IOException {
        //判斷上傳的檔案是否為空
        if (multipartFile != null) {
            //擷取檔案的原始名
            String filename = multipartFile.getOriginalFilename();
            //擷取檔案類型
            String substring = filename.substring(filename.lastIndexOf("."));
            //建立臨時檔案
            File tempFile = File.createTempFile(filename.substring(0, filename.lastIndexOf(".")), substring);
            //列印臨時檔案名
            System.out.println(tempFile.getName());
            //删除臨時檔案名
            tempFile.delete();
        }
        //修改使用者資訊
        iUserService.updateUser(user);
    }

    /**
     * 删除使用者
     *
     * @param ids
     */
    @DeleteMapping(value = "deleteUserByIds")
    public void deleteUserByIds(@RequestParam(value = "ids") Integer[] ids) {
        iUserService.deleteByUserIds(ids);
    }

    /**
     * 根據id查一個
     *
     * @param id
     * @return
     */
    @GetMapping(value = "queryById")
    public User queryById(@RequestParam(value = "id") Integer id) {
        return iUserService.queryUserById(id);
    }

    /**
     * 模糊分頁查詢
     *
     * @param pageNow
     * @param pageSize
     * @param column
     * @param value
     * @return
     */
    @GetMapping(value = "/queryUserByPage")
    public List<User> queryUserByPage(@RequestParam(value = "page", defaultValue = "1")
                                              Integer pageNow,
                                      @RequestParam(value = "rows", defaultValue =
                                              "10") Integer pageSize,
                                      @RequestParam(value = "column", required = false)
                                              String column,
                                      @RequestParam(value = "value", required = false)
                                              String value) {
        HashMap<String, Object> results = new HashMap<>();
        results.put("total", iUserService.queryUserCount(column, value));

        results.put("rows", iUserService.queryUserByPage(pageNow, pageSize, column, value));
        return iUserService.queryUserByPage(pageNow, pageSize, column, value);
    }

    /**
     * 統計總人數
     *
     * @param column
     * @param value
     * @return
     */
    @GetMapping(value = "/queryUserCount")
    public Integer queryUserCount(@RequestParam(value = "column", required = false) String column,
                                  @RequestParam(value = "value", required = false) String value) {
        return iUserService.queryUserCount(column, value);

    }
}
           

RestTemplate

• FormUserControllerTests

/**
 * controller測試類
 */
@SpringBootTest(classes = {Application.class})
@RunWith(SpringRunner.class)
public class FormUserControllerTests {
    @Resource
    private RestTemplate restTemplate;

    private String urlPrefix = "http://127.0.0.1:8888/formUserManager";

    /*
     @PostMapping(value = "/registerUser")
     public User registerUser(User user,
     @RequestParam(value = "multipartFile",required = false)
    MultipartFile multipartFile)
    */
    @Test
    public void testRegisterUser() {
        String url = urlPrefix + "/registerUser";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>();
        formData.add("name", "李⼩四");
        formData.add("password", "123456");
        formData.add("sex", "true");
        formData.add("birthDay", "2018-01-26");
        formData.add("photo", "user.png");
        formData.add("email", "[email protected]");
        //模拟⽂件上傳
        FileSystemResource fileSystemResource = new
                FileSystemResource("/Users/admin/Desktop/head.png");
        formData.add("multipartFile", fileSystemResource);
        User user = restTemplate.postForObject(url, formData, User.class);
        assertNotEquals("⽤戶ID", user.getId());
    }

    /*
     @PostMapping(value = "/userLogin")
     public User userLogin(User user)
    */
    @Test
    public void testUserLogin() {
        String url = urlPrefix + "/userLogin";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>
                ();
        formData.add("name", "李⼩四");
        formData.add("password", "123456");
        User user = restTemplate.postForObject(url, formData, User.class);
        assertNotEquals("⽤戶ID", user.getId());
    }

    /*
    @PostMapping(value = "/addUser")
    public User addUser(User user,
    @RequestParam(value = "multipartFile",required = false)
   MultipartFile multipartFile) throws IOException
    */
    @Test
    public void testAddUser() {
        String url = urlPrefix + "/addUser";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>();
        formData.add("name", "趙曉麗");
        formData.add("password", "123456");
        formData.add("sex", "true");
        formData.add("birthDay", "2018-01-26");
        formData.add("photo", "user.png");
        formData.add("email", "[email protected]");
        //模拟⽂件上傳
        FileSystemResource fileSystemResource = new
                FileSystemResource("/Users/admin/Desktop/head.png");
        formData.add("multipartFile", fileSystemResource);
        User user = restTemplate.postForObject(url, formData, User.class);
        assertNotNull("⽤戶ID", user.getId());
    }

    /*
     @PutMapping(value = "/updateUser")
     public void updateUser(User user,
     @RequestParam(value = "multipartFile",required = false)
    MultipartFile multipartFile) throws IOException
     */
    @Test
    public void testUpdateUser() {
        String url = urlPrefix + "/updateUser";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>
                ();
        formData.add("id", "6");
        formData.add("name", "趙曉麗");
        formData.add("password", "123456");
        formData.add("sex", "true");
        formData.add("birthDay", "2018-01-27");
        formData.add("photo", "user1.png");
        formData.add("email", "[email protected]");
        //模拟⽂件上傳
        FileSystemResource fileSystemResource = new
                FileSystemResource("/Users/admin/Desktop/head.png");
        formData.add("multipartFile", fileSystemResource);
        restTemplate.put(url, formData);
    }

    /*
     @DeleteMapping(value = "/deleteUserByIds")
     public void delteUserByIds(@RequestParam(value = "ids") Integer[] ids)
    */
    @Test
    public void testDeleteUser() {
        String url = urlPrefix + "/deleteUserByIds?ids={id}";
        Map<String, Object> parameters = new HashMap<String, Object>();
        parameters.put("id", "4,5,6");
        restTemplate.delete(url, parameters);
    }

    /*@GetMapping(value = "/queryUserByPage")
     public List<User> queryUserByPage(@RequestParam(value = "page",defaultValue = "1")
    Integer pageNow,
     @RequestParam(value = "rows",defaultValue =
    "10") Integer pageSize,
     @RequestParam(value = "column",required = false)
    String column,
     @RequestParam(value = "value",required = false)
    String value)
    */
    @Test
    public void testQueryUserByPage() {
        String url = urlPrefix + "/queryUserByPage?page={page}&rows={rows}&column= { column }&value = {value} ";
        Map<String, Object> params = new HashMap<String, Object>();
        params.put("page", 1);
        params.put("rows", 10);
        params.put("column", "name");
        params.put("value", "⼩");
        User[] users = restTemplate.getForObject(url, User[].class, params);
        assertNotNull("⽤戶資料", users);
    }

    /*
     @GetMapping(value = "/queryUserCount")
     public Integer queryUserCount(@RequestParam(value = "column",required = false)
    String column,
     @RequestParam(value = "value",required = false)
    String value)
    */
    @Test
    public void testQueryUserCount() {
        String url = urlPrefix + "/queryUserCount?column={column}&value={value}";
        //模拟表單資料
        Map<String, Object> parameters = new HashMap<>();
        parameters.put("column", "name");
        parameters.put("value", "曉");
        Integer count = restTemplate.getForObject(url, Integer.class, parameters);
        assertNotNull(count);
    }

    /*
    @GetMapping(value = "/queryUserById")
    public User queryUserById(@RequestParam(value = "id") Integer id)
    */
    @Test
    public void testQueryUserById() {
        String url = urlPrefix + "/queryUserById?id={id}";
        //模拟表單資料
        Map<String, Object> parameters = new HashMap<>();
        parameters.put("id", "2");
        User user = restTemplate.getForObject(url, User.class, parameters);
        assertNotNull("⽤戶ID", user.getId());
    }
}
           

• RestUserControllerTests

/**
 * RestUserController測試類
 */
@SpringBootTest(classes = {Application.class})
@RunWith(SpringRunner.class)
public class RestUserControllerTests {
    @Resource
    private RestTemplate restTemplate;
    private String urlPrefix = "http://127.0.0.1:8888/restUserManager";

    /*
     @PostMapping(value = "/registerUser")
     public User registerUser(@RequestPart(value = "user") User user,
     @RequestParam(value = "multipartFile",required = false)
    MultipartFile multipartFile) throws IOException {
    */
    @Test
    public void testRegisterUser() {
        String url = urlPrefix + "/registerUser";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>
                ();
        User user = new User(null, "張曉磊", true, "123456", new Date(), "aa.png", "[email protected]");
        formData.add("user", user);
        //模拟⽂件上傳
        FileSystemResource fileSystemResource = new
                FileSystemResource("/Users/admin/Desktop/head.png");
        formData.add("multipartFile", fileSystemResource);
        User registerUser = restTemplate.postForObject(url, formData, User.class);
        assertNotEquals("⽤戶ID", registerUser.getId());
    }

    /*
    @PostMapping(value = "/userLogin")
    public User userLogin(@RequestBody User user)
    */
    @Test
    public void testUserLogin() {
        String url = urlPrefix + "/userLogin";
        //模拟表單資料
        User user = new User();
        user.setName("張曉磊");
        user.setPassword("123456");
        User loginUser = restTemplate.postForObject(url, user, User.class);
        assertNotEquals("⽤戶ID", loginUser.getId());
    }

    /*
    @PostMapping(value = "/addUser")
    public User addUser(@RequestPart(value = "user") User user,
    @RequestParam(value = "multipartFile",required = false)
   MultipartFile multipartFile) throws IOException
    */
    @Test
    public void testAddUser() {
        String url = urlPrefix + "/addUser";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>
                ();
        User user = new User(null, "溫曉琪", true, "123456", new Date(), "aa.png", "[email protected]");
        formData.add("user", user);
        //模拟⽂件上傳
        FileSystemResource fileSystemResource = new
                FileSystemResource("/Users/admin/Desktop/head.png");
        formData.add("multipartFile", fileSystemResource);
        User dbUser = restTemplate.postForObject(url, formData, User.class);
        assertNotNull("⽤戶ID", dbUser.getId());
    }

    /*
     @PutMapping(value = "/updateUser")
     public void updateUser(@RequestPart(value = "user") User user,
     @RequestParam(value = "multipartFile",required = false)
    MultipartFile multipartFile) throws IOException {
     */
    @Test
    public void testUpdateUser() {
        String url = urlPrefix + "/updateUser";
        //模拟表單資料
        MultiValueMap<String, Object> formData = new LinkedMultiValueMap<String, Object>
                ();
        User user = new User(20, "溫曉琪", false, "123456", new
                Date(), "aa.png", "[email protected]");
        user.setId(9);
        formData.add("user", user);
        //模拟⽂件上傳
        FileSystemResource fileSystemResource = new
                FileSystemResource("/Users/admin/Desktop/head.png");
        formData.add("multipartFile", fileSystemResource);
        restTemplate.put(url, formData);
    }

    /*
    @DeleteMapping(value = "/deleteUserByIds")
    public void delteUserByIds(@RequestParam(value = "ids") Integer[] ids)
    */
    @Test
    public void testDeleteUser() {
        String url = urlPrefix + "/deleteUserByIds?ids={id}";
        Map<String, Object> parameters = new HashMap<String, Object>();
        parameters.put("id", "4,5,6");
        restTemplate.delete(url, parameters);
    }

    /*
     @GetMapping(value = "/queryUserByPage")
     public List<User> queryUserByPage(@RequestParam(value = "page",defaultValue = "1")
    Integer pageNow,
     @RequestParam(value = "rows",defaultValue =
    "10") Integer pageSize,
     @RequestParam(value = "column",required = false)
    String column,
     @RequestParam(value = "value",required = false)
    String value)
    */
    @Test
    public void testQueryUserByPage() {
        String url = urlPrefix + "/queryUserByPage?page={page}&rows={rows}&column= { column }&value = {value} ";
        Map<String, Object> params = new HashMap<String, Object>();
        params.put("page", 1);
        params.put("rows", 10);
        params.put("column", "name");
        params.put("value", "⼩");
        User[] users = restTemplate.getForObject(url, User[].class, params);
        assertNotNull("⽤戶資料", users);
    }

    /*
     @GetMapping(value = "/queryUserCount")
     public Integer queryUserCount(@RequestParam(value = "column",required = false)
    String column,
     @RequestParam(value = "value",required = false)
    String value)
    */
    @Test
    public void testQueryUserCount() {
        String url = urlPrefix + "/queryUserCount?column={column}&value={value}";
        //模拟表單資料
        Map<String, Object> parameters = new HashMap<>();
        parameters.put("column", "name");
        parameters.put("value", "曉");
        Integer count = restTemplate.getForObject(url, Integer.class, parameters);
        assertNotNull(count);
    }

    /*
    @GetMapping(value = "/queryUserById")
    public User queryUserById(@RequestParam(value = "id") Integer id)
    */
    @Test
    public void testQueryUserById() {
        String url = urlPrefix + "/queryUserById?id={id}";
        //模拟表單資料
        Map<String, Object> parameters = new HashMap<>();
        parameters.put("id", "2");
        User user = restTemplate.getForObject(url, User.class, parameters);
        assertNotNull("⽤戶ID", user.getId());
    }
}
           

• Application

/**
 * SpringBoot入口類
 */
@SpringBootApplication
@MapperScan(basePackages = "com.baizhi.dao")
public class Application extends SpringBootServletInitializer {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    public SpringApplicationBuilder configure(SpringApplicationBuilder builder) {
        return builder.sources(Application.class);
    }

    @Bean
    public RedisTemplate redisTemplate() {
        return new RedisTemplate();
    }

    @Bean
    public RedisTemplate redisTemplate(RedisConnectionFactory redisConnectionFactory) {
        RedisTemplate redisTemplate = new RedisTemplate();
        redisTemplate.setConnectionFactory(redisConnectionFactory);
        redisTemplate.setValueSerializer(new GenericJackson2JsonRedisSerializer());
        return redisTemplate;
    }
}
           

內建Redis實作⼆級緩存

• 環境準備

  虛拟機已安裝Redis，并正常運作

• ApplicationContextHolder

/**
 * 該接口為标記接口、Spring工廠在初始化的時候會自動注入applicationContext
 */
@Component
public class ApplicationContextHolder implements ApplicationContextAware {
    private static ApplicationContext applicationContext = null;

    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.applicationContext = applicationContext;
    }

    public static Object getBean(String beanName) {
        return applicationContext.getBean(beanName);
    }
}
           

• UserDefineRedisCache

/**
 * 內建Redis開啟Mybatis二級緩存
 */
public class UserDefineRedisCache implements Cache {
    private static final Logger LOGGER = LoggerFactory.getLogger(UserDefineRedisCache.class);
    //記錄的是Mapper的namespace
    private String id;
    private final ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
    private RedisTemplate redisTemplate = (RedisTemplate) ApplicationContextHolder.getBean("redisTemplate");


    public UserDefineRedisCache(String id) {
        this.id = id;
    }

    @Override
    public String getId() {
        return id;
    }

    @Override
    public void putObject(Object key, Object value) {
        LOGGER.debug("将查詢結果緩存到Redis");
        ValueOperations operations = redisTemplate.opsForValue();
        operations.set(key, value, 30, TimeUnit.MINUTES);
    }

    @Override
    public Object getObject(Object key) {
        LOGGER.debug("擷取緩存結果");
        ValueOperations opsForValue = redisTemplate.opsForValue();
        return opsForValue.get(key);
    }

    @Override
    public Object removeObject(Object key) {
        LOGGER.debug("删除Redis中的key:" + key);
        ValueOperations opsForValue = redisTemplate.opsForValue();
        Object value = opsForValue.get(key);
        redisTemplate.delete(key);
        return value;
    }

    @Override
    public void clear() {
        LOGGER.debug("删除所有Redis中的緩存");
        redisTemplate.execute((RedisCallback) connection -> {
            connection.flushDb();
            return null;
        });
    }

    @Override
    public int getSize() {
        return 0;
    }

    @Override
    public ReadWriteLock getReadWriteLock() {
        return readWriteLock;
    }
}
           

實作Mysql的讀寫分離

背景

⼀個項⽬中資料庫最基礎同時也是最主流的是單機資料庫，讀寫都在⼀個庫中。當⽤戶逐漸增多，單機資料庫⽆法滿⾜性能要求時，就會進⾏讀寫分離改造（适⽤于讀多寫少），寫操作⼀個庫，讀操作多個庫，通常會做⼀個資料庫叢集，開啟主從備份，⼀主多從，以提⾼讀取性能。當⽤戶更多讀寫分離也⽆法滿⾜時，就需要分布式資料庫了-NoSQL。 正常情況下讀寫分離的實作，⾸先要做⼀個⼀主多從的資料庫叢集，同時還需要進⾏資料同步。

資料庫主從搭建

Master配置

• 修改/etc/my.cnf

[mysqld]
server-id=1
log-bin=mysql-bin
log-slave-updates
slave-skip-errors=all
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

symbolic-links=0

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
           

• 重新開機MySQL服務

• 登入MySQL主機檢視狀态

[[email protected] ~]# mysql -uroot -proot
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.42-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>  show master status\G;
*************************** 1. row ***************************
             File: mysql-bin.000007
         Position: 120
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)

ERROR:
No query specified
           

Slave 配置

• 修改/etc/my.cnf⽂件

[mysqld]
server-id=2
log-bin=mysql-bin
log-slave-updates
slave-skip-errors=all
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
symbolic-links=0

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
           

• 重新開機MySQL服務

systemctl restart mysqld
           

• MySQL配置從機

[[email protected] ~]# mysql -u root -proot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.73-log Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> change master to
    -> master_host='192.168.157.196',
    -> master_user='root',
    -> master_password='root',
    -> master_log_file='mysql-bin.000007',
    -> master_log_pos=120;
Query OK, 0 rows affected, 2 warnings (0.00 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.157.196
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000007
          Read_Master_Log_Pos: 120
               Relay_Log_File: mysqld-relay-bin.000002
                Relay_Log_Pos: 283
        Relay_Master_Log_File: mysql-bin.000007
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 120
              Relay_Log_Space: 457
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 012f29df-4688-11ea-b41c-000c297c7433
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
1 row in set (0.00 sec)

ERROR:
No query specified
           

讀寫分離實作

讀寫分離要做的事情就是對于⼀條SQL該選擇哪個資料庫去執⾏，⾄于誰來做選擇資料庫這件事⼉，⽆⾮兩個，要麼中間件幫我們做，要麼程式⾃⼰做。是以，⼀般來講，讀寫分離有兩種實作⽅式。第⼀種是依靠中間件（⽐如：MyCat），也就是說應⽤程式連接配接到中間件，中間件幫我們做SQL分離；第⼆種是應⽤程式⾃⼰去做分離。

編碼思想

所謂的⼿寫讀寫分離，需要⽤戶⾃定義⼀個動态的資料源，該資料源可以根據目前上下⽂中調⽤⽅法是讀或者是寫⽅法決定傳回主庫的連結還是從庫的連結。這⾥我們使⽤Spring提供的⼀個代理資料源AbstractRoutingDataSource接⼝。

該接⼝需要⽤戶完善⼀個determineCurrentLookupKey抽象法，系統會根據這個抽象傳回值決定使⽤系統中定義的資料源。

@Nullable
protected abstract Object determineCurrentLookupKey();
           

其次該類還有兩個屬性需要指定

defaultTargetDataSource

和

targetDataSources

，其中defaultTargetDataSource需要指定為Master資料源。targetDataSources是⼀個Map需要将所有的資料源添加到該Map中，以後系統會根據determineCurrentLookupKey⽅法的傳回值作為key從targetDataSources查找相應的實際資料源。如果找不到則使⽤defaultTargetDataSource指定的資料源。

實作步驟

• 配置application.yml

# 通路接口配置
server:
  port: 8989
  servlet:
    context-path: /UserModel
spring:
  # 資料源配置---單機模式
  #  datasource:
  #    type: com.alibaba.druid.pool.DruidDataSource
  #    username: root
  #    password: root
  #    driver-class-name: com.mysql.jdbc.Driver
  #    url: jdbc:mysql://CentOS:3306/usermodel?useSSL=false&characterEncoding=UTF8&serverTimezone=GMT
  #  資料源配置---叢集模式
  datasource:
    # 主節點
    master:
      type: com.alibaba.druid.pool.DruidDataSource
      username: root
      password: root
      driver-class-name: com.mysql.jdbc.Driver
      jdbc-url: jdbc:mysql://CentOSA:3306/usermodel?useUnicode=true&characterEncoding=UTF8&serverTimezone=UTC&useSSL=false
    # 從節點1
    slave1:
      type: com.alibaba.druid.pool.DruidDataSource
      username: root
      password: root
      driver-class-name: com.mysql.jdbc.Driver
      jdbc-url: jdbc:mysql://CentOSB:3306/usermodel?useUnicode=true&characterEncoding=UTF8&serverTimezone=UTC&useSSL=false
    # 從節點2
    slave2:
      type: com.alibaba.druid.pool.DruidDataSource
      username: root
      password: root
      driver-class-name: com.mysql.jdbc.Driver
      jdbc-url: jdbc:mysql://CentOSC:3306/usermodel?useUnicode=true&characterEncoding=UTF8&serverTimezone=UTC&useSSL=false

    # 上傳檔案控制
  servlet:
    multipart:
      max-request-size: 50MB
      max-file-size: 50MB
      enabled: true
  # 亂碼解決
  http:
    encoding:
      charset: utf-8
  # 連接配接redis
  redis:
    host: CentOS
    port: 6379

# Mybatis配置
mybatis:
  mapper-locations: classpath:mapper/*.xml
  type-aliases-package: com.baizhi.entity
  # //在resource目錄下建立config檔案夾
  config-location: classpath:config/mybatis.xml
  # 開啟Mybatis批處理模式
logging:
  level:
    root: info
           

• 配置資料源

/**
 * 自定義資料源
 */
@Configuration
public class UserDefineDatasourceConfig {

    /**
     * 主節點
     *
     * @return
     */
    @Bean
    @ConfigurationProperties("spring.datasource.master")
    public DataSource masterDatasource() {
        return DataSourceBuilder.create().build();
    }

    /**
     * 從節點1
     *
     * @return
     */
    @Bean
    @ConfigurationProperties("spring.datasource.slave1")
    public DataSource slave1Datasource() {
        return DataSourceBuilder.create().build();
    }

    /**
     * 從節點2
     *
     * @return
     */
    @Bean
    @ConfigurationProperties("spring.datasource.slave2")
    public DataSource slave2Datasource() {
        return DataSourceBuilder.create().build();
    }

    /**
     * 資料源配置
     *
     * @param masterDatasource
     * @param slave1Datasource
     * @param slave2Datasource
     * @return
     */
    @Bean
    public DataSource proxyDataSource(@Qualifier("masterDatasource") DataSource masterDatasource,
                                 @Qualifier("slave1Datasource") DataSource slave1Datasource,
                                 @Qualifier("slave2Datasource") DataSource slave2Datasource
    ) {
        DataSourceProxy proxy = new DataSourceProxy();
        //設定預設資料源
        proxy.setDefaultTargetDataSource(masterDatasource);
        HashMap<Object, Object> map = new HashMap<>();
        map.put("master", masterDatasource);
        map.put("slave1", slave1Datasource);
        map.put("slave2", slave2Datasource);
        //注冊所有資料源
        proxy.setTargetDataSources(map);
        return proxy;
    }

    /**
     * 覆寫SqlSessionFactory自定義資料源
     *
     * @param dataSource
     * @return
     * @throws Exception
     */
    @Bean
    public SqlSessionFactory sqlSessionFactory(@Qualifier("proxyDataSource") DataSource dataSource) throws Exception {
        SqlSessionFactoryBean sqlSessionFactoryBean = new SqlSessionFactoryBean();
        sqlSessionFactoryBean.setDataSource(dataSource);
        sqlSessionFactoryBean.setTypeAliasesPackage("com.baizhi.entity");
        sqlSessionFactoryBean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath*:mapper/*.xml"));

        SqlSessionFactory sqlSessionFactory = sqlSessionFactoryBean.getObject();

        return sqlSessionFactory;
    }

    /**
     * 覆寫SqlSessionTemplate，開啟BATCH處理模式
     *
     * @param sqlSessionFactory
     * @return
     */
    @Bean
    public SqlSessionTemplate sqlSessionTemplate(@Qualifier("sqlSessionFactory") SqlSessionFactory sqlSessionFactory) {
        ExecutorType executorType = ExecutorType.BATCH;
        if (executorType != null) {
            return new SqlSessionTemplate(sqlSessionFactory, executorType);
        } else {
            return new SqlSessionTemplate(sqlSessionFactory);
        }
    }

    /**
     * 事務手動注入，否則事務不生效
     *
     * @param dataSource
     * @return
     */
    @Bean
    public PlatformTransactionManager platformTransactionManager(@Qualifier("proxyDataSource") DataSource dataSource) {
        return new DataSourceTransactionManager(dataSource);

    }

}
           

• 配置切⾯

/**
 * 自定義切面，負責讀取slaveDB注解，并且在DBTypeContextHolder中設定讀寫類型
 */
@Aspect
@Order(0)
@Component
public class UserDefineDataSourceAOP {
    private static final Logger LOGGER = LoggerFactory.getLogger(UserDefineDataSourceAOP.class);

    /**
     * 配置環繞切面，該切面的作用是設定目前上下文的讀寫類型
     *
     * @param pjp
     * @return
     */
    @Around("execution(* com.baizhi.service..*.*(..))")
    public Object methodInterceptor(ProceedingJoinPoint pjp){
        Object result = null;
        try {
            //擷取目前的方法資訊
            MethodSignature methodSignature = (MethodSignature) pjp.getSignature();
            Method method = methodSignature.getMethod();
            //判斷方法上是否存在注解@SlaveDB
            boolean present = method.isAnnotationPresent(SlaveDB.class);

            OperType operType=null;
            if(!present){
                operType=OperType.WRIRTE;
            }else{
                operType=OperType.READ;
            }
            OPTypeContextHolder.setOperType(operType);
            LOGGER.debug("目前操作："+operType);
            result = pjp.proceed();
            //清除線程變量
            OPTypeContextHolder.clear();
        } catch (Throwable throwable) {
            throwable.printStackTrace();
        }
        return result;
    }
}
           

• 動态資料源

/**
 * 配置動态資料源
 */
public class DataSourceProxy extends AbstractRoutingDataSource {
    private static final Logger logger = LoggerFactory.getLogger(DataSourceProxy.class);

    private String masterDBKey = "master";
    private List<String> slaveDBKeys = Arrays.asList("slave1", "slave2");

    private static final AtomicInteger round = new AtomicInteger(0);

    /**
     * 需要在該方法中，判斷目前使用者的操作是讀操作還是寫操作
     *
     * @return
     */
    @Override
    protected Object determineCurrentLookupKey() {
        String dbKey = null;
        OperType operType = OPTypeContextHolder.getOperType();
        if (operType.equals(OperType.WRIRTE)) {
            dbKey = masterDBKey;
        } else {
            //輪詢傳回  0 1 2 3 4 5 6
            int value = round.getAndIncrement();
            if (value < 0) {
                round.set(0);
            }
            Integer index = round.get() % slaveDBKeys.size();

            dbKey = slaveDBKeys.get(index);
        }
        logger.debug("目前的DBkey:" + dbKey);
        return dbKey;
    }
}
           

• 讀寫類型

/**
* 定義操作類型
*/
public enum OperType {
 WRIRTE,READ; 
 }
           

• 記錄操作類型

/**
 * 該類主要是用于存儲，目前使用者的操作類型，将目前的操作存儲在目前線程的上下文中
 */
public class OPTypeContextHolder {
    private static final ThreadLocal<OperType> OPER_TYPE_THREAD_LOCAL=new ThreadLocal<>();

    public static void setOperType(OperType operType){
        OPER_TYPE_THREAD_LOCAL.set(operType);
    }
    public static OperType getOperType(){
        return OPER_TYPE_THREAD_LOCAL.get();
    }
    public static void clear(){
        OPER_TYPE_THREAD_LOCAL.remove();
    }
}
           

• 業務⽅法标記注解

/**
 * 該注解用于标注，目前使用者的調用方法是讀還是寫
 */
@Retention(RetentionPolicy.RUNTIME) //表示運作時解析注解
@Target(value = {ElementType.METHOD})//表示隻能在方法上加
public @interface SlaveDB {
}
           

• logback.xml配置

<logger name="com.baizhi.datasource" level="DEBUG" additivity="false">
 <appender-ref ref="STDOUT" />
 </logger>
           

• 代碼結構

  

分布式檔案系統內建

概述

分布式⽂件系統（Distributed File System）是指⽂件系統管理的實體存儲資源不⼀定直接連接配接在本地節點上，⽽是通過計算機⽹絡與節點相連。

計算機通過⽂件系統管理、存儲資料，⽽資訊爆炸時代中⼈們可以擷取的資料成指數倍的增⻓，單純通過增加硬碟個數來擴充計算機⽂件系統的存儲容量的⽅式，在容量⼤⼩、容量增⻓速度、資料備份、資料安全等⽅⾯的表現都差強⼈意。分布式⽂件系統可以有效解決資料的存儲和管理難題：将固定于某個地點的某個⽂件系統，擴充到任意多個地點/多個⽂件系統，衆多的節點組成⼀個⽂件系統⽹絡。每個節點可以分布在不同的地點，通過⽹絡進⾏節點間的通信和資料傳輸。⼈們在使⽤分布式⽂件系統時，⽆需關⼼資料是存儲在哪個節點上、或者是從哪個節點從擷取的，隻需要像使⽤本地⽂件系統⼀樣管理和存儲⽂件系統中的資料。

⽂件系統最初設計時，僅僅是為局域⽹内的本地資料服務的。⽽分布式⽂件系統将服務範圍擴充到了整個⽹絡。不僅改變了資料的存儲和管理⽅式，也擁有了本地⽂件系統所⽆法具備的資料備份、資料安全等優點。判斷⼀個分布式⽂件系統是否優秀，取決于以下三個因素：

資料的存儲⽅式：例如有1000萬個資料⽂件，可以在⼀個節點存儲全部資料⽂件，在其他N個節點上每個節點存儲1000/N萬個資料⽂件作為備份；或者平均配置設定到N個節點上存儲，每個節點上存儲1000/N萬個資料⽂件。⽆論采取何種存儲⽅式，⽬的都是為了保證資料的存儲安全⽅便擷取。

資料的讀取速率：包括響應⽤戶讀取資料⽂件的請求、定位資料⽂件所在的節點、讀取實際硬碟中資料⽂件的時間、不同節點間的資料傳輸時間以及⼀部分處理器的處理時間等。各種因素決定了分布式⽂件系統的⽤戶體驗。即分布式⽂件系統中資料的讀取速率不能與本地⽂件系統中資料的讀取速率相差太⼤，否則在本地⽂件系統中打開⼀個⽂件需要2秒，⽽在分布式⽂件系統中各種因素的影響下⽤時超過10秒，就會嚴重影響⽤戶的使⽤體驗。

資料的安全機制：由于資料分散在各個節點中，必須要采取備援、備份、鏡像等⽅式保證節點出現故障的情況下，能夠進⾏資料的恢複，確定資料安全。

⽂件系統分類

• 塊存儲：MongoDB資料庫中的GridFS、Hadoop中的HDFS，這些系統在存儲⽂件的的時候會嘗試先将⽂件打碎存儲（拆分成Data Block）。這樣存儲的優點可以存儲超⼤型⽂件，更加⾼效的利⽤磁盤資源。但是需要額外存儲⽂件碎⽚的中繼資料資訊。

在塊存儲中HDFS存儲的塊128MB，但是在MongoDB中預設Chunk 255 KB，雖然都⽀持塊存儲但是應⽤場景有很⼤差異。HDFS使⽤于超⼤⽂本⽇志⽂件存儲。但是MongoDB适合存儲超⼤的流媒體⽂件例如操⼤的⾳頻和視訊，可以實作流媒體資料流的區間加載。

• ⽂件存儲:GlusterFS、NFS、FastDFS等都是基于⽂件機關存儲，這種存儲并不會将⽂件系統打碎。⽽是⽂件存儲到系統中的某⼀台伺服器中。這樣存儲的優點可以應對⼀些⼩⽂件系統，系統維護簡單，⽆需存儲⽂件的中繼資料，系統設計和維護成本低。

FastDFS 介紹

特點

FastDFS 是⼀款開源的輕量級分布式⽂件系統如下特點：

• 純粹C語⾔實作，⽀持Linux、FreeBSD等unix系統。
• 類似GoogleFS/HDFS，但是不是通⽤的⽂件系統，隻能通過專有的API通路，⽬前提供了C、Java和PHPAPI 互聯⽹量身定做，最求⾼性能，⾼擴充.
• FastDFS不僅僅可以存儲⽂件，還可以存儲⽂件的中繼資料資訊（可選）。

架構

整個FastDFS架構中有Client、Tracker和Storage服務。其中Client⽤于送出⽂件給FastDFS叢集。Storage 服務負責實際資料的存儲，Tracker服務負責監控和排程Storage服務，起到負載均衡器的作⽤.

如果Storage的中

卷

是⼀樣的也就意味着這些服務彼此資料互相備份，實作資料的備援備份。不同

卷

存儲整個叢集中的部分⽂件，類似于傳統單機⽂件系統的的分區概念（C槽、D盤、…）Tracker Server 主要做⼯作排程，在通路上起負載均衡的作⽤。在記憶體中記錄叢集中group/

卷

和Storage server的狀态資訊，是連接配接Client和Storage Server的樞紐。因為相關資訊存儲在記憶體中，是以Tracker Server性能⾮常⾼,⼀個較⼤的叢集（上百個group/

卷

）中3台就夠了。

StorageServer ：存儲伺服器，⽂件和⽂件屬性資訊（meta資料）都存儲在伺服器的磁盤上。

上傳、下載下傳機制

• Storage Server會定期的向Tracker伺服器彙報⾃身狀态資訊，例如健康狀态和存儲容量。
• Client連接配接Tracker Server發送⽂件請求。
• Tracker Server根據注冊的Storage Server的資訊傳回⼀台可⽤的Storage Server的調⽤資訊。
• Client拿到資訊後直接連接配接對應的Storage Server進⾏點到點的⽂件上傳（附加中繼資料-可選）。
• Storage Server收到⽂件請求後會根據⾃⼰位置資訊⽣成File_ID資訊，并且将File_ID和⽤戶攜帶的中繼資料資訊進⾏關聯，然後将File_ID傳回給Client。
• 傳回的File_ID⽂件伺服器并不會存儲，需要Client端保留在外圍資料庫中，以後Client端可以通過File_ID下載下傳對應的⽂件或者中繼資料。

• Storage Server會定期的向Tracker伺服器彙報⾃身狀态資訊，例如健康狀态和存儲容量。
• Client連接配接Tracker Server攜帶File_ID參數發送⽂件下載下傳請求。
• Tracker Server根據注冊的Storage Server的資訊傳回⼀台可⽤的Storage Server的調⽤資訊(主從伺服器)。
• Client拿到資訊後直接連接配接對應的Storage Server進⾏點到點的⽂件下載下傳（讀取中繼資料）。
• Storage Server收到⽂件請求後解析File_ID資訊，讀取本地的⽂件流，将資料寫會Client端。

File_ID組成

File_ID是由Storage Server⽣成并傳回給Client端。File_ID包含了組/ 卷 和⽂件路徑。Storage Server可以直接根據該⽂件名定位到該⽂件。

FastDFS叢集搭建

資源下載下傳安裝

進⼊ https://github.com/happyfish100 ⽹站下載下傳FastDFS相關資源。建議使⽤⼩編給⼤家處理過的安裝包。

環境準備

準備三台機器，保證Mysql叢集正常運作

• 安裝C語言環境

具體步驟

• 1.安裝依賴包libfastcommon https://github.com/happyfish100/libfastcommon/archive/V1.0.35.tar.gz

[[email protected] ~]# tar -zxf V1.0.35.tar.gz
[[email protected] ~]# cd libfastcommon-1.0.35
[[email protected] libfastcommon-1.0.35]# ./make.sh
[[email protected] libfastcommon-1.0.35]# ./make.sh install
           

• 2.安裝FastDFS https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz

[[email protected] ~]# yum install -y perl-devel
[[email protected] ~]# tar -zxf fastdfs-5.11.tar.gz
[[email protected] ~]# cd fastdfs-5.11
[[email protected] fastdfs-5.11]# ./make.sh
[[email protected] fastdfs-5.11]# ./make.sh install
           

提示:當軟體安裝結束後，預設FastDFS啟動所需的配置⽂件放置在/etc/fdfs⽬錄下。

[[email protected] ~]# yum install -y tree
[[email protected] ~]# tree /etc/fdfs/
/etc/fdfs/
├── client.conf.sample
├── storage.conf.sample
├── storage_ids.conf.sample
└── tracker.conf.sample

0 directories, 4 files

# 可運⾏腳本
[[email protected] ~]# ls -l /etc/init.d/fdfs_*
-rwxr-xr-x. 1 root root 961 Jun 30 20:22 /etc/init.d/fdfs_storaged
-rwxr-xr-x. 1 root root 963 Jun 30 20:22 /etc/init.d/fdfs_trackerd
# 執⾏程式
[[email protected] ~]# whereis fdfs_storaged fdfs_trackerd
fdfs_storaged: /usr/bin/fdfs_storaged
fdfs_trackerd: /usr/bin/fdfs_trackerd
           

配置服務

• 1.建立fdfs運⾏所需的資料⽬錄

[[email protected] ~]# mkdir -p /data/fdfs/{tracker,storage/store01,storage/store02}
[[email protected] ~]# tree /data/
/data/
└── fdfs
    ├── storage
    │   ├── store01
    │   └── store02
    └── tracker

5 directories, 0 files
           

• 2.建立啟動所需的配置⽂件

[[email protected] ~]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf
[[email protected] ~]# cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf
[[email protected] ~]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf
[[email protected] ~]# tree /etc/fdfs/
/etc/fdfs/
├── client.conf
├── client.conf.sample
├── storage.conf
├── storage.conf.sample
├── storage_ids.conf.sample
├── tracker.conf
└── tracker.conf.sample

0 directories, 7 files
           

• 3.配置Tracker Server

[[email protected] ~]# vim /etc/fdfs/tracker.conf
base_path=/data/fdfs/tracker
           

• 4.配置Storage Server

[[email protected] ~]# vim /etc/fdfs/storage.conf
group_name=group`[1,2,3]`
base_path=/data/fdfs/storage
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122
           

• 修改Client端

[[email protected] ~]# vim /etc/fdfs/client.conf
base_path=/tmp
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122
           

啟動伺服器

[[email protected] ~]# /etc/init.d/fdfs_trackerd start
Reloading systemd:[  确定  ]
Starting fdfs_trackerd (via systemctl):[  确定  ]
[[email protected] ~]# /etc/init.d/fdfs_storaged start
Starting fdfs_storaged (via systemctl):[  确定  ]
[[email protected] ~]# ps -aux | grep fdfs
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 78950 0.0 0.1 144784 2040 ? Sl 21:06 0:00
/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root 79000 13.0 3.2 83520 67144 ? Sl 21:06 0:06
/usr/bin/fdfs_storaged /etc/fdfs/storage.conf
root 79324 0.0 0.0 103320 884 pts/0 S+ 21:07 0:00 grep fdfs
[[email protected] ~]#
           

FastDFS Shell-運維

• 上傳⽂件

[[email protected] ~]# fdfs_upload_file /etc/fdfs/client.conf t_employee
group2/M00/00/00/wKidw1522Y-AWFqVAAAC_y4JXtg0728036
           

• 下載下傳

• 資訊

[[email protected] ~]# fdfs_file_info /etc/fdfs/client.conf group2/M00/00/00/wKidw1522Y-AWFqVAAAC_y4JXtg0728036
 source storage id: 0
 source ip address: 192.168.157.195
 file create timestamp: 2020-03-22 11:20:47
 file size: 767
 file crc32: 772366040 (0x2E095ED8)
           

• 删除⽂件

• ⽂件追加

[[email protected] ~]# clear
[[email protected] ~]# echo "hello" > 1.txt
[[email protected] ~]# echo "word" > 2.txt
[[email protected] ~]# fdfs_upload_appender /etc/fdfs/client.conf /root/1.txt
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
[[email protected] ~]# fdfs_append_file /etc/fdfs/client.conf
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt 2.txt
[[email protected] ~]# fdfs_download_file /etc/fdfs/client.conf
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
[[email protected] ~]# cat wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
hello
word
           

• 監視狀态

• ⽂件校驗和

[[email protected] ~]# fdfs_crc32 /etc/fdfs/client.conf
group2/M00/00/00/wKikgl0qX4KEep_PAAAAAIpZx2Y751.txt
2911662598
           

Nginx內建FastDFS

Nginx配置安裝

• 下載下傳fastdfs-nginx-module（不建議使⽤github上，因為編譯有問題）

[[email protected] ~]# tar -zxf fastdfs-nginx-module.tar.gz
[[email protected] ~]# yum install -y pcre-devel
[[email protected] ~]# yum install -y openssl-devel
[[email protected] ~]# tar -zxf nginx-1.11.1.tar.gz
[[email protected] nginx-1.11.1]# ./configure --prefix=/usr/local/nginx-1.11.1/ --add-module=/root/fastdfs-nginx-module/src
[[email protected] nginx-1.11.1]# make
[[email protected] nginx-1.11.1]# make install
           

拷⻉配置

[[email protected] ~]# cp /root/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
[[email protected] ~]# cd /root/fastdfs-5.11/conf/
[[email protected] conf]# cp http.conf mime.types anti-steal.jpg /etc/fdfs/
           

配置nginx.conf

[[email protected] ~]# vim /usr/local/nginx-1.11.1/conf/nginx.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
 worker_connections 1024;
}
http {
 include mime.types;
 default_type application/octet-stream;
 #log_format main '$remote_addr - $remote_user [$time_local] "$request" '
 # '$status $body_bytes_sent "$http_referer" '
 # '"$http_user_agent" "$http_x_forwarded_for"';
 #access_log logs/access.log main;
 sendfile on;
 #tcp_nopush on;
 #keepalive_timeout 0;
 keepalive_timeout 65;
 #gzip on;
 server {
 listen 80;
 server_name localhost;
 #charset koi8-r;
 #access_log logs/host.access.log main;
 location ~ /group[0-9]+/M00 {
 root /data/fdfs/storage/store01;
 ngx_fastdfs_module;
 }
 location ~ /group[0-9]+/M01 {
 root /data/fdfs/storage/store02;
 ngx_fastdfs_module;
 }
 
 location / {
 root html;
 index index.html index.htm;
 }
 #error_page 404 /404.html;
 # redirect server error pages to the static page /50x.html
 #
 error_page 500 502 503 504 /50x.html;
 location = /50x.html {
 root html;
 }
 # proxy the PHP scripts to Apache listening on 127.0.0.1:80
 #
 #location ~ \.php$ {
 # proxy_pass http://127.0.0.1;
 #}
 # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
 #
 #location ~ \.php$ {
 # root html;
 # fastcgi_pass 127.0.0.1:9000;
 # fastcgi_index index.php;
 # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
 # include fastcgi_params;
 #}
 # deny access to .htaccess files, if Apache's document root
 # concurs with nginx's one
 #
 #location ~ /\.ht {
 # deny all;
 #}
 }
 # another virtual host using mix of IP-, name-, and port-based configuration
 #
 #server {
 # listen 8000;
 # listen somename:8080;
 # server_name somename alias another.alias;
 # location / {
 # root html;
 # index index.html index.htm;
 # }
 #}
 # HTTPS server
 #
 #server {
 # listen 443 ssl;
 # server_name localhost;
 # ssl_certificate cert.pem;
 # ssl_certificate_key cert.key;
 # ssl_session_cache shared:SSL:1m;
 # ssl_session_timeout 5m;
 # ssl_ciphers HIGH:!aNULL:!MD5;
 # ssl_prefer_server_ciphers on;
 # location / {
 # root html;
 # index index.html index.htm;
 # }
 #}
}
           

• 修改mod_fastdfs.conf

[[email protected] ~]# vim /etc/fdfs/mod_fastdfs.conf
tracker_server=CentOSA:22122
tracker_server=CentOSB:22122
tracker_server=CentOSC:22122
group_name=group`[1,2,3]`
url_have_group_name = true
store_path_count=2
store_path0=/data/fdfs/storage/store01
store_path1=/data/fdfs/storage/store02
           

啟動nginx

[[email protected] ~]# cd /usr/local/nginx-1.11.1/
[[email protected] nginx-1.11.1]# ./sbin/nginx -t
ngx_http_fastdfs_set pid=6207
ngx_http_fastdfs_set pid=6207
nginx: the configuration file /usr/local/nginx-1.11.1//conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.11.1//conf/nginx.conf test is successful

[[email protected] nginx-1.11.1]# ./sbin/nginx  nginx啟動
           

測試下載下傳

[[email protected] ~]# fdfs_upload_file /etc/fdfs/client.conf t_employee
group2/M01/00/00/wKidw1525qKAbNHkAAAC_y4JXtg9705075
           

随便通路⼀個nginx服務檢視效果

http://CentOS[A|B|C]/group2/M01/00/00/wKidw1525qKAbNHkAAAC_y4JXtg9705075?filename=t_employee

⽤戶在請求的時候，可以選擇性添加⽂件名，⽤于修改下載下傳的⽂件名

FastDHT⽂件去重

FastDFS除了提供了與nginx的內建，已提供了去重的⽂件解決⽅案。該解決⽅案FastDFS的作者yuqing也 在github上以FastDHT分⽀貢獻出來了。

FastDHT is a high performance distributed hash table (DHT) which based key value pairs. It can store mass key value pairs such as filename mapping, session data and user related data.

安裝

• 1.安裝BerkeleyDB 下載下傳db-4.7.25.tar.gz

[[email protected] ~]# tar -zxf db-4.7.25.tar.gz
[[email protected] ~]# cd db-4.7.25
[[email protected] db-4.7.25]# cd build_unix/
[[email protected] build_unix]# ./../dist/configure
[[email protected] build_unix]# make
[[email protected] build_unix]# make install
           

• 2.安裝FastDHT

[[email protected] ~]# tar zxf FastDHT_v2.01.tar.gz
[[email protected] ~]# cd FastDHT
[[email protected] FastDHT]# ./make.sh
[[email protected] FastDHT]# ./make.sh install
           

安裝結束後會在/etc⽬錄下産⽣fdht⽂件夾

[[email protected] FastDHT]# tree /etc/fdht/
/etc/fdht/
├── fdht_client.conf
├── fdhtd.conf
└── fdht_servers.conf
           

• 3.修改fdhtd.conf

[[email protected] ~]# mkdir /data/fastdht
[[email protected] ~]# vim /etc/fdht/fdhtd.conf
base_path=/data/fastdht
           

• 4.修改fdht_servers.conf

[[email protected] ~]# vim /etc/fdht/fdht_servers.conf
group_count = 3
group0 = CentOSA:11411
group1 = CentOSB:11411
group2 = CentOSC:11411
           

• 5.修改fdht_client.conf配置⽂件

[[email protected] ~]# vim /etc/fdht/fdht_client.conf
base_path=/tmp/
           

• 6.啟動FDHT服務

[[email protected] ~]# fdhtd /etc/fdht/fdhtd.conf start
[[email protected] ~]# ps -axu| grep fd
root       3381  0.0  0.0 276652  1660 ?        Sl   11:14   0:01 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root       3405  0.0  1.7  86396 67392 ?        Sl   11:14   0:04 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf
root      34263  1.3  0.4 198712 17340 ?        Sl   12:45   0:00 fdhtd /etc/fdht/fdhtd.conf start
root      34275  0.0  0.0 112728   972 pts/0    S+   12:45   0:00 grep --color=auto fd
           

操作FastDHT服務

• 設定值

[[email protected] ~]# fdht_set /etc/fdht/fdht_client.conf lrh:user001 name='lrh',age=24;
This is FastDHT client test program v2.01

Copyright (C) 2008, Happy Fish / YuQing

FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.

success set key count: 2, fail count: 0
           

• 讀取值

[[email protected] ~]# fdht_get /etc/fdht/fdht_client.conf lrh:user001 name,age
This is FastDHT client test program v2.01

Copyright (C) 2008, Happy Fish / YuQing

FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.

name=lrh
age=24

success get key count: 2, fail count: 0
           

• 删除值

[[email protected] ~]# fdht_delete /etc/fdht/fdht_client.conf lrh:user001 name;
This is FastDHT client test program v2.01

Copyright (C) 2008, Happy Fish / YuQing

FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.

success delete keys: name

success delete key count: 1, fail count: 0
           

內建FastDHT

• 1.修改etc/fdfs/storage.conf配置⽂件

[[email protected] ~]# vim /etc/fdfs/storage.conf
check_file_duplicate=1
keep_alive=1
#include /etc/fdht/fdht_servers.conf
           

• 2.分别啟動fdhtd服務、fastfs

[[email protected] usr]# /usr/local/bin/fdhtd /etc/fdht/fdhtd.conf restart
[[email protected] usr]# /etc/init.d/fdfs_trackerd restart
[[email protected] usr]# /etc/init.d/fdfs_storaged restart
           

• 上傳⽂件測試

[[email protected] ~]# fdfs_upload_file /etc/fdfs/client.conf t_employee
group2/M00/00/00/wKidw1528uuAEQ5NAAAC_2opwzY9266001
[[email protected] ~]# fdfs_upload_file /etc/fdfs/client.conf t_employee
group2/M00/00/00/wKidw1528weAbL6mAAAC_wNkuWg9131724

[[email protected] ~]# ls -l /data/fdfs/storage/store01/data/00/00/
總用量 8
-rw-r--r--. 1 root root 767 3月  22 11:20 wKidw1522Y-AWFqVAAAC_y4JXtg0728036
lrwxrwxrwx. 1 root root  72 3月  22 13:09 wKidw1528uuAEQ5NAAAC_2opwzY9266001 -> /data/fdfs/storage/store01/data/00/00/wKidw1528uyAHyPOAAAC_y4JXtg8258231
-rw-r--r--. 1 root root 767 3月  22 13:09 wKidw1528uyAHyPOAAAC_y4JXtg8258231
lrwxrwxrwx. 1 root root  72 3月  22 13:09 wKidw1528weAbL6mAAAC_wNkuWg9131724 -> /data/fdfs/storage/store01/data/00/00/wKidw1528uyAHyPOAAAC_y4JXtg8258231
           

可以看出系統産⽣了wKikgl0qC4KAErBTAAAixXWAIyY133.log的兩個連結

SpringBoot內建FastDFS

引⼊依賴

<dependency>
 <groupId>com.github.tobato</groupId>
 <artifactId>fastdfs-client</artifactId>
 <version>1.26.6</version>
</dependency>
           

配置application.yml

#FastDFS配置
fdfs:
  tracker-list: CentOSA:22122,CentOSB:22122,CentOSC:22122
  # 配置預設縮略圖
  thumb-image.height: 80
  thumb-image.width: 80
           

檔案上傳

@Autowired
private FastFileStorageClient fastFileStorageClient;
 /**
     * 添加使用者
     *
     * @param user
     * @param multipartFile
     * @return
     * @throws IOException
     */
    @PostMapping(value = "/addUser")
    public User addUser(User user, @RequestParam(value = "multipartFile", required = false) MultipartFile multipartFile) {
        //上傳照片
        //擷取檔案原始名
        String originalFilename = multipartFile.getOriginalFilename();
        //擷取字尾名
        String substring = originalFilename.substring(originalFilename.lastIndexOf("."));

        /*
        SpringBoot 內建FastDFS
         */
        try {
            //擷取檔案的輸入流
            InputStream inputStream = multipartFile.getInputStream();
            //                                              檔案的輸入流  上傳檔案的大小           字尾名      中繼資料
            FastImageFile fastImageFile = new FastImageFile(inputStream, inputStream.available(), substring, new HashSet<MetaData>());
            StorePath storePath = fastFileStorageClient.uploadFile(fastImageFile);
            String fullPath = storePath.getFullPath();
            //傳回的path存入mysql
            user.setPhoto(fullPath);
            tUserService.saveUser(user);
            //注冊成功傳回登陸頁
            return user;
        } catch (Exception e) {
            //注冊失敗從新注冊
            return user;
        }
    }
           

• 圖檔上傳

FileInputStream inputStream = new FileInputStream("G:/素材資料/61850.png");
FastImageFile fastImageFile=new
FastImageFile(inputStream,inputStream.available(),"png",new HashSet<MetaData>(),new
ThumbImage(150,150));
StorePath storePath = fastFileStorageClient.uploadImage(fastImageFile);
System.out.println(storePath.getFullPath());
           

• 删除⽂件

/**
     * 删除使用者
     *
     * @param ids
     */
    @DeleteMapping(value = "/deleteUserByIds")
    public void delteUserByIds(@RequestParam(value = "ids") Integer[] ids) {
        for (Integer id : ids) {
            User user = tUserService.queryUserById(id);
            fastFileStorageClient.deleteFile(user.getPhoto());
        }

        tUserService.deleteByUserIds(ids);
    }
           

-附錄：⽂件下載下傳

ByteArrayOutputStream baos = fastFileStorageClient.downloadFile("group3",
"M00/00/00/wKjvgl0prTSAMjeTAL26hhzQmiQ959.png", new
DownloadCallback<ByteArrayOutputStream>() {
 @Override
 public ByteArrayOutputStream recv(InputStream ins) throws IOException {
 ByteArrayOutputStream baos = new ByteArrayOutputStream();
 IOUtils.copy(ins, baos);
 return baos;
 }
 });
IOUtils.copy(new ByteArrayInputStream(baos.toByteArray()),new FileOutputStream("G:/素材
資料/baby.png"))