Logstash安裝及資料導入

一、安裝前準備

1）下載下傳與ES相同版本号的logstash，（7.1.0），并解壓到相應目錄

官網下載下傳位址：https://www.elastic.co/cn/downloads/logstash

華為過年鏡像位址：https://mirrors.huaweicloud.com/logstash/7.1.0/

2）下載下傳最MovieLens最小測試資料集

位址：https://grouplens.org/datasets/movielens/

對應檔案：Small: 100,000 ratings and 3,600 tag applications applied to 9,000 movies by 600 users. Last updated 9/2018.

3）運作環境需要提前安裝jdk

4）啟動Kibana、Elasticsearch

啟動安裝說明前面章節已經描述

啟動Kibana: https://blog.csdn.net/qq_36918149/article/details/104224625

啟動Elasticsearch :https://blog.csdn.net/qq_36918149/article/details/104221934

二、配置

1）在Logstash目錄下建立配置檔案目錄

2）将MovieLens下載下傳的測試資料拷貝到movielens_dome_data目錄

3）建立配置檔案logstash.conf

配置檔案内容

input {
  file {
    # 這是在加載demo資料，請改為你本地資料路徑,注意路徑用的是‘/’而非‘\’,否則會導緻無法讀入資料
    path => "D:/step/logstash-7.1.0/movielens_dome_data/movies.csv"
    start_position => "beginning"
    sincedb_path => "nul"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     # 連接配接elasticserach 并導入資料
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}
           

3）啟動 logstash 并開始導入資料

在Logstash安裝包下執行指令

bin\logstash -f D:\step\logstash-7.1.0\movielens_dome_data\logstash.conf
           

如圖已經啟動成功并開始導入資料

4）kibana 驗證資料是否已經導入

http://localhost:5601/app/kibana#/management/elasticsearch/index_management/indices?_g=()

備注：如上圖，我們看到Elasticsearch的索引已經新增