05 Logstash安裝與導入資料

Logstash安裝與導入資料

下載下傳最 MovieLens 最小測試資料集：https://grouplens.org/datasets/movielens/

Logstash 下載下傳：https://www.elastic.co/cn/downloads/logstash

Logstash 參考文檔：https://www.elastic.co/guide/en/logstash/current/index.html

Logstash 的作用就是一個資料收集器，将各種格式各種管道的資料通過它收集解析之後格式化輸出到 Elasticsearch

一.Logstash的安裝

1.安裝流程

• 到官網下載下傳壓縮包并解壓
• 準備logstash.conf配置檔案，執行

  bin/logstash -f logstash.conf

  啟動

2.使用執行個體

• 在grouplens下載下傳資料集
• 執行個體logstash.conf檔案内容

  input {
    file {
      path => "實際的movies.csv路徑:movies.csv"
      start_position => "beginning"
      sincedb_path => "/dev/null"
    }
  }
  
  # 對CSV進行處理部分
  filter {
    csv {
      separator => ","
      columns => ["id","content","genre"]
    }
  
    mutate {
      split => { "genre" => "|" }
      remove_field => ["path", "host","@timestamp","message"]
    }
  
    mutate {
  
      split => ["content", "("]
      add_field => { "title" => "%{[content][0]}"}
      add_field => { "year" => "%{[content][1]}"}
    }
  
    mutate {
      convert => {
        "year" => "integer"
      }
      strip => ["title"]
      remove_field => ["path", "host","@timestamp","message","content"]
    }
  }
  output {
     elasticsearch {
     	 # 将結果輸出到ES中
       hosts => "http://localhost:9200"
       index => "movies"
       document_id => "%{id}"
     }
    stdout {}
  }
             

• 執行

  sudo ./logstash -f logstash.conf

  啟動Logstash
• 啟動後可以在控制台中看到大量資料在寫入ES中