資料采集學習劄記

• • 開發環境及工具
  • APP代碼
  • • pom.xml
    • app.java
  • 運作結果截圖
  • • APP
    • Logstash
    • Kibana
    • • discover
      • dashboard

開發環境及工具

基于centos，使用ELK工具對資料進行采集并進行相關分析。

在使用模拟app進行資料生成後，使用Filebeat工具收集資料，并将資料發送給Logstash，Logstash在将資料轉存到Elasticsearch後，通過Kibana讀取資料，并最終将結果輸出到Dashboard進行可視化展示。

APP代碼

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
 <modelVersion>4.0.0</modelVersion>

 <groupId>com.qust</groupId>
 <artifactId>app</artifactId>
 <version>0.0.1-SNAPSHOT</version>
 <packaging>jar</packaging>
 <name>app</name>
 <url>http://maven.apache.org</url>
 <properties>
 <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
 </properties>
 <dependencies>
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12 -->
 <dependency>
 <groupId>org.slf4j</groupId>
 <artifactId>slf4j-log4j12</artifactId>
 <version>1.7.30</version>
 <scope>test</scope>
 </dependency>
 <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->
 <dependency>
 <groupId>org.slf4j</groupId>
 <artifactId>slf4j-api</artifactId>
 <version>1.7.30</version>
 </dependency>
 <dependency>
 <groupId>org.slf4j</groupId>
 <artifactId>slf4j-simple</artifactId>
 <version>1.7.25</version>
 <scope>compile</scope>
 </dependency>
 <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
 <dependency>
 <groupId>org.apache.commons</groupId>
 <artifactId>commons-lang3</artifactId>
 <version>3.6</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/joda-time/joda-time -->
 <dependency>
 <groupId>joda-time</groupId>
 <artifactId>joda-time</artifactId>
 <version>2.10.6</version>
 </dependency>
 <!-- https://mvnrepository.com/artifact/log4j/log4j -->
 <dependency>
 <groupId>log4j</groupId>
 <artifactId>log4j</artifactId>
 <version>1.2.17</version>
 </dependency>
 </dependencies>
<build>
 <plugins>
 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-shade-plugin</artifactId>
 <version>2.3</version>
 <executions>
 <execution>
 <phase>package</phase>
 <goals>
 <goal>shade</goal>
 </goals>
 <configuration>
 <transformers>
 <transformer

implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
 <!--指定main方法 -->
 <mainClass>com.qust.app.App</mainClass>
 </transformer>
 </transformers>
 </configuration>
 </execution>
 </executions>
 </plugin>
 </plugins>
 </build>
 </project>
           

app.java

package com.qust.app;

import org.apache.commons.lang3.RandomUtils;
import org.apache.log4j.chainsaw.Main;
import org.joda.time.DateTime;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Hello world!
 *
 */
public class App {

	private static final Logger LOGGER = LoggerFactory.getLogger(Main.class);

	public static final String[] VISIT = new String[] { "登入官網", "新增賬號", "開始比對", "進入遊戲", "登入用戶端", "選擇英雄", "選擇召喚師技能", "英雄BP",
			"交換英雄" };

	public static void main(String[] args) throws Exception {

		while (true) {

			Long sleep = RandomUtils.nextLong(200, 1000 * 5);
			Thread.sleep(sleep);
			Long maxUserId = 9999L;
			Long userId = RandomUtils.nextLong(1, maxUserId);
			String visit = VISIT[RandomUtils.nextInt(0, VISIT.length)];
			DateTime now = new DateTime();
			int maxHour = now.getHourOfDay();
			int maxMillis = now.getMinuteOfHour();
			int maxSeconds = now.getSecondOfMinute();
			String date = now.plusHours(-(RandomUtils.nextInt(0, maxHour)))
					.plusMinutes(-(RandomUtils.nextInt(0, maxMillis)))
					.plusSeconds(-(RandomUtils.nextInt(0, maxSeconds))).toString("yyyy-MM-dd HH:mm:ss");

			String result = "DAU|" + userId + "|" + visit + "|" + date;
			
			System.out.println(result);
			LOGGER.info(result);
		}
	}
}


           

運作結果截圖

APP

通過運作模拟APP獲得日志資訊，用于之後的分析。

Logstash

啟動Logstash，将日志資訊收集到ElasticSearch。

Kibana

discover

将ElasticSearch存儲的日志輸出到Kibana，進行可視化處理。

dashboard

建立一個dashboard。

在之前建立的dashboard上建立餅圖、表格與柱狀圖。

建立的餅圖。

最終結果