kafka的安裝和基本使用

kafka概述

詳見：http://kafka.apache.org/intro

Kafka架構：

producer：生産者，就是生産饅頭（老媽）

cunsumer：消費者，吃饅頭（你）

broker：籃子

topic：主題，給饅頭帶一個标簽，topic a是給你吃的，topic b 是給你弟弟吃的

Kafka部署以及使用

本文主要講

Kafka

單節點部署和使用

參考http://kafka.apache.org/quickstart

安裝

kafka

之前要先安裝zookeeper

安裝過程：

1. 下載下傳：

官網

或 這裡

2. 解壓：

tar -zvxf kafka_2.11-0.9.0.0.tgz  -C ~/app/
           

3. 配置環境變量：

export KAFKA_HOME=/home/hadoop/app/kafka_2.11-0.9.0.0
export PATH=$KAFKA_HOME/bin:$PATH
           

4. 重新整理

source ~/.bash_profile

5. 配置

server.properties

cd /home/hadoop/app/kafka_2.11-0.9.0.0/config
vi server.properties
           

要修改的地方

#修改主機名
host.name=hadoop000
#修改kafka日志檔案目錄
log.dirs=/home/hadoop/app/tmp/kafka-logs
#修改zookeeper的連接配接資訊
zookeeper.connect=hadoop000:2181

           

6. 啟動：

zkServer.sh start        #先啟動zookeeper：

#bin/kafka-server-start.sh  #嘗試單獨啟動
#USAGE: bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*   #提示應該指定server.properties	
#啟動kafka
kafka-server-start.sh $KAFKA_HOME/config/server.properties
           

檢視程序

jps

[[email protected] ~]$ jps
8917 ZooKeeperMain
9158 Jps
9063 Kafka    #kafka程序
9004 QuorumPeerMain
8525 ZooKeeperMain
[[email protected] ~]$ 
           

jps -m

[[email protected] ~]$ jps -m
8917 ZooKeeperMain
9206 Jps -m
9063 Kafka /home/hadoop/app/kafka_2.11-0.9.0.0/config/server.properties    #加載的配置
9004 QuorumPeerMain /home/hadoop/app/zookeeper-3.4.5-cdh5.7.0/bin/../conf/zoo.cfg
8525 ZooKeeperMain
           

7. kafka單節點練習

7.1 建立topic（指定zookeeper）

kafka-topics.sh --create --zookeeper hadoop000:2181 --replication-factor 1 --partitions 1 --topic topic_hello
           

7.2 檢視所有topic

[[email protected] ~]$ kafka-topics.sh --list --zookeeper hadoop000:2181
hello_topic
kafka_streaming_topic
my-replicated-topic
streamingtopic
topic_hello
           

7.3 生産消息(和broker-list打交道：9092端口，詳情可以看config/server.properties)

[[email protected] ~]$ kafka-console-producer.sh --broker-list hadoop000:9092 --topic topic_hello
hello hadoop
hello 
spark
           

7.4 消費消息：（zookeeper）

[[email protected] ~]$ kafka-console-consumer.sh --zookeeper hadoop000:2181 --topic topic_hello  --from-beginning
hello hadoop
hello 
spark

           

--from-beginning

代表從頭開始消費

單節點操作的其他指令

[[email protected] bin]$ kafka-topics.sh --describe --zookeeper hadoop000:2181
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:kafka_streaming_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: kafka_streaming_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 1	Replicas: 3,1,2	Isr: 1
Topic:streamingtopic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: streamingtopic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic:topic_hello	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: topic_hello	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

           

注:

Partition: 0

代表分區從0開始

看單獨的一個

topic

[[email protected] bin]$ kafka-topics.sh --describe --zookeeper hadoop000:2181 --topic hello_topic
Topic:hello_topic	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: hello_topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0