天天看點
傳回

Kafka：搭建Kafka叢集

部落客在之前已經介紹過如何部署​

​Kafka​

​​，​

​Kafka​

​​的部署模式隻有叢集模式，​

​Kafka​

​的架構本就是天然的叢集架構，是以單節點的部署和多節點的部署是類似的。

叢集節點：

節點 位址
ZooKeeper 192.168.1.5
Kafka0 192.168.1.4
Kafka1 192.168.1.6
Kafka2 192.168.1.199

為了簡單起見，這裡不使用​

​ZooKeeper​

​​叢集，單機版的​

​ZooKeeper​

​即可滿足要求。

  • ​​ZooKeeper ：Shell腳本搭建單機版ZooKeeper​​
  • ​​ZooKeeper ：搭建ZooKeeper叢集​​

這​

​4​

​​個節點都需要​

​Java​

​​環境，以及​

​Kafka​

​​叢集的​

​3​

​​個節點需要下載下傳和上傳​

​Kafka​

​相關檔案，可參考下面這篇部落格：

  • ​​Kafka：部署Kafka​​

ZooKeeper

​

​ZooKeeper​

​的配置檔案：

Kafka：搭建Kafka叢集

啟動​

​ZooKeeper​

​：

[root@192 ~]# cd /usr/local/apache-zookeeper-3.6.3-bin/bin/
[root@192 bin]# ./zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@192 bin]# ./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Client port found: 9000. Client address: localhost. Client SSL: false.
Mode: standalone

​

​ZooKeeper​

​啟動成功。

Kafka叢集 

[root@192 ~]# cd /usr/local/kafka_2.13-3.0.0
[root@192 kafka_2.13-3.0.0]# vim config/server.properties
  • Kafka0
  • Kafka1
  • Kafka2

啟動每個​

​Kafka​

​節點：

[root@192 kafka_2.13-3.0.0]# bin/kafka-server-start.sh config/server.properties &
Kafka：搭建Kafka叢集

在​

​Kafka​

​​節點上除了修改​

​broker.id​

​​用于差別叢集節點，就沒有修改其他有關叢集的配置資訊了，比如沒有添加​

​Kafka​

​​叢集所有節點的位址資訊，而在搭建​

​ZooKeeper​

​​叢集時，是需要将​

​ZooKeeper​

​​叢集所有節點的位址資訊在每個​

​ZooKeeper​

​​節點上進行配置。那麼​

​Kafka​

​​節點是如何去發現叢集中的其他節點？這其實就是靠​

​ZooKeeper​

​​來維護​

​Kafka​

​的叢集資訊。

查詢​

​ZooKeeper​

​中的資料：

[root@192 bin]# ./zkCli.sh -timeout 40000 -server 127.0.0.1:9000
...
[zk: 127.0.0.1:9000(CONNECTED) 4] ls -R /brokers 
/brokers
/brokers/ids
/brokers/seqid
/brokers/topics
/brokers/ids/0
/brokers/ids/1
/brokers/ids/2
[zk: 127.0.0.1:9000(CONNECTED) 5] get /brokers/ids/0
{"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.4:9092"],"jmx_port":-1,"port":9092,"host":"192.168.1.4","version":5,"timestamp":"1642677446098"}
[zk: 127.0.0.1:9000(CONNECTED) 6] get /brokers/ids/1
{"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.6:9092"],"jmx_port":-1,"port":9092,"host":"192.168.1.6","version":5,"timestamp":"1642677466209"}
[zk: 127.0.0.1:9000(CONNECTED) 7] get /brokers/ids/2
{"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.1.199:9092"],"jmx_port":-1,"port":9092,"host":"192.168.1.199","version":5,"timestamp":"1642677476386"}

很顯然​

​ZooKeeper​

​​維護了​

​Kafka​

​​的叢集資訊，将​

​Kafka1​

​節點上的服務停止。

Kafka：搭建Kafka叢集

​

​ZooKeeper​

​​也能動态更新​

​Kafka​

​​的叢集資訊，這其實是​

​ZooKeeper​

​臨時節點的相關特性。

[zk: 127.0.0.1:9000(CONNECTED) 8] ls -R /brokers 
/brokers
/brokers/ids
/brokers/seqid
/brokers/topics
/brokers/ids/0
/brokers/ids/2
  • ​​ZooKeeper ：重要概念 & 用戶端指令介紹​​

在​

​Kafka0​

​​節點上建立​

​topic​

​：

[root@192 kafka_2.13-3.0.0]# bin/kafka-topics.sh --create --bootstrap-server 192.168.1.4:9092 --replication-factor 2 --partitions 2 --topic kaven-topic
[2022-01-20 19:43:51,408] INFO Creating topic kaven-topic with configuration {} and initial partition assignment HashMap(0 -> ArrayBuffer(0, 2), 1 -> ArrayBuffer(2, 0)) (kafka.zk.AdminZkClient)
[2022-01-20 19:43:51,530] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(kaven-topic-0) (kafka.server.ReplicaFetcherManager)
[2022-01-20 19:43:51,601] INFO [LogLoader partition=kaven-topic-0, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log$)
[2022-01-20 19:43:51,611] INFO Created log for partition kaven-topic-0 in /tmp/kafka-logs/kaven-topic-0 with properties {} (kafka.log.LogManager)
[2022-01-20 19:43:51,612] INFO [Partition kaven-topic-0 broker=0] No checkpointed highwatermark is found for partition kaven-topic-0 (kafka.cluster.Partition)
[2022-01-20 19:43:51,613] INFO [Partition kaven-topic-0 broker=0] Log loaded for partition kaven-topic-0 with initial high watermark 0 (kafka.cluster.Partition)
[2022-01-20 19:43:51,703] INFO [LogLoader partition=kaven-topic-1, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log$)
[2022-01-20 19:43:51,704] INFO Created log for partition kaven-topic-1 in /tmp/kafka-logs/kaven-topic-1 with properties {} (kafka.log.LogManager)
[2022-01-20 19:43:51,704] INFO [Partition kaven-topic-1 broker=0] No checkpointed highwatermark is found for partition kaven-topic-1 (kafka.cluster.Partition)
[2022-01-20 19:43:51,704] INFO [Partition kaven-topic-1 broker=0] Log loaded for partition kaven-topic-1 with initial high watermark 0 (kafka.cluster.Partition)
[2022-01-20 19:43:51,704] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions HashSet(kaven-topic-1) (kafka.server.ReplicaFetcherManager)
[2022-01-20 19:43:51,730] INFO [ReplicaFetcherManager on broker 0] Added fetcher to broker 2 for partitions Map(kaven-topic-1 -> InitialFetchState(BrokerEndPoint(id=2, host=192.168.1.199:9092),0,0)) (kafka.server.ReplicaFetcherManager)
[2022-01-20 19:43:51,751] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Starting (kafka.server.ReplicaFetcherThread)
[2022-01-20 19:43:51,757] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Truncating partition kaven-topic-1 to local high watermark 0 (kafka.server.ReplicaFetcherThread)
[2022-01-20 19:43:51,758] INFO [Log partition=kaven-topic-1, dir=/tmp/kafka-logs] Truncating to 0 has no effect as the largest offset in the log is -1 (kafka.log.Log)
Created topic kaven-topic.

在​

​Kafka2​

​​節點上查詢已經建立的​

​topic​

​：

[root@192 kafka_2.13-3.0.0]# bin/kafka-topics.sh --list --bootstrap-server 192.168.1.199:9092
kaven-topic

很顯然​

​Kafka​

​​叢集的資料已經同步了，​

​ZooKeeper​

​中也有這些資訊：

[zk: 127.0.0.1:9000(CONNECTED) 9] ls -R /brokers 
/brokers
/brokers/ids
/brokers/seqid
/brokers/topics
/brokers/ids/0
/brokers/ids/2
/brokers/topics/kaven-topic
/brokers/topics/kaven-topic/partitions
/brokers/topics/kaven-topic/partitions/0
/brokers/topics/kaven-topic/partitions/1
/brokers/topics/kaven-topic/partitions/0/state
/brokers/topics/kaven-topic/partitions/1/state

建立的​

​topic​

​​有兩個​

​partition​

​​以及每個​

​partition​

​​有兩個​

​replication​

​​（​

​--replication-factor 2 --partitions 2​

​​），第​

​0​

​​個​

​partition​

​​的資訊如下所示，可見兩個​

​replication​

​​分布在​

​Kafka0​

​​和​

​Kafka2​

​​節點上，而其中的​

​leader​

​​在​

​Kafka0​

​節點上。

[zk: 127.0.0.1:9000(CONNECTED) 11] get /brokers/topics/kaven-topic/partitions/0/state
{"controller_epoch":3,"leader":0,"version":1,"leader_epoch":0,"isr":[0,2]}

第​

​1​

​​個​

​partition​

​​的資訊如下所示，可見兩個​

​replication​

​​分布在​

​Kafka0​

​​和​

​Kafka2​

​​節點，而其中的​

​leader​

​​在​

​Kafka1​

​節點上。

[zk: 127.0.0.1:9000(CONNECTED) 12] get /brokers/topics/kaven-topic/partitions/1/state
{"controller_epoch":3,"leader":2,"version":1,"leader_epoch":0,"isr":[2,0]}
  • ​​Kafka：Topic概念與API介紹​​
kafka zookeeper 分布式 apache
上一篇: adroid 中公裡除時間_又一豐田車被國人捧紅，一箱油可跑1000公裡，92天狂甩28299輛...
下一篇: python時間加一個月減一個月_python – 從pandas中的datetime列減去一年

繼續閱讀