用c++ 連接配接kafka我所踩過的坑(Connection refused || desired partition does not exist)

今天我在cetos上安裝好了開啟了kafka,網上找了幾個例子,想用c++寫一個生産者和消費者模型的例子.

然後踩了幾個坑,公布出來,希望大家以後不要再踩

我是用它自帶的sh工作作為生産者,配置好了主題和partition等資訊,如下所示

[[email protected] bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test11959

Created topic "test11959".

[[email protected] bin]# ./kafka-console-producer.sh --broker-list ${IP ADDRESS}:9092 --topic test11959

> 此時等待你的輸入

然後我啟動了我的消費者程式(用的是librdkafka這個庫),但是出現了兩個問題

首先出現的現象是這樣的:

(這一條是網上的:Connecting to IPv6 addresses: 

Connect to ipv6#[::1]:9092 failed: Connection refused)

%3|1567166247.579|FAIL|rdkafka#consumer-1| [thrd:${IP address}:9092/bootstrap]: ${IP address}:9092/0: Connect to ipv4#${IP address}:9092 failed: Connection refused (after 1ms in state CONNECT)

%3|1567166247.579|ERROR|rdkafka#consumer-1| [thrd:${IP address}:9092/bootstrap]: ${IP address}:9092/0: Connect to ipv4#${IP address}:9092 failed: Connection refused (after 1ms in state CONNECT)

解決方法是 :

cd到kafka的安裝路徑,然後到config目錄下.在檔案server.properties 将下面這一項給打開,他的說明中解釋的很清楚了

# Hostname and port the broker will advertise to producers and consumers. If not set,

# it uses the value for "listeners" if configured.  Otherwise, it will use the value

# returned from java.net.InetAddress.getCanonicalHostName().

advertised.listeners=PLAINTEXT://172.16.100.121:9092

然後出現的問題是:

%3|1567165624.868|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: testTopic [1]: desired partition does not exist in cluster (Local: Unknown partition)

最終我找到了這個連結:https://github.com/edenhill/librdkafka/issues/327 他裡面有一層樓是這樣說的

If you create a topic with 4 partitions then those partitions are numbered 0..3,

you seem to try to consume partition 4 which is indeed an Unknown partition.

看懂了嗎?假如partitions 是4 則可用的partition是從0 開始計數的:0,1,2,3 是以,我的生産者的設定的partition=1 ,我縮寫的用戶端用的也是1 就導緻用戶端找不到partition了

希望大家少走彎路!!!