kafka 的 auto.offset.reset 含義詳解

最近也是有人問我kafka的auto.offset.reset設定為earliest後怎麼結果和自己想象的不一樣呢,相信很多人都對這個參數心存疑惑,今天來詳細講解一下:

kafka-0.10.1.X版本之前: auto.offset.reset 的值為smallest,和,largest.(offest儲存在zk中)

kafka-0.10.1.X版本之後: auto.offset.reset 的值更改為:earliest,latest,和none (offest儲存在kafka的一個特殊的topic名為:__consumer_offsets裡面)

顧名思義,earliest就是從最開始消費資料,latest即為從最新的資料開始消費,但我們在使用的時候發現并不是這樣的.下面就來詳細測試一下.

先看一下官網對auto.offset.reset的解釋吧(測試版本為0.10.2.1)

這個解釋也比較的抽象,不太好了解,是以我們還是自己動手去驗證一下吧.

建立一個topic,名為jason_1122,先檢視一下這個topic的資訊:

Topic:jason_1122	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: jason_1122	Partition: 0	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: jason_1122	Partition: 1	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: jason_1122	Partition: 2	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: jason_1122	Partition: 3	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: jason_1122	Partition: 4	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: jason_1122	Partition: 5	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: jason_1122	Partition: 6	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: jason_1122	Partition: 7	Leader: 1	Replicas: 1,3,2	Isr: 1,3,2
	Topic: jason_1122	Partition: 8	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: jason_1122	Partition: 9	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
      

可以看到jason_1122這個topic有10分分區,3分副本.

測試的過程:

1.先寫入10條資料

2.設定offest為false,分别測試earliest,latest,和none,這三種情況

3,設定offest為true,分别測試earliest,latest,和none,這三種情況

3.在寫入10條資料

4.設定offest為false,分别測試earliest,latest,和none,這三種情況

5.設定offest為true,分别測試earliest,latest,和none,這三種情況

測試一:

先寫入10條資料, 設定自動送出offest為false,然後分别測試auto.offset.reset設定為earliest,latest,和none的情況.

測試結果:

(1).當auto.offset.reset為earliest時,消費到了10條資料如下:

0--null--hello jason what are you doing--1542886948685--6
0--null--hello jason what are you doing--1542886948685--3
0--null--hello jason what are you doing--1542886948686--0
0--null--hello jason what are you doing--1542886948658--8
0--null--hello jason what are you doing--1542886948684--7
0--null--hello jason what are you doing--1542886948683--5
0--null--hello jason what are you doing--1542886948684--4
0--null--hello jason what are you doing--1542886948683--2
0--null--hello jason what are you doing--1542886948685--1      

(2).當auto.offset.reset為latest時,沒有消費到資料.

(3).當auto.offset.reset為none時,抛出異常如下:

Exception in thread "main" org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partition: jason_1122-4
	at org.apache.kafka.clients.consumer.internals.Fetcher.resetOffset(Fetcher.java:369)
	at org.apache.kafka.clients.consumer.internals.Fetcher.updateFetchPositions(Fetcher.java:247)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1602)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1035)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
	at kafka.KafkaConsumer$.main(KafkaConsumer.scala:19)
	at kafka.KafkaConsumer.main(KafkaConsumer.scala)
      

測試二:

設定自動送出offest為true,然後分别測試auto.offset.reset設定為earliest,latest,和none的情況.

(1)當auto.offset.reset為earliest時,消費到了10條資料如下:

0--null--hello jason what are you doing--1542886948658--8
0--null--hello jason what are you doing--1542886948684--7
0--null--hello jason what are you doing--1542886948685--6
0--null--hello jason what are you doing--1542886948683--5
0--null--hello jason what are you doing--1542886948684--4
0--null--hello jason what are you doing--1542886948685--3
0--null--hello jason what are you doing--1542886948683--2
0--null--hello jason what are you doing--1542886948685--1
0--null--hello jason what are you doing--1542886948686--0      

(2)當auto.offset.reset為latest時,沒有消費到資料.

(3)當auto.offset.reset為none時,沒有消費到資料,也沒有抛出異常.

測試三:

在寫入10條資料,一共為20條資料,設定自動送出offest為false,然後分别測試auto.offset.reset設定為earliest,latest,和none的情況.

1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0      

(2)當auto.offset.reset為latest時,同樣消費到了10條資料如下:

0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1      

(3)當auto.offset.reset為none時,同樣消費到了10條資料如下:

0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
      

測試四:

0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1      

(2)當auto.offset.reset為latest時,沒有消費到資料

(3)當auto.offset.reset為none時,也沒有消費到資料

根據上面的測試,得到最終的結論:

如果存在已經送出的offest時,不管設定為earliest 或者latest 都會從已經送出的offest處開始消費