目錄
1、建立、修改以及删除索引
(1)手動建立索引文法
(2)修改索引
(3)删除索引
2、 修改分詞器以及定義自己的分詞器
(1)預設分詞器 standard
(2)修改分詞器的設定
(3)定制化自己的分詞器
3、深入探秘type底層資料結構
(1)理論知識
(2)案例實踐
(3)最後總結
4、mapping root object深入剖析
(1)root object:某個type對應的mapping json
(2)properties:包含type、index、analyzer
(3)_source
(4)_all
(5)辨別性metadata:_index、_type、_id
5、定制化自己的dynamic mapping政策
(1)定制dynamic政策
(2)定制dynamic mapping政策
a、date_detection
b、定制自己的動态映射模闆dynamic mapping template(type level)
c、定制自己的default mapping template(index level)
6、基于scroll滾動搜尋和alias别名實作zero downtime(零停機) reindex
(1)重建索引
1、建立、修改以及删除索引
(1)手動建立索引文法
PUT /my_index
{
"settings": {... any settings...},
"mappings": {
"type_one":{... any mappings...},
"type_two":{... any mappings...},
...
}
}
PUT /my_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
"my_type":{
"properties": {
"my_field":{
"type":"text"
}
}
}
}
}
(2)修改索引
//---------修改replica 數量
PUT /my_index/_settings
{
"number_of_replicas": 1
}
(3)删除索引
- DELETE /my_index
- DELETE /index_one,index_two
- DELETE /index_*
- DELETE /_all -----删除所有索引
- elasticsearch.yml 中配置action.destructive_requires_name:true,就不能用 DELETE /_all 這種方式删除全部索引
2、 修改分詞器以及定義自己的分詞器
(1)預設分詞器 standard
- standard tokenizer:以單詞邊界進行切分
- standard token filter:什麼都不做
- lowercase token filter:将所有字母轉換為小寫
- stop token filter(預設被禁用):移除停用詞,比如a the it 等等
(2)修改分詞器的設定
啟用english 停用詞 token filter
PUT /my_index
{
"settings": {
"analysis": {
"analyzer": {
"es_std":{
"type":"standard",
"stopwords":"_englist_"
}
}
}
}
}
修改後檢視分詞效果
//-----------用standard ,被分詞成 a , dog , is , in , the , house
GET /my_index/_analyze
{
"analyzer":"standard",
"text":"a dog is in the house"
}
//----------用es_std, 被分詞成 dog , house
GET /my_index/_analyze
{
"analyzer": "es_std",
"text": "a dog is in the house"
}
(3)定制化自己的分詞器
PUT /my_index
{
"settings": {
"analysis": {
"char_filter": {
"&_to_and":{
"type":"mapping",
"mappings":["&=>and"] //&轉化成and
}
},
"filter": {
"my_stopwords":{
"type":"stop",
"stopwords":["the","a"] //忽略the a
}
},
"analyzer": {
"my_analyzer":{
"type":"custom",
"char_filter":["html_strip","&_to_and"], //忽略html标簽,&轉成and
"tokenizer":"standard",
"filter":["lowercase","my_stopwords"]
}
}
}
}
}
//--------------測試,分詞結果:tomandjerry,are, friend,in, house,haha
GET /my_index/_analyze
{
"analyzer": "my_analyzer",
"text":"tom&jerry are a friend in the house,<a>,HAHA!!"
}
在type中應用自己的分詞器
PUT /my_index/_mapping/my_type
{
"properties": {
"content":{
"type": "text",
"analyzer": "my_analyzer"
}
}
}
3、深入探秘type底層資料結構
(1)理論知識
- type,是一個index中用來區分類似的資料的,類似的資料,但是可能有不同的fields,而且有不同的屬性來控制索引建立、分詞器
- field的value,在底層的Lucene中建立索引的時候,全部是opaque(不透明)bytes類型,即:不區分類型的。
- Lucene是沒有type的概念的,在document中,實際上将type作為一個document的field來存儲,即_type,es通過_type來進行type的過濾和篩選。
- 一個index中的多個type,實際上是放在一起存儲的,是以一個index下,不能有多個type重名,二類型或者其他設定不同的,因為那樣是無法處理的。
(2)案例實踐
(1)插入兩條資料
PUT goods_index/electronic_goods/1
{
"name": "geli kongtiao",
"price": 1999.0,
"service_period": "one year"
}
PUT goods_index/eat_goods/2
{
"name": "aozhou dalongxia",
"price": 199.0,
"eat_period": "one week"
}
索引名稱為goods_index,在該索引下面分别有兩個type:electronic_goods和eat_goods
我們來看下索引對應的映射
(2)檢視mapping
GET /goods_index/_mapping
{
"goods_index": {
"mappings": {
"electronic_goods": {
"properties": {
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"price": {
"type": "float"
},
"service_period": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"eat_goods": {
"properties": {
"eat_period": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"price": {
"type": "float"
}
}
}
}
}
}
一個index中的多個type,實際上是放在一起存儲的,在Lucene底層的存儲結構如下:
(3)lucene 底層的存儲
{
"ecommerce": {
"mappings": {
"_type": {
"type": "string",
"index": "not_analyzed"
},
"name": {
"type": "string"
}
"price": {
"type": "double"
}
"service_period": {
"type": "string"
}
"eat_period": {
"type": "string"
}
}
}
}
上述兩條資料在底層存儲結構如下:
{
"_type": "electronic_goods",
"name": "geli kongtiao",
"price": 1999.0,
"service_period": "one year",
"eat_period": ""
}
{
"_type": "eat_goods",
"name": "aozhou dalongxia",
"price": 199.0,
"service_period": "",
"eat_period": "one week"
}
_type字段就是type的名稱,兩個type中都有name字段,這裡兩個type中同有name字段,意味type同享一個存儲空間,如果electronic_goods中的name為data類型,eat_goods中name為text類型。如果二者的類型不一樣,這裡就會存在問題,Lucene底層的資料結構會将“electronic_goods”和“eat_goods”的字段取并集存儲起來,
(3)最後總結
将類似結構的type放在一個index下,這些type應該有多個field是相同的。假如說,你将兩個type的field完全不同,放在一個index下,那麼就每條資料都至少有一半的field在底層的Lucene中是空值,會有嚴重的性能問題。
4、mapping root object深入剖析
(1)root object:某個type對應的mapping json
就是某個type對應的mapping json,包括了properties,metadata(_id,_source,_type),settings(analyzer),其他settings(比如include_in_all)
(2)properties:包含type、index、analyzer
PUT my_index/_mapping/my_type
{
"properties":{
"title":{
"type":"string", //類型
"index":"analyzed", //要不要分詞
"analyzer":"standard" //使用哪個分詞器
}
}
}
(3)_source
好處:
- 查詢的時候,直接可以拿到完整的document,不需要先拿document id,再發送一次請求拿document
- partial update基于_source實作
- reindex時,直接基于_source實作,不需要從資料庫(或者其他外部存儲)查詢資料再修改
- 可以基于_source定制傳回field
- debug query更容易,因為可以直接看到_source
如果不需要上述好處,可以禁用_source:
禁用_source:
PUT /my_index/_mapping/my_type2
{
"_source":{
"enabled":false
}
}
(4)_all
将所有field打包在一起,作為一個_all field,建立索引。沒指定任何field進行搜尋時,就是使用_all field在搜尋。
禁用 _all:
PUT /my_index/_mapping/my_type3
{
"_all":{
"enabled":false
}
}
也可以在field級别設定include_in_all field,設定是否要将field的值包含在_all field中
PUT /my_index/_mapping/my_type4
{
"properties":{
"my_field":{
"type":"text",
"include_in_all":false
}
}
}
(5)辨別性metadata:_index、_type、_id
5、定制化自己的dynamic mapping政策
(1)定制dynamic政策
- true:遇到陌生字段,就進行dynamic mapping
- false:遇到陌生字段,就忽略
- strict:遇到陌生字段,就報錯
//----------定制dynamic政策
PUT /my_index
{
"mappings": {
"my_type": {
"dynamic":"strict",
"properties": {
"title":{
"type": "text"
},
"address":{
"type": "object",
"dynamic":"true"
}
}
}
}
}
//---------1、測試頂層strict(遇到陌生字段報錯),添加陌生字段content
PUT my_index/my_type/1
{
"title":"my article",
"content":"this is my article",
"address":{
"province":"guangdong",
"city":"guangzhou"
}
}
錯誤資訊如下:
{
"error": {
"root_cause": [
{
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [content] within [my_type] is not allowed"
}
],
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [content] within [my_type] is not allowed"
},
"status": 400
}
//----------2、測試address(dynamic:true -- 遇到陌生字段進行dynamic mapping)中添加陌生字段
PUT my_index/my_type/1
{
"title":"my article",
"address":{
"province":"guangdong",
"city":"guangzhou"
}
}
結果如下:
{
"_index": "my_index",
"_type": "my_type",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}
//-----------檢視
GET /my_index/_mapping/my_type
結果如下:
{
"my_index": {
"mappings": {
"my_type": {
"dynamic": "strict",
"properties": {
"address": {
"dynamic": "true",
"properties": {
"city": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"province": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"title": {
"type": "text"
}
}
}
}
}
}
(2)定制dynamic mapping政策
a、date_detection
預設會按照一定格式識别date,比如yyyy-MM-dd,但是如果某個field先過來一個2017-01-01的值,就會被自動dynamic mapping成date,後面如果再來一個hello world之類的值,就會報錯,可以手動關閉某個type的date_detection,如果有需要,自己手動指定某個field為date類型
PUT /my_index/_mapping/my_type
{
"date_detection":false
}
b、定制自己的動态映射模闆dynamic mapping template(type level)
//---------------1、定制模闆:以“_en”結尾的字段,比對如下mapping
PUT my_index
{
"mappings": {
"my_type": {
"dynamic_templates":[
{"en":{
"match":"*_en",
"match_mapping_type":"string",
"mapping":{
"type":"string",
"analyzer":"english"
}
}}
]
}
}
}
//-----------------2、插入測試資料
PUT my_index/my_type/1
{
"title":"this is my first article"
}
PUT my_index/my_type/2
{
"title_en":"this is my first article"
}
//------------------3、查詢
GET my_index/my_type/_search
{
"query": {
"match": {
"title": "is"
}
}
}
結果:
{
"took": 145,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.2824934,
"hits": [
{
"_index": "my_index",
"_type": "my_type",
"_id": "1",
"_score": 0.2824934,
"_source": {
"title": "this is my first article"
}
}
]
}
}
GET my_index/my_type/_search
{
"query": {
"match": {
"title_en": "is"
}
}
}
結果:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
title沒有比對到任何的dynamic模闆,預設就是standard分詞器,不會過濾停用詞,is會進入反向索引,用is來搜尋是可以搜尋到的;title_en比對到了dynamic模闆,就是English分詞器,會過濾停用詞,is這種停用詞就會被過濾掉,用is來搜尋就搜尋不到了
c、定制自己的default mapping template(index level)
PUT /my_index
{
"mappings":{
"_default":{
"_all":{"enabled":false}
},
"blog":{
"_all":{"enabled": false}
}
}
}
6、基于scroll滾動搜尋和alias别名實作zero downtime reindex
(1)重建索引
一個field的設定是不能被修改的,如果要修改一個field,那麼應該重新按照新的mapping,建立一個index,然後将資料批量查詢出來,重新用bulk api寫入index中,批量查詢的時候,建議采用scroll api,并且采用多線程并發的方式來reindex資料,每次scroll就查詢指定日期的一段資料,交給一個線程即可。
(1)一開始,依靠dynamic mapping,插入資料,但是不小心有些資料是2017-01-01這種日期格式的,是以title這種field被自動映射為了date類型,實際上他應該是string類型
PUT my_index/my_type/1
{
"title":"2017-01-01"
}
PUT my_index/my_type/2
{
"title":"2017-01-02"
}
PUT my_index/my_type/3
{
"title":"2017-01-03"
}
GET my_index/_mapping/my_type
結果:
{
"my_index": {
"mappings": {
"my_type": {
"properties": {
"title": {
"type": "date"
}
}
}
}
}
}
(2)當後期向索引中加入string類型的title值時,就會報錯
PUT my_index/my_type/4
{
"title":"my first article"
}
結果:
{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failed to parse [title]"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse [title]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Invalid format: \"my first article\""
}
},
"status": 400
}
(3)如果此時想修改title的類型,是不可能的
PUT my_index/_mapping/my_type
{
"properties": {
"title":{
"type": "text"
}
}
}
結果:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "mapper [title] of different type, current_type [date], merged_type [text]"
}
],
"type": "illegal_argument_exception",
"reason": "mapper [title] of different type, current_type [date], merged_type [text]"
},
"status": 400
}
(4)此時,唯一的辦法就是進行reindex,也就是說,重建立立一個索引,将舊索引的資料查詢出來,在導入新索引
(5)如果說舊索引的名字,是old_index,新索引的名字是new_index,終端java應用,已經在使用old_index在操作了,難道還要去停止java應用,修改使用的index為new_index,才重新啟動java應用嗎?這個過程中,就會導緻java應用停機,可用性降低
(6)是以說,給java應用一個别名,這個别名是指向舊索引的,java應用先用着,java應用先用goods_index alias來操作,此時實際指向的是舊的my_index
PUT my_index/_alias/goods_index
(7)建立一個index,調整其title的類型為string
PUT my_index_new
{
"mappings": {
"my_type":{
"properties": {
"title":{
"type": "text"
}
}
}
}
}
(8)使用scroll api将資料批量查詢出來
GET my_index/_search?scroll=1m
{
"query": {
"match_all": {}
},
"sort": ["_doc"],
"size": 1
}
結果:
{
"_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAABvLFjU4LWIxRl9CU1BLR1RHMHd6SWpuX3cAAAAAAAAbyRY1OC1iMUZfQlNQS0dURzB3eklqbl93AAAAAAAAG8oWNTgtYjFGX0JTUEtHVEcwd3pJam5fdwAAAAAAABvMFjU4LWIxRl9CU1BLR1RHMHd6SWpuX3cAAAAAAAAbzRY1OC1iMUZfQlNQS0dURzB3eklqbl93",
"took": 113,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 3,
"max_score": null,
"hits": [
{
"_index": "my_index",
"_type": "my_type",
"_id": "2",
"_score": null,
"_source": {
"title": "2017-01-02"
},
"sort": [
0
]
}
]
}
}
(9)采用bulk api将scroll查出來的一批資料,批量寫入新索引
POST /_bulk
{"index":{"_index":"my_index_new", "_type":"my_type","_id":"2"}}
{"title":"2017-01-02"}
(10)反複循環(8)-(9),查詢一批又一批的資料出來,采取bulk api将每一批資料批量寫入新索引
(11)将goods_index alias切換到my_index_new上去,java應用會直接通過index别名使用新的索引中的資料,java應用程式不需要停機,零停機,高可用
POST /_aliases
{
"actions": [
{"remove": {"index": "my_index","alias": "goods_index"}},
{"add": {"index": "my_index_new","alias": "goods_index"}}
]
}
(12)直接通過goods_index别名來查詢
GET /goods_index/my_type/_search