1、背景

我們知道在sql中是可以實作 group by 字段a,字段b，那麼這種效果在elasticsearch中該如何實作呢？此處我們記錄在elasticsearch中的3種方式來實作這個效果。

2、實作多字段聚合的思路

實作多字段聚合的思路

圖檔來源：https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html 從上圖中，我們可以知道，可以通過3種方式來實作多字段的聚合操作。

3、需求

根據省(province)和性别(sex)來進行聚合，然後根據聚合後的每個桶的資料，在根據每個桶中的最大年齡(age)來進行倒序排序。

4、資料準備

4.1 建立索引

PUT /index_person
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "long"
      },
      "name": {
        "type": "keyword"
      },
      "province": {
        "type": "keyword"
      },
      "sex": {
        "type": "keyword"
      },
      "age": {
        "type": "integer"
      },
      "address": {
        "type": "text",
        "analyzer": "ik_max_word",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

4.2 準備資料

PUT /_bulk
{"create":{"_index":"index_person","_id":1}}
{"id":1,"name":"張三","sex":"男","age":20,"province":"湖北","address":"湖北省黃岡市羅田縣匡河鎮"}
{"create":{"_index":"index_person","_id":2}}
{"id":2,"name":"李四","sex":"男","age":19,"province":"江蘇","address":"江蘇省南京市"}
{"create":{"_index":"index_person","_id":3}}
{"id":3,"name":"王武","sex":"女","age":25,"province":"湖北","address":"湖北省武漢市江漢區"}
{"create":{"_index":"index_person","_id":4}}
{"id":4,"name":"趙六","sex":"女","age":30,"province":"北京","address":"北京市東城區"}
{"create":{"_index":"index_person","_id":5}}
{"id":5,"name":"錢七","sex":"女","age":16,"province":"北京","address":"北京市西城區"}
{"create":{"_index":"index_person","_id":6}}
{"id":6,"name":"王八","sex":"女","age":45,"province":"北京","address":"北京市朝陽區"}

5、實作方式

5.1 multi_terms實作

5.1.1 dsl

GET /index_person/_search
{
  "size": 0,
  "aggs": {
    "agg_province_sex": {
      "multi_terms": {
        "size": 10,
        "shard_size": 25,
        "order":{
          "max_age": "desc"    
        },
        "terms": [
          {
            "field": "province",
            "missing": "defaultProvince"
          },
          {
            "field": "sex"
          }
        ]
      },
      "aggs": {
        "max_age": {
          "max": {
            "field": "age"
          }
        }
      }
    }
  }
}

5.1.2 java 代碼

@Test
    @DisplayName("多term聚合-根據省和性别聚合，然後根據最大年齡倒序")
    public void agg01() throws IOException {

        SearchRequest searchRequest = new SearchRequest.Builder()
                .size(0)
                .index("index_person")
                .aggregations("agg_province_sex", agg ->
                        agg.multiTerms(multiTerms ->
                                        multiTerms.terms(term -> term.field("province"))
                                                .terms(term -> term.field("sex"))
                                                .order(new NamedValue<>("max_age", SortOrder.Desc))
                                )
                                .aggregations("max_age", ageAgg ->
                                        ageAgg.max(max -> max.field("age")))

                )
                .build();
        System.out.println(searchRequest);
        SearchResponse<Object> response = client.search(searchRequest, Object.class);
        System.out.println(response);
    }

5.1.3 運作結果

運作結果

5.2 script實作

5.2.1 dsl

GET /index_person/_search
{
  "size": 0,
  "runtime_mappings": {
    "runtime_province_sex": {
      "type": "keyword",
      "script": """
          String province = doc['province'].value;
          String sex = doc['sex'].value;
          emit(province + '|' + sex);
      """
    }
  },
  "aggs": {
    "agg_province_sex": {
      "terms": {
        "field": "runtime_province_sex",
        "size": 10,
        "shard_size": 25,
        "order": {
          "max_age": "desc"
        }
      },
      "aggs": {
        "max_age": {
          "max": {
            "field": "age"
          }
        }
      }
    }
  }
}

5.2.2 java代碼

@Test
    @DisplayName("多term聚合-根據省和性别聚合，然後根據最大年齡倒序")
    public void agg02() throws IOException {

        SearchRequest searchRequest = new SearchRequest.Builder()
                .size(0)
                .index("index_person")
                .runtimeMappings("runtime_province_sex", field -> {
                    field.type(RuntimeFieldType.Keyword);
                    field.script(script -> script.inline(new InlineScript.Builder()
                            .lang(ScriptLanguage.Painless)
                            .source("String province = doc['province'].value;\n" +
                                    "          String sex = doc['sex'].value;\n" +
                                    "          emit(province + '|' + sex);")
                            .build()));
                    return field;
                })
                .aggregations("agg_province_sex", agg ->
                        agg.terms(terms ->
                                        terms.field("runtime_province_sex")
                                                .size(10)
                                                .shardSize(25)
                                                .order(new NamedValue<>("max_age", SortOrder.Desc))
                                )
                                .aggregations("max_age", minAgg ->
                                        minAgg.max(max -> max.field("age")))
                )
                .build();
        System.out.println(searchRequest);
        SearchResponse<Object> response = client.search(searchRequest, Object.class);
        System.out.println(response);
    }

5.2.3 運作結果

運作結果

5.3 通過copyto實作

我本地測試過，通過copyto沒實作，此處故先不考慮

5.5 通過pipeline來實作

實作思路：

建立mapping時，多建立一個字段pipeline_province_sex，該字段的值由建立資料時指定pipeline來生産。

5.4.1 建立mapping

PUT /index_person
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "long"
      },
      "name": {
        "type": "keyword"
      },
      "province": {
        "type": "keyword"
      },
      "sex": {
        "type": "keyword"
      },
      "age": {
        "type": "integer"
      },
      "pipeline_province_sex":{
        "type": "keyword"
      },
      "address": {
        "type": "text",
        "analyzer": "ik_max_word",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

此處指定了一個字段pipeline_province_sex，該字段的值會由pipeline來處理。

5.4.2 建立pipeline

PUT _ingest/pipeline/pipeline_index_person_provice_sex
{
  "description": "将provice和sex的值拼接起來",
  "processors": [
    {
      "set": {
        "field": "pipeline_province_sex",
        "value": ["{{province}}", "{{sex}}"]
      }, 
      "join": {
        "field": "pipeline_province_sex",
        "separator": "|"
      }
    }
  ]
}

5.4.3 插入資料

PUT /_bulk?pipeline=pipeline_index_person_provice_sex
{"create":{"_index":"index_person","_id":1}}
{"id":1,"name":"張三","sex":"男","age":20,"province":"湖北","address":"湖北省黃岡市羅田縣匡河鎮"}
{"create":{"_index":"index_person","_id":2}}
{"id":2,"name":"李四","sex":"男","age":19,"province":"江蘇","address":"江蘇省南京市"}
{"create":{"_index":"index_person","_id":3}}
{"id":3,"name":"王武","sex":"女","age":25,"province":"湖北","address":"湖北省武漢市江漢區"}
{"create":{"_index":"index_person","_id":4}}
{"id":4,"name":"趙六","sex":"女","age":30,"province":"北京","address":"北京市東城區"}
{"create":{"_index":"index_person","_id":5}}
{"id":5,"name":"錢七","sex":"女","age":16,"province":"北京","address":"北京市西城區"}
{"create":{"_index":"index_person","_id":6}}
{"id":6,"name":"王八","sex":"女","age":45,"province":"北京","address":"北京市朝陽區"}

注意：此處的插入需要指定上一步的pipeline PUT /_bulk?pipeline=pipeline_index_person_provice_sex

5.4.4 聚合dsl

GET /index_person/_search
{
  "size": 0,
  "aggs": {
    "agg_province_sex": {
      "terms": {
        "field": "pipeline_province_sex",
        "size": 10,
        "shard_size": 25,
        "order": {
          "max_age": "desc"   
        }
      }, 
      "aggs": {
        "max_age": {
          "max": {
            "field": "age"
          }
        }
      }
    }
  }
}

5.4.5 運作結果

運作結果

6、實作代碼

https://gitee.com/huan1993/spring-cloud-parent/blob/master/es/es8-api/src/main/java/com/huan/es8/aggregations/bucket/MultiTermsAggs.java

7、參考文檔

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html

elasticsearch多字段聚合實作方式

1、背景

2、實作多字段聚合的思路

3、需求

4、資料準備

4.1 建立索引

4.2 準備資料

5、實作方式

5.1 multi_terms實作

5.1.1 dsl

5.1.2 java 代碼

5.1.3 運作結果

5.3 通過copyto實作

5.5 通過pipeline來實作

5.4.1 建立mapping

5.4.2 建立pipeline

5.4.3 插入資料

5.4.4 聚合dsl

5.4.5 運作結果

6、實作代碼

7、參考文檔

繼續閱讀

DOM4J 例子

elasticsearch 的 Percolator操作

es使用項目中遇到的問題

RHCE033 Unit9 vim: An Advanced Text Editor

高亮度顯示搜尋關鍵字

配置檔案2012.2.14

VIM 配置Python開發環境 Taglist + Pydiction+ Highlight+Python_fold+NERD_tree+ autocomplete

15.profile-api

Algorithms Review: Divide and Conquer(Binary Search & Merge Sort)

[zz]The Most Important Algorithms (in CS and Math)

【轉】ElasticSearch是什麼以及應用場景

ElasticSearch是什麼以及應用場景ES是如何産生的？ES 基礎一網打盡ES特點和優勢為什麼要用ES？ES的應用場景是怎樣的？

延雲行業搜尋資料庫在大資料生态中位置和重要性大資料的挑戰大資料技術的現狀延雲行業搜尋資料庫

查找算法學習之二分查找（Python版本）——BinarySearch

尚矽谷—韓順平—圖解 Java設計模式（結構型）（55～）

30天了解30種技術系列---(10)面向Cloud的搜尋引擎 ElasticSearch