天天看点
返回

ES数据的操作（二）kibanakibana的安装和使用：ES查询的两种方式六、ES的bool查询 (must、should)七、ES之查询结果过滤八、ES之精确查询与模糊查询九、ES的聚合查询avg、max、min、sum十、ES的分组查询

kibana

是一一个针对Elasticsearch的开源分析及可视化平台,用来搜索、查看交互存储在Elasticsearch索引中的数据。 使用Kibana ,可以通过各种图表进行高级数据分析及展示。Kibana让海量数据更容易理解。它操作简单,基于浏览器的用户界面可以快速创建仪表板( dashboard )实时显示Elasticsearch查询动态。设置Kibana非常简单。 无需编码或者额外的基础架构,几分钟内就可以完成Kibana安装并启动Elasticsearch索引监测。

kibana的安装和使用：

https://blog.csdn.net/qq_18769269/article/details/80843810

ES查询的两种方式

基础知识

命令

命令 url 解释
put /索引名称/类型名称/文档ID 创建文档（指定文档ID）
POST /索引名称/索引类型 创建文档（随机文档ID）
POST /索引名称/类型名称/文档id/_update 修改文档
POST /索引名称/类型名称/_search 查询数据
DELETE /索引名称/类型名称/文档id 删除文档/或者索引
GET /索引名称/类型名称/文档id 查询文档通过文档ID

字段类型

类型 对应类型 说明
字符串 text keyword text自动分词，keyword全文匹配
整型 byte short integer long
浮点型 float double half_float scaled_float
日期 date
布尔 boolean
二进制 binary
范围 range
数组 array
对象 object
嵌套 nested
ip ip (IPv4 和 IPv6 地址)

1、查询字符串搜索

GET alias_productbatches/_search?q=id:1e8ca3d2-5515-45b9-9a7d-4d9ac3bcc0d3


{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 10.914281,
    "hits" : [
      {
        "_index" : "productbatches_v2",
        "_type" : "_doc",
        "_id" : "1e8ca3d2-5515-45b9-9a7d-4d9ac3bcc0d3",
        "_score" : 10.914281,
        "_source" : {
          "id" : "1e8ca3d2-5515-45b9-9a7d-4d9ac3bcc0d3",
          "num" : "202000004107210719150940932",
          "product_id" : "56e9dab5-8ff8-410a-8e2d-995a98eb54b1",
          "store_id" : "1471d41e-c30c-4ffb-90ba-98febbfae394",
          "store_product_id" : "5eaf6b02-c727-4bf7-b9af-15f11421f900",
          }
      }
    ]
  }
}

2、结构化查询(单字段查询,不能多字段组合查询)

GET alias_productbatches/_search
{
  "query":{
    "match":{
      "id":"1e8ca3d2-5515-45b9-9a7d-4d9ac3bcc0d3"
    }
  }
}

3、match系列之操作 

造数：

PUT test1/doc/1
{
  "title": "中国是世界上人口最多的国家",
   "desc": "china is the most people in the world"
}
PUT test1/doc/2
{
  "title": "美国是世界上军事实力最强大的国家",
   "desc": "The United States is the most powerful military country in the world"
}
PUT test1/doc/3
{
  "title": "北京是中国的首都",
  "desc": "Beijing is the capital of China"
}
           
match: 查询匹配key的values值

GET test1/doc/_search
{
  "query":{
    "match":{
      "title":"中国"
    }
  }
}

结果：
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 0.68324494,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.68324494,
        "_source" : {
          "title" : "中国是世界上人口最多的国家"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "北京是中国的首都"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "2",
        "_score" : 0.39556286,
        "_source" : {
          "title" : "美国是世界上军事实力最强大的国家"
        }
      }
    ]
  }
}



匹配id=1e8ca3d2-5515-45b9-9a7d-4d9ac3bcc0d3

match查询中文时，会把中文拆分后进行匹配查询，如需要则使用短语查询match_phrase

GET test1/doc/_search
{
  "query":{
    "match_phrase":{
      "title":"中国"
    }
  }
}

结果：
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 0.5753642,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "中国是世界上人口最多的国家"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "北京是中国的首都"
        }
      }
    ]
  }
}

slop:相当于正则中的中国.*?世界。这个间隔默认为0，2代表中国与世界中间数字在两个及以下
GET test1/doc/_search
{
  "query":{
    "match_phrase": {
      "title": {
        "query": "中国世界",
        "slop":2
      }
    }
  }
}

结果：
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.7445889,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.7445889,
        "_source" : {
          "title" : "中国是世界上人口最多的国家"
        }
      }
    ]
  }
}


match_phrase_prefix（最左前缀查询）智能搜索--以什么开头（主要是英文）
GET test1/doc/_search
{
  "query":{
    "match_phrase_prefix": {
      "desc": "china is"
    }
  }
}

结果：
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.5753642,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "中国是世界上人口最多的国家",
          "desc" : "china is the most people in the world"
        }
      }
    ]
  }
}

max_expansions 参数理解 前缀查询会非常的影响性能，要对结果集进行限制，就加上这个参数。

GET test1/doc/_search
{
  "query": {
    "match_phrase_prefix": {
      "desc": {
        "query": "bea",
        "max_expansions":1
      }
    }
  }
}

match系列之multi_match（多字段查询) 

multi_match是要在多个字段中查询同一个关键字 除此之外，mulit_match甚至可以当做match_phrase和match_phrase_prefix使用，只需要指定type类型即可

GET test1/doc/_search
{
  "query": {
    "multi_match": {
      "query": "中国",
      "fields": ["title"]
    }
  }
}

同上第一个match查询

当设置属性 type:phrase 时 等同于 短语查询
GET test1/doc/_search
{
  "query": {
    "multi_match": {
      "query": "中国",
      "fields": ["title"],
      "type": "phrase"
    }
  }
}


当设置属性 type:phrase_prefix时 等同于 最左前缀查询
GET test1/doc/_search
{
  "query": {
    "multi_match": {
      "query": "china is",
      "fields": ["desc"],
      "type": "phrase_prefix"
    }
  }
}

四、ES的排序查询

sort：对字段进行排序

GET test1/doc/_search
GET test1/doc/_search
{
  "query": {
    "match_all": {}
  },
  "sort": [
    {
      "title.keyword": {
        "order": "asc"
      }
    }
  ]
}

为什么不能直接title呢？
查看mapping：
{
  "test1" : {
    "mappings" : {
      "doc" : {
        "properties" : {
          "desc" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "title" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    }
  }
}

对text进行排序需要对字段索引两次，一次索引分词（用于搜索）一次索引不分词（用于排序）

五、ES的分页查询

from：从哪开始查 size：返回几条结果

GET test1/doc/_search
{
  "query": {
    "match": {
      "title": "中国"
    }
  },
  "from": 0,
  "size": 2
}

结果：

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 0.68324494,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.68324494,
        "_source" : {
          "title" : "中国是世界上人口最多的国家",
          "desc" : "china is the most people in the world"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "北京是中国的首都",
          "desc" : "Beijing is the capital of China"
        }
      }
    ]
  }
}

六、ES的bool查询 (must、should)

must (must字段对应的是个列表，也就是说可以有多个并列的查询条件，一个文档满足各个子条件后才最终返回)

GET test1/doc/_search
{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "title": "中国"
                    }
                }
            ]
        }
    }
}

结果：
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 0.68324494,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.68324494,
        "_source" : {
          "title" : "中国是世界上人口最多的国家",
          "desc" : "china is the most people in the world"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "北京是中国的首都",
          "desc" : "Beijing is the capital of China"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "2",
        "_score" : 0.39556286,
        "_source" : {
          "title" : "美国是世界上军事实力最强大的国家",
          "desc" : "The United States is the most powerful military country in the world"
        }
      }
    ]
  }
}


多个条件：
GET test1/doc/_search
{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "title": "中国"
                    }
                },
               {
                    "match_phrase_prefix": {
                        "desc": "china is"
                    }
                }
            ]
        }
    }
}

结果：
{
  "took" : 13,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.258609,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 1.258609,
        "_source" : {
          "title" : "中国是世界上人口最多的国家",
          "desc" : "china is the most people in the world"
        }
      }
    ]
  }
}

should (只要符合其中一个条件就返回)

GET test1/doc/_search
{
    "query": {
        "should": {
            "must": [
                {
                    "match": {
                        "title": "中国"
                    }
                },
               {
                    "match_phrase_prefix": {
                        "desc": "mytest"
                    }
                }
            ]
        }
    }
}

结果和查询条件满足中国的结果一样

filter：满足过滤条件 

filter(条件过滤查询，过滤条件的范围用range表示gt表示大于、lt表示小于、gte表示大于等于、lte表示小于等于)

GET test1/_search
{
      "query": {
        "bool": {
          "must": [
            {
              "range": {
                  "time_create": {
                      "gte": "1626667200000",
                      "lte": "1626710400000"
                  }
              }
            }
          ]
        }
      }
    }

boost：提权，控制每个查询子句的相对权重，该值默认为1。一个大于1的boost会增加该查询子句的相对权重

可以参考：https://www.jianshu.com/p/98888942e737

minimum_should_match：最小匹配度

参考：https://blog.csdn.net/xiao_jun_0820/article/details/51095521

must_not：不等于

ES数据的操作（二）kibanakibana的安装和使用：ES查询的两种方式六、ES的bool查询 (must、should)七、ES之查询结果过滤八、ES之精确查询与模糊查询九、ES的聚合查询avg、max、min、sum十、ES的分组查询

七、ES之查询结果过滤

_source

GET test1/doc/_search
{
  "query": {
    "match": {
      "title": "中国"
    }
  },
  "_source": ["title"]
}

结果：
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 0.68324494,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 0.68324494,
        "_source" : {
          "title" : "中国是世界上人口最多的国家"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 0.5753642,
        "_source" : {
          "title" : "北京是中国的首都"
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "2",
        "_score" : 0.39556286,
        "_source" : {
          "title" : "美国是世界上军事实力最强大的国家"
        }
      }
    ]
  }
}


=======
次数最终的结果没有展示desc字段

八、ES之精确查询与模糊查询

term:查询查找包含文档精确的倒排索引指定的词条。也就是精确查找。

term和match的区别是：match是经过analyer的，也就是说，文档首先被分析器给处理了。根据不同的分析器，分析的结果也稍显不同，然后再根据分词结果进行匹配。term则不经过分词，它是直接去倒排索引中查找了精确的值了。

GET test1/doc/_search
{
  "query": {
    "term": {
      "title": "中国"
    }
  },
  "_source": ["title"]
}
===如果这样查询，就无法查出结果


GET test1/doc/_search
{
  "query": {
    "term": {
      "title.keyword": "中国是世界上人口最多的国家"
    }
  }
}

======
这样查询title为“中国是世界上人口最多的国家”的数据,
为什么要加keyword呢，因为索引的mapping title是text:
"keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }

多个term查询

查询方式一：
GET test1/doc/_search
{
  "query": {
    "bool": {
      "must": [
        {"term": {
          "title.keyword": {
            "value": "中国是世界上人口最多的国家"
          }
        }
        },
        {"term": {
          "desc.keyword": {
            "value": "china is the most people in the world"
          }
        }}
      ]
      
    }
    
  }
}


方式二：
GET test1/doc/_search
{
  "query": {
    "bool": {
      "must": [
        {"terms": {
          "title.keyword": [
            "中国是世界上人口最多的国家",
            "北京是中国的首都"
          ]
        }}
      ]
    }
  }
}

九、ES的聚合查询avg、max、min、sum

聚合函数的使用，一定是先查出结果，然后对结果使用聚合函数做处理

avg：求平均

max：最大值

min：最小值

sum：求和

GET test1/doc/_search
{
  "query": {
    "bool": {
      "must": [
      ]
    }
  },
  "aggs": {
    "my_aggs": {
      "avg": {
        "field": "score"
      }
    }
  }
}

=====
查询结果：
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "title" : "美国是世界上军事实力最强大的国家",
          "desc" : "The United States is the most powerful military country in the world",
          "score" : 2.5
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "title" : "中国是世界上人口最多的国家",
          "desc" : "china is the most populous country in the world",
          "score" : 9.5
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "title" : "北京是中国的首都",
          "desc" : "Beijing is the capital of China",
          "score" : 5.5
        }
      }
    ]
  },
  "aggregations" : {
    "my_aggs" : {
      "value" : 5.833333333333333
    }
  }
}

=========
query:先查询结果
aggs：对查询出来的结果进行聚合，my_aggs是聚合的别名 avg是聚合类型，field的值是聚合的字段
其他聚合方式一样
GET test1/doc/_search
{
  "query": {
    "bool": {
      "must": [
      ]
    }
  },
  "aggs": {
    "my_aggs": {
      "max": {
        "field": "score"
      }
    }
  }
}

十、ES的分组查询

在aggs的中，使用range来做分组，field是以age为分组，分组使用ranges来做，from和to是范围

GET test1/doc/_search
{
  "query": {
    "bool": {
      "must": [
      ]
    }
  },
  "aggs": {
    "my_aggs_group": {
      "range": {
        "field": "score",
        "ranges": [
          {"from": 1.0,
          "to": 5.0
          },
          {"from": 5.0,
            "to": 10.0
          }
        ]
      },
      "aggs": {
        "my_agg": {
          "avg": {
            "field": "score"
          }
        }
      }
    }
  }
}

==结果：
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "title" : "美国是世界上军事实力最强大的国家",
          "desc" : "The United States is the most powerful military country in the world",
          "score" : 2.5
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "title" : "中国是世界上人口最多的国家",
          "desc" : "china is the most populous country in the world",
          "score" : 9.5
        }
      },
      {
        "_index" : "test1",
        "_type" : "doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "title" : "北京是中国的首都",
          "desc" : "Beijing is the capital of China",
          "score" : 5.5
        }
      }
    ]
  },
  "aggregations" : {
    "my_aggs_group" : {
      "buckets" : [
        {
          "key" : "1.0-5.0",
          "from" : 1.0,
          "to" : 5.0,
          "doc_count" : 1,
          "my_agg" : {
            "value" : 2.5
          }
        },
        {
          "key" : "5.0-10.0",
          "from" : 5.0,
          "to" : 10.0,
          "doc_count" : 2,
          "my_agg" : {
            "value" : 7.5
          }
        }
      ]
    }
  }
}
大数据测试 elasticsearch
上一篇: Python数据结构与算法之希尔排序
下一篇: MongoDB分片式安装部署 OpenEuler/Centos

继续阅读