ES安裝、SpringBoot內建ES

一、安裝ES

下載下傳ES包，去官網下載下傳頁可以：

#大約280M
連結：https://pan.baidu.com/s/1LaWlIwrI7UxVEdytoD1JWw 
提取碼：6p0j

壓縮包上傳到/usr/local目錄下
tar -xzvf elasticsearch-7.5.0-linux-x86_64.tar.gz
adduser es
#賦予檔案夾權限
chown -R es /usr/local/elasticsearch-7.5.0/
#切換使用者
su es
/usr/local/elasticsearch-7.5.0/bin/elasticsearch -d #-d 在背景以守護程序模式運作

#校驗運作情況，出現下圖則代表啟動成功：
netstat -lntp
curl 'http://localhost:9200/?pretty'

ES安裝、SpringBoot內建ES

二、簡單配置ES

#修改配置檔案
vi /usr/local/elasticsearch-7.5.0/config/elasticsearch.yml

# Use a descriptive name for your cluster:
# 起的名字
cluster.name: big-data
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
# 節點名字
node.name: node-1
#

#
# Set the bind address to a specific IP (IPv4 or IPv6):
# 目前主節點的host
network.host: 123.57.133.22
#目前區域網路節點 單機es則配置127.0.0.1
transport.host: 127.0.0.1
#網絡限制ip  目前配置不做任何限制，任何ip都能通路目前es
http.host: 0.0.0.0
#
# Set a custom port for HTTP:
# 端口，預設也為9200
http.port: 9200

三、一般報錯處理

1、max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

每個程序最大同時打開檔案數太小，可通過下面2個指令檢視目前數量

ulimit -Hn
ulimit -Sn

修改/etc/security/limits.conf檔案，增加配置，使用者退出後重新登入生效

*               soft    nofile          65536
*               hard    nofile          65536

2、max number of threads [3818] for user [es] is too low, increase to at least [4096]

問題同上，最大線程個數太低。修改配置檔案/etc/security/limits.conf（和問題1是一個檔案），增加配置
*               soft    nproc           4096
*               hard    nproc           4096
#可通過指令檢視
ulimit -Hu
ulimit -Su

3、max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

　修改/etc/sysctl.conf檔案，增加配置vm.max_map_count=262144
vi /etc/sysctl.conf
sysctl -p  #執行指令sysctl -p生效

4、Exception in thread “main” java.nio.file.AccessDeniedException: /usr/local/elasticsearch/elasticsearch-6.2.2-1/config/jvm.options

　　elasticsearch使用者沒有該檔案夾的權限，執行指令

chown -R es:es /usr/local/elasticsearch/

四、springBoot內建ES

1. 引入maven

<!--es依賴   -->
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>7.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.5.0</version>
            <scope>compile</scope>
        </dependency>

2.配置application.properties或yml檔案

elasticsearch.user:   #使用者 未設定則空
elasticsearch.password:   #密碼 未設定則空
elasticsearch.host: 123.57.xx.xx:9200    #ip和端口号

3.簡單新增操作

/**
     * 添加
     * @return
     */
    @RequestMapping(value="add",method = RequestMethod.POST)
    public void add( ){

        String index = "my_index";
        try {
            for(int i=0;i<20;i++) {
                XContentBuilder builder = XContentFactory.jsonBuilder()
                        .startObject()
                        .field("id", i+"")
                        .field("content", "内容"+i)
                        .field("age",(20+i)+"")
                        .endObject();
                String id = ESUtil.addData(builder, index, i+"");
                logger.info("插入一條資料,id為{}",i);
            }
        } catch (IOException e) {
             e.printStackTrace();
        }
    }

4.簡單删除操作

/**
     * 删除
     * @return
     */
    @RequestMapping(value="delete",method = RequestMethod.POST)
    @ResponseBody
    public String delete(@RequestParam String id){

        String index = "my_index";
        //根據id删除
        String status = ESUtil.deleteById(index, id);
        //列印查詢結果
        return "success";
    }

5.簡單修改操作

/**
     * 更新
     * @return
     */
    @RequestMapping(value="update",method = RequestMethod.POST)
    public String update(@RequestParam String id)throws  Exception{

        String index = "my_index";
        XContentBuilder builder =XContentFactory.jsonBuilder()
                .startObject()
                .field("id",id)
                .field("content","這是内容"+id)
                .field("age",id).endObject();
        String responseId = ESUtil.updateData(builder, index, id);
        logger.info("列印更新的id"+responseId);
        //列印查詢結果
        return responseId;
    }

6.簡單查詢操作

/**
     * 分頁+條件查詢
     * @return
     */
    @RequestMapping(value="get",method = RequestMethod.POST)
    public void get( ){

        String index = "my_index";
        //分頁+條件查詢
        BoolQueryBuilder builder = QueryBuilders.boolQuery();
        builder.must(QueryBuilders.matchQuery("content","内容2"));
        SearchSourceBuilder sourceBuilder =new SearchSourceBuilder();
        List<Map<String, Object>> list = ESUtil.SearchDataPage(index, 1, 20, sourceBuilder, builder);
        //列印查詢結果
        System.out.println(list);
    }

7.上面代碼用到的工具類

ESUtil

package com.example.util;

import com.example.pojo.EsEntity;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.reindex.DeleteByQueryRequest;
import org.elasticsearch.rest.RestStatus;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Component;

import javax.annotation.PostConstruct;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class ESUtil {

    private static final Logger log = LoggerFactory.getLogger(ESUtil.class);
    @Qualifier("highLevelClient")
    @Autowired
    private RestHighLevelClient rhlClient;

    private static RestHighLevelClient client;

    /**
     * spring容器初始化的時候執行該方法
     */
    @PostConstruct
    public void init() {
        client = this.rhlClient;
    }

    /**
     * 添加資料
     *
     * @param content 資料内容
     * @param index   索引
     * @param id      id
     */
    public static String addData(XContentBuilder content, String index, String id) {
        String Id = null;
        try {
            IndexRequest request = new IndexRequest(index).id(id).source(content);
            IndexResponse response = client.index(request, RequestOptions.DEFAULT);
            Id = response.getId();
            log.info("索引:{},資料添加,傳回碼:{},id:{}", index, response.status().getStatus(), Id);
        } catch (IOException e) {
            log.error("添加資料失敗,index:{},id:{}", index, id);
        }
        return Id;
    }

    /**
     * 修改資料
     *
     * @param content 修改内容
     * @param index   索引
     * @param id      id
     */
    public static String updateData(XContentBuilder content, String index, String id) {
        String Id = null;
        try {
            UpdateRequest request = new UpdateRequest(index, id).doc(content);
            UpdateResponse response = client.update(request, RequestOptions.DEFAULT);
            Id = response.getId();
            log.info("資料更新,傳回碼:{},id:{}", response.status().getStatus(), Id);
        } catch (IOException e) {
            log.error("資料更新失敗,index:{},id:{}", index, id);
        }
        return Id;
    }

    /**
     * 批量插入資料
     *
     * @param index 索引
     * @param list  批量增加的資料
     */
    public static String insertBatch(String index, List<EsEntity> list) {
        String state = null;
        BulkRequest request = new BulkRequest();
        list.forEach(item -> request.add(new IndexRequest(index)
                .id(item.getId()).source(JsonUtils.toJson(item.getData()), XContentType.JSON)));
        try {
            BulkResponse bulk = client.bulk(request, RequestOptions.DEFAULT);
            int status = bulk.status().getStatus();
            state = Integer.toString(status);
            log.info("索引:{},批量插入{}條資料成功!", index, list.size());
        } catch (IOException e) {
            log.error("索引:{},批量插入資料失敗", index);
        }
        return state;
    }

    /**
     * 根據條件删除資料
     *
     * @param index   索引
     * @param builder 删除條件
     */
    public static void deleteByQuery(String index, QueryBuilder builder) {
        DeleteByQueryRequest request = new DeleteByQueryRequest(index);
        request.setQuery(builder);
        //設定此次删除的最大條數
        request.setBatchSize(1000);
        try {
            client.deleteByQuery(request, RequestOptions.DEFAULT);
        } catch (IOException e) {
            log.error("根據條件删除資料失敗,index:{}", index);
        }
    }

    /**
     * 根據id删除資料
     *
     * @param index 索引
     * @param id    id
     */
    public static String deleteById(String index, String id) {
        String state = null;
        DeleteRequest request = new DeleteRequest(index, id);
        try {
            DeleteResponse response = client.delete(request, RequestOptions.DEFAULT);
            int status = response.status().getStatus();
            state = Integer.toString(status);
            log.info("索引:{},根據id{}删除資料:{}", index, id, JsonUtils.toJson(response));
        } catch (IOException e) {
            log.error("根據id删除資料失敗,index:{},id:{}", index, id);
        }
        return state;
    }


    /**
     * 根據條件查詢資料
     *
     * @param index         索引
     * @param startPage     開始頁
     * @param pageSize      每頁條數
     * @param sourceBuilder 查詢傳回條件
     * @param queryBuilder  查詢條件
     */
    public static List<Map<String, Object>> SearchDataPage(String index, int startPage, int pageSize,
                                                           SearchSourceBuilder sourceBuilder, QueryBuilder queryBuilder) {
        SearchRequest request = new SearchRequest(index);
        //設定逾時時間
        sourceBuilder.timeout(new TimeValue(120, TimeUnit.SECONDS));
        //設定是否按比對度排序
        sourceBuilder.explain(true);
        //加載查詢條件
        sourceBuilder.query(queryBuilder);
        //設定分頁
        sourceBuilder.from((startPage - 1) * pageSize).size(pageSize);
        log.info("查詢傳回條件：" + sourceBuilder.toString());
        request.source(sourceBuilder);
        try {
            SearchResponse searchResponse = client.search(request, RequestOptions.DEFAULT);
            long totalHits = searchResponse.getHits().getTotalHits().value;
            log.info("共查出{}條記錄", totalHits);
            RestStatus status = searchResponse.status();
            if (status.getStatus() == 200) {
                List<Map<String, Object>> sourceList = new ArrayList<>();
                for (SearchHit searchHit : searchResponse.getHits().getHits()) {
                    Map<String, Object> sourceAsMap = searchHit.getSourceAsMap();
                    sourceList.add(sourceAsMap);
                }
                return sourceList;
            }
        } catch (IOException e) {
            log.error("條件查詢索引{}時出錯", index);
        }
        return null;
    }

}

用到的EsEntity

package com.example.pojo;
public class EsEntity {
    private String id;
    private Object data;

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public Object getData() {
        return data;
    }

    public void setData(Object data) {
        this.data = data;
    }
}

用到的JsonUtils

package com.example.util;

import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.core.JsonGenerationException;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.*;

/**
 * @author xiaolang
 * @version 1.0.0
 */
public class JsonUtils {
    private static Logger logger = LoggerFactory.getLogger(JsonUtils.class);
    private static final ObjectMapper objectMapper;
    static {
        objectMapper = new ObjectMapper();
        //去掉預設的時間戳格式
        objectMapper.configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false);
        //設定為中國上海時區
        objectMapper.setTimeZone(TimeZone.getTimeZone("GMT+8"));
//        objectMapper.configure(SerializationFeature.WRITE_NULL_MAP_VALUES, false);
        //空值不序列化
        objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
        //反序列化時，屬性不存在的相容處理
        objectMapper.getDeserializationConfig().withoutFeatures(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
        //序列化時，日期的統一格式
        objectMapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));

        objectMapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false);
        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
        //單引号處理
        objectMapper.configure(com.fasterxml.jackson.core.JsonParser.Feature.ALLOW_SINGLE_QUOTES, true);

    }

    public static <T> T toObject(String json, Class<T> clazz) {
        try {
            return objectMapper.readValue(json, clazz);
        } catch (JsonParseException e) {
            logger.error(e.getMessage(), e);
        } catch (JsonMappingException e) {
            logger.error(e.getMessage(), e);
        } catch (IOException e) {
            logger.error(e.getMessage(), e);
        }
        return null;
    }

    public static <T> String toJson(T entity) {
        try {
            return objectMapper.writeValueAsString(entity);
        } catch (JsonGenerationException e) {
            logger.error(e.getMessage(), e);
        } catch (JsonMappingException e) {
            logger.error(e.getMessage(), e);
        } catch (IOException e) {
            logger.error(e.getMessage(), e);
        }
        return null;
    }

    public static <T> T toCollection(String json, TypeReference<T> typeReference) {
        try {
            return objectMapper.readValue(json, typeReference);
        } catch (JsonParseException e) {
            logger.error(e.getMessage(), e);
        } catch (JsonMappingException e) {
            logger.error(e.getMessage(), e);
        } catch (IOException e) {
            logger.error(e.getMessage(), e);
        }
        return null;
    }

    /**
     * json string convert to map
     */
    public static <T> Map<String, Object> json2map(String jsonStr)
            throws Exception {
        return objectMapper.readValue(jsonStr, Map.class);
    }

    /**
     * json string convert to map with javaBean
     */
    public static <T> Map<String, T> json2map(String jsonStr, Class<T> clazz)
            throws Exception {
        Map<String, Map<String, Object>> map = objectMapper.readValue(jsonStr,
                new TypeReference<Map<String, T>>() {
                });
        Map<String, T> result = new HashMap<String, T>();
        for (Map.Entry<String, Map<String, Object>> entry : map.entrySet()) {
            result.put(entry.getKey(), map2pojo(entry.getValue(), clazz));
        }
        return result;
    }

    /**
     * json array string convert to list with javaBean
     */
    public static <T> List<T> json2list(String jsonArrayStr, Class<T> clazz)
            throws Exception {
        List<Map<String, Object>> list = objectMapper.readValue(jsonArrayStr,
                new TypeReference<List<T>>() {
                });
        List<T> result = new ArrayList<T>();
        for (Map<String, Object> map : list) {
            result.add(map2pojo(map, clazz));
        }
        return result;
    }

    /**
     * map convert to javaBean
     */
    public static <T> T map2pojo(Map map, Class<T> clazz) {
        return objectMapper.convertValue(map, clazz);
    }

}

ES安裝、SpringBoot內建ES

一、安裝ES

二、簡單配置ES

三、一般報錯處理

繼續閱讀

關于Gradle配置的小結

Java小案例——随機數猜測随機數猜測

nginx location中斜線的位置的重要性

27 Best Free Eclipse Plug-ins for Java Developer to be ProductiveCode Quality PluginsText Editor PluginsDependency ManagementVersion Control Integration PluginsFramework Development Continuous Integration Related PluginsOther Utility Plugins

Java String.format方法的簡單使用

neo4j之cypher使用文檔

GitHub連夜封殺！這份阿裡 10W 字内部 Java 字面試手冊到底有多強？

spark/scala關于【資源檔案】加載方法概述外部檔案加載方案測試資源檔案打包入jar包中小結

mybatis_入門程式Mybatis入門

AOP程式設計_Android優雅權限架構(1)概念基礎，2021金三銀四前言正文大綱正文

Effective Java 8:通用程式設計

OOM三種類型

工廠模式-三種類型

【遞歸】高效率求2的n次幂

win10本地scala和spark安裝安裝scala安裝spark

scala (3) Function 和 Method