Java Heap Pressure
Elasticsearch has so many wildly different use cases that I could not write a reasonably short blog post describing what can and cannot consume memory. However, there is one thing that constantly stands out above all of the other concerns that you might have while running an Elasticsearch cluster at scale.
The inverted index is the magic that makes Elasticsearch queries so fast. This data structure holds a sorted list of all the unique terms that appear in a field, and each term points to the list of documents that contain that term:
Search asks the question: What documents contain term <code>brown</code> in the <code>foo</code> field? The inverted index is the perfect structure to answer this question: look up the term of interest in the sorted list and you immediately know which documents match your query.
Sorting or aggregations, however, need to be able to answer this question: What terms does Document 1 contain in the <code>foo</code> field? To answer this, we need a data structure that is the opposite of the inverted index:
This is the purpose of fielddata. Fielddata can be generated at query time by reading the inverted index, inverting the term <-> doc data structure, and storing the results in memory.（就是doc->terms的正向索引啊，不過它是在查詢階段通過讀取反向索引得到的？如果真是這樣，那麼如何能夠比doc values更快？） The two major downsides of this approach should be obvious:
Loading fielddata can be slow, especially with big segments.
It consumes a lot of valuable heap space.
Because loading fielddata is costly, we try to do it as seldom as possible. Once loaded, we keep it in memory for as long as possible.
By default, fielddata is loaded on demand, which means that you will not see it until you are using it. Also, by being loaded per segment, it means that new segments that get created will slowly add to your overall memory usage until the field's fielddata is evicted from memory. Eviction happens in only a few ways:
Deleting the index or indices that contains it.
Closing the index or indices that contains it.
Segment fielddata is removed when segments are removed (e.g., background merging).
This usually just means that the problem is moving rather than going away.
Restarting the node containing the fielddata.
Clearing the relevant fielddata cache.
Automatically evicting the fielddata to make room for other fielddata.
This will not happen with default settings.
While the first two ways will cause the memory to be evicted, they're not useful in terms of solving the problem because they make the index unusable. Segment merging is happening in the background and it is not a way to clear fielddata. The fourth and fifth ways are unlikely to be a long term solution because they do not prevent fielddata from being reloaded.
The sixth option, evicting fielddata when the cache is full, leads to different issues: one request triggers fielddata loading for one field and the next request triggers loading for another, causing the first field to be evicted. This causes memory thrashing and slow garbage collections, and your users suffer from very slow queries while they wait for their fielddata to be loaded.
Simply put, once fielddata becomes a problem, then it stays a problem.
At small scales, you can generally get away with fielddata usage without even realizing that you are using it. In highly controlled environments, you may even enjoy that specific fields are being loaded into memory for theoretically faster access.
However, almost without fail, you are bound to run into a problem with it eventually. Whether it's because someone ran a test request on the production system without thinking that it would be a problem (it's just one query, right?), your queries changed to match new data, or you just finally reached a scale where it no longer works: you will eventually run into memory pressure that does not go away.
For example, if a node has 10 GBs of Java heap, then 60% of that is going to be 6 GBs. If a new request requires 1 GB of fielddata to be loaded for that node that is already using 4 GBs of the heap for fielddata, then it will allow it because 4 GB, plus 1 GB, is less than 6 GB. However, if the next request needed 2 GB for yet another field's fielddata, then the entire request would be rejected because the fielddata is exhausted (<code>5 GB + 2 GB = 7 GB</code>, which is clearly greater than 6 GB).
Note: for versions prior to Elasticsearch 1.3, we allowed an unlimited amount of your Java heap to be consumed by fielddata.
Fortunately, it's not all bad news. Not only do we have a solution to the problem, but we also provide a way to find and understand your problem with it.
This will provide a list of each node with its fielddata usage. For instance, at startup, my local node is using absolutely no fielddata:
To see it change, it's as simple as sorting, scripting, or aggregating any field. So let's do all three!
Although order is irrelevant for this, the first field that will be impacted will be the <code>percentage</code> field that is accessed inside of the scripted filter. The second field used will be the <code>number</code> field from the aggregation. Finally, the last field is the <code>@timestamp</code> field used to sort the filtered results. Taking another look at the <code>_cat/fielddata</code> command above confirms this:
By taking the burden out of your heap, you get fast access to the on-disk fielddata through the file system cache, which gives in-memory performance without the cost of garbage collections coming into play. This also frees up a lot of headroom for the Elasticsearch heap so that more operations (e.g., bulk indexing and concurrent searches) can use the heap without placing the node under memory pressure, which leads to garbage collection that will slow it down.
本文轉自張昺華-sky部落格園部落格，原文連結：http://www.cnblogs.com/bonelee/p/6401686.html，如需轉載請自行聯系原作者
![Java小案例——随機數猜測随機數猜測[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)