列存儲壓縮技巧，除公共除數或者同時減去最小數，字元串壓縮的話，直接去重後用數字ID壓縮

Column-store compression

At a high level, doc values are essentially a serialized column-store. As we discussed in the last section, column-stores excel at certain operations because the data is naturally laid out in a fashion that is amenable to those queries.

But they also excel at compressing data, particularly numbers. This is important for both saving space on disk and for faster access. Modern CPU’s are many orders of magnitude faster than disk drives (although the gap is narrowing quickly with upcoming NVMe drives). That means it is often advantageous to minimize the amount of data that must be read from disk, even if it requires extra CPU cycles to decompress.

To see how it can help compression, take this set of doc values for a numeric field:

The column-stride layout means we have a contiguous block of numbers:<code>[100,1000,1500,1200,300,1900,4200]</code>. 

xxx

Doc values use several tricks like this. In order, the following compression schemes are checked:

If all values are identical (or missing), set a flag and record the value

If there are fewer than 256 values, a simple table encoding is used

If there are &gt; 256 values, check to see if there is a common divisor

If there is no common divisor, encode everything as an offset from the smallest value

You’ll note that these compression schemes are not "traditional" general purpose compression like DEFLATE or LZ4. Because the structure of column-stores are rigid and well-defined, we can achieve higher compression by using specialized schemes rather than the more general compression algorithms like LZ4.

You may be thinking "Well that’s great for numbers, but what about strings?" Strings are encoded similarly, with the help of an ordinal table. The strings are de-duplicated and sorted into a table, assigned an ID, and then those ID’s are used as numeric doc values. Which means strings enjoy many of the same compression benefits that numerics do.

The ordinal table itself has some compression tricks, such as using fixed, variable or prefix-encoded strings.

摘自：https://www.elastic.co/guide/en/elasticsearch/guide/current/_deep_dive_on_doc_values.html

本文轉自張昺華-sky部落格園部落格，原文連結：http://www.cnblogs.com/bonelee/p/6401472.html，如需轉載請自行聯系原作者