Erlang, delta replica system

erlang节点间传输eterm前，erlang会对eterm进行编码，如果eterm本身比较大，编解码会对系统性能造成较大的影响。对于某些特定的应用，例如备份系统，是可以通过某些方法进行优化的。

我们有一套N+1备份系统，enode里，一个worker流程结束时，会向备份enode发送状态数据，我们简单的使用{replica_handler, rep_enode} ! Msg来实现enode间消息的传递。随着系统日益庞大以及用户量的增长，平均每个worker的状态数据达到了4M左右，状态数据编解码对系统性能造成了不小的影响。

通过系统一些调试工具我们发现再我们的系统内, 单个worker相邻的两次状态数据，大部分是相同的。因此我们引入了差量备份(delta replica)机制。核心思想比较简单（类似于视频编码的关键帧和参考帧)

1. 状态备份时，首次必须发送完整的状态信息给备份节点
2. 随后i次状态备份，仅发送与上一次状态信息的差量部分给备份节点
3. 当i%10 == 0时，强制发送完整的状态信息

详见

https://github.com/dead911/erlang/wiki/erlang-delta_replica

差量备份核心代码开源如下，仅供娱乐~

https://github.com/dead911/erlang/tree/master/delta_replica

%%% -------------------------------------------------------------------------
%%% Delta spec util 
%%% 
%%% delta_spec_util:get_delta_spec(Old, New) -> DeltaSpec
%%%   to cacluate the delta spec of two given tuples
%%%      Old - {user_info, {timestamp,}, {tid, }, {loc, {,}, na}}
%%%      New - {user_info, {timestamp,}, {tid, }, {loc, {,}, na}}
%%%
%%%   When delta_spec_level = , the output should be
%%%      DeltaSpec -[{'$1',
%%%                     [{'$2',[{'$2',}]},
%%%                      {'$4',[{'$2',[{'$2',}]}]}]
%%%                  }]
%%%   DeltaSpec indicates the difference between New and Old, each difference 
%%%   is presented in format [{<pos>, <value>}].
%%%      {'$2',[{'$2',}]} the first '$2' implies the the nd element 
%%%      of New is different from Old. In this case, it is the 'timestamp'.
%%%      The following [{'$2',}] means the second element of timestamp
%%%      of New is differnt from of Old and the latest value is 
%%%
%%% merge_delta_spec(OldSpec, NewSpec) -> NewSpec
%%%   to merge two specs
%%%   Considering the following scneario, when value changes from
%%%   {hello, ,,} -> {hello, ,,} -> {hello, ,,}
%%%   Two specs will be generated,
%%%   OldSpec = [{'$4',}], {hello, ,,} -> {hello, ,,}, 
%%%   NewSpec = [{'$3',}], {hello, ,,} -> {hello, ,,} 
%%%   After merge, NewSpec = [{'$3',}, {'$4',}]
%%%
%%% apply_spec_to_data(Data, Spec) -> NewData
%%%   to apply spec to data
%%%   Data    - {, [ ,], }
%%%   Spec    - [{'$1, {'$2',[{'$1',12}]},{'$3',5}]}]
%%%   NewData - {1, [12,3], 5}
%%% -------------------------------------------------------------------------