python中讀寫Protobuf總結

Protobuf是谷歌開源的協定架構，以高效傳輸著稱并且支援多種語言。工作中也用到了，在此做個總結。

1. Protobuf運作原理

下面一張圖可以說明：

1. Python中使用Protobuf

2.1安裝protoc程式

protoc --version

我安裝的是3.6.1版本

2.2定義協定

syntax = "proto3";
package com.union.fun;

message Metric {
    string name = 1;
    string type = 2;
    float value = 3;
    repeated string tags = 4;
  }
           

2.3生成接口

//Python接口生成
protobuf/bin/protoc  -I=/home/tiger/python  --python_out=/home/tiger/python  metric.proto

//另外java和C++生成方式如下
//Java接口生成
protobuf/bin/protoc  -I=/home/tiger/python  --java_out=/home/tiger/python  metric.proto

//C++接口生成
protobuf/bin/protoc  -I=/home/tiger/python  --cpp_out=/home/tiger/python metric.proto
           

2.4讀寫調用

（1）單個對象

import metric_pb2


def writepb():
    my_metric = metric_pb2.Metric()
    my_metric.name = 'sys.cpu'
    my_metric.type = 'gauge'
    my_metric.value = 99.9
    my_metric.tags.extend(['my_tag', 'foo:bar'])

    with open('out.bin', 'wb') as f:
        f.write(my_metric.SerializeToString())
    
def readpb():
    with open('out.bin', 'rb') as f:
        read_metric = metric_pb2.Metric()
        read_metric.ParseFromString(f.read())
        print(read_metric)
        # do something with read_metric

if __name__ == "__main__":
    writepb()
    readpb()
           

運作結果如下：python3 method1.py

（2）連續多個對象

import metric_pb2
from google.protobuf.internal.encoder import _VarintBytes
from google.protobuf.internal.decoder import _DecodeVarint32
import random

def writepb():
    with open('out.bin', 'wb') as f:
        my_tags = ("my_tag", "foo:bar")
        for i in range(128):
            my_metric = metric_pb2.Metric()
            my_metric.name = 'sys.cpu'
            my_metric.type = 'gauge'
            my_metric.value = round(random.random(), 2)
            my_metric.tags.extend(my_tags)
            size = my_metric.ByteSize()
            f.write(_VarintBytes(size))
            f.write(my_metric.SerializeToString())
    
def readpb():
    with open('out.bin', 'rb') as f:
        buf = f.read()
        n = 0
        while n < len(buf):
            msg_len, new_pos = _DecodeVarint32(buf, n)
            n = new_pos
            msg_buf = buf[n:n+msg_len]
            n += msg_len
            read_metric = metric_pb2.Metric()
            read_metric.ParseFromString(msg_buf)
            print(read_metric)
            # do something with read_metric

if __name__ == "__main__":
    writepb()
    readpb()
           

運作結果如下：python3 method2.py

參考文檔如下：

https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/