python中读写Protobuf总结

Protobuf是谷歌开源的协议框架，以高效传输著称并且支持多种语言。工作中也用到了，在此做个总结。

1. Protobuf运行原理

下面一张图可以说明：

1. Python中使用Protobuf

2.1安装protoc程序

protoc --version

我安装的是3.6.1版本

2.2定义协议

syntax = "proto3";
package com.union.fun;

message Metric {
    string name = 1;
    string type = 2;
    float value = 3;
    repeated string tags = 4;
  }
           

2.3生成接口

//Python接口生成
protobuf/bin/protoc  -I=/home/tiger/python  --python_out=/home/tiger/python  metric.proto

//另外java和C++生成方式如下
//Java接口生成
protobuf/bin/protoc  -I=/home/tiger/python  --java_out=/home/tiger/python  metric.proto

//C++接口生成
protobuf/bin/protoc  -I=/home/tiger/python  --cpp_out=/home/tiger/python metric.proto
           

2.4读写调用

（1）单个对象

import metric_pb2


def writepb():
    my_metric = metric_pb2.Metric()
    my_metric.name = 'sys.cpu'
    my_metric.type = 'gauge'
    my_metric.value = 99.9
    my_metric.tags.extend(['my_tag', 'foo:bar'])

    with open('out.bin', 'wb') as f:
        f.write(my_metric.SerializeToString())
    
def readpb():
    with open('out.bin', 'rb') as f:
        read_metric = metric_pb2.Metric()
        read_metric.ParseFromString(f.read())
        print(read_metric)
        # do something with read_metric

if __name__ == "__main__":
    writepb()
    readpb()
           

运行结果如下：python3 method1.py

（2）连续多个对象

import metric_pb2
from google.protobuf.internal.encoder import _VarintBytes
from google.protobuf.internal.decoder import _DecodeVarint32
import random

def writepb():
    with open('out.bin', 'wb') as f:
        my_tags = ("my_tag", "foo:bar")
        for i in range(128):
            my_metric = metric_pb2.Metric()
            my_metric.name = 'sys.cpu'
            my_metric.type = 'gauge'
            my_metric.value = round(random.random(), 2)
            my_metric.tags.extend(my_tags)
            size = my_metric.ByteSize()
            f.write(_VarintBytes(size))
            f.write(my_metric.SerializeToString())
    
def readpb():
    with open('out.bin', 'rb') as f:
        buf = f.read()
        n = 0
        while n < len(buf):
            msg_len, new_pos = _DecodeVarint32(buf, n)
            n = new_pos
            msg_buf = buf[n:n+msg_len]
            n += msg_len
            read_metric = metric_pb2.Metric()
            read_metric.ParseFromString(msg_buf)
            print(read_metric)
            # do something with read_metric

if __name__ == "__main__":
    writepb()
    readpb()
           

运行结果如下：python3 method2.py

参考文档如下：

https://www.datadoghq.com/blog/engineering/protobuf-parsing-in-python/