天天看点

mapreduce中counter的使用

   mapreduce counter为提供我们一个窗口:观察mapreduce job运行期的各种细节数据。mapreduce自带了许多默认counter。

    counter有"组group"的概念,用于表示逻辑上相同范围的所有数值。mapreduce job提供的默认counter分为三个组

map-reduce frameword

map input records,map skipped records,map input bytes,map output records,map output bytes,combine input records,combine output records,reduce input records,reduce input groups,reduce output records,reduce skipped groups,reduce skipped records,spilled records

file systems

filesystem bytes read,filesystem bytes written

job counters

launched map tasks,launched reduce tasks,failed map tasks,failed reduce tasks,data-local map tasks,rack-local map tasks,other local map tasks

        这些 counters你在web ui中,或是job结束后在控制台生成的统计报告中都看得到。 见如下mr运行日志:

使用java enum自定义counter

一个counter可以是任意的enum类,见如下代码示例: