Impala查询流程日志分析

UHP博客文章地址：http://yuntai.1kapp.com/?p=1060

原创文章，转载请注明出处：http://blog.csdn.net/wind5shy/article/details/8563355

物理环境

主节点	10.28.171.33
处理节点1	10.28.171.40
处理节点2	10.28.171.41
处理节点3	10.28.169.42
处理节点4	10.28.171.43

查询SQL语句

select jobinfo.dt,user,max(taskinfo.finish_time-taskinfo.start_time),max(jobinfo.finish_time-jobinfo.submit_time)

fromtaskinfo join jobinfo on jobinfo.jobid=taskinfo.jobid

where  jobinfo.job_status='SUCCESS' and taskinfo.task_status='SUCCESS'

group by jobinfo.dt,user

连接到处理节点3上进行处理。

处理节点3日志

解释SQL语句，生成查询计划树

Explain query: selectjobinfo.dt,user,max(taskinfo.finish_time-taskinfo.start_time),max(jobinfo.finish_time-jobinfo.submit_time)from taskinfo join jobinfo on jobinfo.jobid=taskinfo.jobid where  jobinfo.job_status='SUCCESS' andtaskinfo.task_status='SUCCESS' group by jobinfo.dt,user

协调器PlanFragment，查询计划树的根

Plan Fragment 0

 UNPARTITIONED

 AGGREGATE

  从各backend结果中再次处理MAX与GROUP BY

 OUTPUT: MAX(<slot 12>), MAX(<slot 13>)

 GROUP BY: <slot 10>, <slot 11>

 TUPLE IDS: 2

   EXCHANGE (5)

     TUPLE IDS: 2

Plan Fragment 1

Plan Fragment 1

 RANDOM

 STREAM DATA SINK

   EXCHANGE ID: 5

   UNPARTITIONED

 AGGREGATE

处理MAX与GROUP BY

 OUTPUT: MAX(taskinfo.finish_time - taskinfo.start_time),MAX(jobinfo.finish_time - jobinfo.submit_time)

 GROUP BY: jobinfo.dt, user

 TUPLE IDS: 2

         处理join

HASH JOIN

     JOIN OP: INNER JOIN

     HASH PREDICATES:

       taskinfo.jobid = jobinfo.jobid

     TUPLE IDS: 0 1

扫描taskinfo表数据并用taskinfo.task_status= 'SUCCESS'进行筛选

       SCAN HDFS table=default.taskinfo (0)

         PREDICATES: taskinfo.task_status = 'SUCCESS'

         TUPLE IDS: 0

       EXCHANGE (4)

         TUPLE IDS: 1

Plan Fragment 2

Plan Fragment 2

 RANDOM

 STREAM DATA SINK

   EXCHANGE ID: 4

   UNPARTITIONED

扫描jobinfo表数据并用jobinfo.job_status= 'SUCCESS'进行筛选

 SCAN HDFS table=default.jobinfo (1) compact

   PREDICATES: jobinfo.job_status = 'SUCCESS'

   TUPLE IDS: 1

查询配置

I0122 20:10:04.561157  1031 impala-server.cc:1516] TClientRequest.queryOptions:TQueryOptions {

  01:abort_on_error (bool) = false,

  02:max_errors (i32) = 0,

  03:disable_codegen (bool) = false,

  04:batch_size (i32) = 0,

  05:return_as_ascii (bool) = true,

  06:num_nodes (i32) = 0,

  07:max_scan_range_length (i64) = 0,

  08:num_scanner_threads (i32) = 0,

  09:max_io_buffers (i32) = 0,

  10:allow_unsupported_formats (bool) = false,

  11:partition_agg (bool) = false,

}

开始执行查询

I0122 20:10:04.561363  1031 impala-server.cc:863] query(): query=selectjobinfo.dt,user,max(taskinfo.finish_time-taskinfo.start_time),max(jobinfo.finish_time-jobinfo.submit_time)from taskinfo join jobinfo on jobinfo.jobid=taskinfo.jobid where  jobinfo.job_status='SUCCESS' andtaskinfo.task_status='SUCCESS' group by jobinfo.dt,user

协调器根据查询计划树给各个Backend节点分配任务

impala-server.cc:1119

I0122 20:10:05.271327  1031 coordinator.cc:219] Exec()query_id=67692cad49234453:8cf581505134c8ca

coordinator.cc:1161

Plan Fragment 1任务分配

I0122 20:10:05.271467  1031 simple-scheduler.cc:171] SimpleSchedulerassignment (data->backend): (10.28.171.43:50010 -> 10.28.171.43:22000), (10.28.171.41:50010 ->10.28.171.41:22000), (10.28.171.42:50010 -> 10.28.171.42:22000),(10.28.171.40:50010 -> 10.28.171.40:22000)

Plan Fragment 1 分配到任务的4个Backend均为本地处理

I0122 20:10:05.271487  1031 simple-scheduler.cc:174] SimpleSchedulerlocality percentage 100% (4 out of 4)

Plan Fragment 2任务分配

I0122 20:10:05.271610  1031 simple-scheduler.cc:171] SimpleSchedulerassignment (data->backend): (10.28.171.43:50010 -> 10.28.171.43:22000), (10.28.171.41:50010 ->10.28.171.41:22000), (10.28.171.42:50010 -> 10.28.171.42:22000),(10.28.171.40:50010 -> 10.28.171.40:22000)

Plan Fragment 2 分配到任务的4个Backend均为本地处理

I0122 20:10:05.271627  1031 simple-scheduler.cc:174] SimpleSchedulerlocality percentage 100% (4 out of 4)

协调器开始在各个Backend执行查询计划

coordinator.cc:255

I0122 20:10:05.271853  1031 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cb

I0122 20:10:05.277370  1031 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8cb

I0122 20:10:05.335903  1031 coordinator.cc:308] starting 8 backendsfor query 67692cad49234453:8cf581505134c8ca

本节点开始处理PlanFragment 1 Backend 2的任务

coordinator.cc:698

I0122 20:10:05.336571  1119 impala-server.cc:1655(1784)]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8cecoord=10.28.171.42:22000 backend#=2

I0122 20:10:05.336617  1119 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8ca instance_id=67692cad49234453:8cf581505134c8ce

I0122 20:10:05.339373  1119 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8ce

I0122 20:10:05.462807 11899plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8ce

本节点开始处理PlanFragment 2 Backend 6的任务

coordinator.cc:698

I0122 20:10:05.468166  1119 impala-server.cc:1655]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8d2coord=10.28.171.42:22000 backend#=6

I0122 20:10:05.468377  1119 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8d2

I0122 20:10:05.475342  1119 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8d2

I0122 20:10:05.486824 11899 aggregation-node.cc:166]AggregationNode(node_id=3) using llvm codegend functions.

I0122 20:10:05.490782 11899hash-join-node.cc:157] HashJoinNode(node_id=2) using llvm codegend function forbuilding hash table.

I0122 20:10:05.499639 11913plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8d2

I0122 20:10:05.499886 11916plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8cb

I0122 20:10:05.499991 11899hash-join-node.cc:170] HashJoinNode(node_id=2) using llvm codegend function forprobing hash table.

协调器接受各个后端的处理进度

I0122 20:10:05.500530  1560 coordinator.cc:904] Backend 4 completed,7 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:10:05.500568  1560 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000

I0122 20:10:05.525993 11916aggregation-node.cc:166] AggregationNode(node_id=6) using llvm codegendfunctions.

I0122 20:10:10.237057  1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 2% Complete (1 out of 36)

I0122 20:10:10.237105  1127 coordinator.cc:904] Backend 5 completed,6 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:10:10.237129  1127 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000

I0122 20:10:10.518767  1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 5% Complete (2 out of 36)

I0122 20:10:11.710980  1129 coordinator.cc:904] Backend 7 completed,5 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:10:11.711033  1129 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000

I0122 20:10:12.263823  1119 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 11% Complete (4 out of 36)

I0122 20:10:12.263871  1119 coordinator.cc:904] Backend 6 completed,4 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:10:12.263895  1119 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000

I0122 20:10:30.025861  1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 13% Complete (5 out of 36)

I0122 20:10:30.759806  1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 16% Complete (6 out of 36)

I0122 20:11:16.298985  1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 25% Complete (9 out of 36)

I0122 20:11:21.365696  1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 27% Complete (10 out of 36)

I0122 20:11:21.532614  1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 33% Complete (12 out of 36)

I0122 20:11:26.448179  1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 36% Complete (13 out of 36)

I0122 20:11:26.581250  1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 41% Complete (15 out of 36)

I0122 20:11:26.614269  1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 47% Complete (17 out of 36)

I0122 20:11:31.624903  1129 progress-updater.cc:55] Query 67692cad49234453:8cf581505134c8ca:50% Complete (18 out of 36)

I0122 20:11:31.693733  1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 52% Complete (19 out of 36)

I0122 20:11:36.556718  1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 55% Complete (20 out of 36)

I0122 20:11:36.770860  1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 58% Complete (21 out of 36)

I0122 20:11:36.770910  1560 coordinator.cc:904] Backend 0 completed,3 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:11:36.770932  1560 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.41:22000

I0122 20:11:36.829344  1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 63% Complete (23 out of 36)

I0122 20:11:37.189350  1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 69% Complete (25 out of 36)

I0122 20:11:37.189409  1129 coordinator.cc:904] Backend 3 completed,2 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:11:37.189440  1129 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.41:22000

I0122 20:11:37.485113  1119 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 77% Complete (28 out of 36)

I0122 20:11:42.486377  1119 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 83% Complete (30 out of 36)

I0122 20:11:46.854956  1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 86% Complete (31 out of 36)

I0122 20:11:49.901679  1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 91% Complete (33 out of 36)

I0122 20:11:49.901748  1127 coordinator.cc:904] Backend 1 completed,1 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:11:49.901774  1127 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.42:22000

I0122 20:12:03.582316  1119 progress-updater.cc:45] Query67692cad49234453:8cf581505134c8ca 100% Complete (36 out of 36)

I0122 20:12:03.582365  1119 coordinator.cc:904] Backend 2 completed,0 remaining: query_id=67692cad49234453:8cf581505134c8ca

I0122 20:12:03.589012 11899data-stream-mgr.cc:190(200)] DeregisterRecvr():fragment_id=67692cad49234453:8cf581505134c8ce, node=4

I0122 20:12:03.761831  1031 plan-fragment-executor.cc:335] Finishedexecuting fragment query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cb

I0122 20:12:03.763402  1031 plan-fragment-executor.cc:335] Finished executingfragment query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cb

I0122 20:12:03.763422  1031 coordinator.cc:481] Coordinator waitingfor backends to finish, 0 remaining

I0122 20:12:03.763437  1031 coordinator.cc:486] All backendsfinished or error.

生成查询过程分析文件

I0122 20:12:03.765897  1031 coordinator.cc:1027] Final profile forquery_id=67692cad49234453:8cf581505134c8ca

Query67692cad49234453:8cf581505134c8ca:(228ms 0.00%)

 Aggregate Profile:

协调器PlanFragment统计

 Coordinator Fragment:(1m58s 0.00%)

    - RowsProduced: 20

   CodeGen:

      - CodegenTime: 2ms

      - CompileTime: 55ms

      - LoadTime: 5ms

      - ModuleFileSize: 34.37 KB

   AGGREGATION_NODE (id=6):(1m58s 0.02%)

      - BuildBuckets: 1.02K

      - BuildTime: 180K clock cycles

      - GetResultsTime: 23K clock cycles

      - MemoryUsed: 33.44 KB

      - RowsReturned: 20

      - RowsReturnedRate: 0

   EXCHANGE_NODE (id=5):(1m58s 99.98%)

      - MemoryUsed: 0.00

      - RowsReturned: 80

      - RowsReturnedRate: 0

PlanFragment 2平均分析

 Averaged Fragment 2:(1s193ms 0.00%)

   completion times: min:2ms max:6s764ms  mean: 4s431ms  stddev:2s662ms

   execution rates: min:0.00 /sec max:6.23 MB/sec  mean:3.61MB/sec  stddev:2.31 MB/sec

   split sizes:  min: 0.00 , max:42.17 MB, avg: 21.54 MB, stddev: 14.93 MB

    - RowsProduced: 10.80K

   CodeGen:

      - CodegenTime: 440K clock cycles

      - CompileTime: 22ms

      - LoadTime: 3ms

      - ModuleFileSize: 34.37 KB

   HDFS_SCAN_NODE (id=1):(1s193ms 15.89%)

       - BytesRead: 21.54 MB

      - DelimiterParseTime: 193ms

      - MaterializeTupleTime: 3ms

      - MemoryUsed: 0.00

      - PerDiskReadThroughput: 6.67 MB/sec

      - RowsReturned: 10.80K

      - RowsReturnedRate: 7.93 K/sec

      - ScanRangesComplete: 1

      - ScannerThreadsReadTime: 2s621ms

      - TotalReadThroughput: 4.07 MB/sec

PlanFragment 1平均分析

 Averaged Fragment 1:(1m41s 0.00%)

   completion times: min:1m31s max:1m58s  mean: 1m41s  stddev:11s004ms

   execution rates: min:4.28 MB/sec max:5.01 MB/sec  mean:4.65MB/sec  stddev:353.17 KB/sec

   split sizes:  min: 447.30 MB, max:512.00 MB, avg: 468.49 MB, stddev: 25.49 MB

    - RowsProduced: 20

   CodeGen:

      - CodegenTime: 5ms

      - CompileTime: 114ms

      - LoadTime: 5ms

      - ModuleFileSize: 34.37 KB

   AGGREGATION_NODE (id=3):(1m41s 1.13%)

      - BuildBuckets: 1.02K

      - BuildTime: 1s043ms

      - GetResultsTime: 21K clock cycles

      - MemoryUsed: 33.44 KB

      - RowsReturned: 20

      - RowsReturnedRate: 0

   HASH_JOIN_NODE (id=2):(1m40s 1.75%)

      - BuildBuckets: 32.77K

      - BuildRows: 43.22K

      - BuildTime: 21ms

      - MemoryUsed: 0.00

      - ProbeRows: 3.75M

      - ProbeTime: 1s608ms

      - RowsReturned: 3.68M

      - RowsReturnedRate: 36.93 K/sec

     EXCHANGE_NODE (id=4):(6s737ms 7.37%)

        - MemoryUsed: 0.00

        - RowsReturned: 43.22K

        - RowsReturnedRate: 6.41 K/sec

   HDFS_SCAN_NODE (id=0):(1m31s 89.76%)

      - BytesRead: 468.49 MB

      - DelimiterParseTime: 4s923ms

      - MaterializeTupleTime: 458ms

      - MemoryUsed: 0.00

      - PerDiskReadThroughput: 5.26 MB/sec

      - RowsReturned: 3.75M

      - RowsReturnedRate: 41.12 K/sec

      - ScanRangesComplete: 8

      - ScannerThreadsReadTime: 1m29s

      - TotalReadThroughput: 4.62MB/sec

Plan Fragment 1分析

 Fragment 1:

Backend 0分析

   Instance 67692cad49234453:8cf581505134c8cc:(1m31s 0.00%)

     Hdfs split stats (:<# splits>/): 0:8/477.50M

      - RowsProduced: 20

     CodeGen:

        - CodegenTime: 5ms

        - CompileTime: 114ms

        - LoadTime: 6ms

        - ModuleFileSize: 34.37 KB

     AGGREGATION_NODE (id=3):(1m31s 1.13%)

        - BuildBuckets: 1.02K

        - BuildTime: 1s006ms

        - GetResultsTime: 21K clock cycles

        - MemoryUsed: 33.44 KB

        - RowsReturned: 20

        - RowsReturnedRate: 0

     HASH_JOIN_NODE (id=2):(1m30s 1.75%)

        - BuildBuckets: 32.77K

        - BuildRows: 43.22K

        - BuildTime: 21ms

        - MemoryUsed: 0.00

        - ProbeRows: 3.63M

        - ProbeTime: 1s555ms

        - RowsReturned: 3.61M

        - RowsReturnedRate: 39.97 K/sec

       EXCHANGE_NODE (id=4):(6s727ms 7.37%)

          - MemoryUsed: 0.00

          - RowsReturned: 43.22K

          - RowsReturnedRate: 6.42 K/sec

     HDFS_SCAN_NODE (id=0):(1m21s 89.76%)

         - BytesRead: 455.38 MB

        - DelimiterParseTime: 4s503ms

        - MaterializeTupleTime: 428ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 5.67 MB/sec

        - RowsReturned: 3.63M

        - RowsReturnedRate: 44.30 K/sec

         - ScanRangesComplete: 8

        - ScannerThreadsReadTime: 1m20s

        - TotalReadThroughput: 4.98 MB/sec

Backend 1分析

   Instance 67692cad49234453:8cf581505134c8cd:(1m44s 0.00%)

     Hdfs split stats (:<# splits>/): 0:8/469.03M

      - RowsProduced: 20

     CodeGen:

        - CodegenTime: 5ms

        - CompileTime: 114ms

        - LoadTime: 6ms

        - ModuleFileSize: 34.37 KB

     AGGREGATION_NODE (id=3):(1m44s 0.98%)

        - BuildBuckets: 1.02K

        - BuildTime: 994ms

        - GetResultsTime: 21K clock cycles

        - MemoryUsed: 33.44 KB

        - RowsReturned: 20

        - RowsReturnedRate: 0

     HASH_JOIN_NODE (id=2):(1m43s 1.52%)

        - BuildBuckets: 32.77K

        - BuildRows: 43.22K

        - BuildTime: 21ms

         - MemoryUsed: 0.00

        - ProbeRows: 3.59M

        - ProbeTime: 1s550ms

        - RowsReturned: 3.56M

        - RowsReturnedRate: 34.42 K/sec

       EXCHANGE_NODE (id=4):(6s739ms 6.45%)

          - MemoryUsed: 0.00

          - RowsReturned: 43.22K

          - RowsReturnedRate: 6.41 K/sec

     HDFS_SCAN_NODE (id=0):(1m35s 91.05%)

        - BytesRead: 447.30 MB

        - DelimiterParseTime: 4s519ms

        - MaterializeTupleTime: 462ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 4.90 MB/sec

        - RowsReturned: 3.59M

        - RowsReturnedRate: 37.79 K/sec

        - ScanRangesComplete: 8

        - ScannerThreadsReadTime: 1m31s

        - TotalReadThroughput: 4.20 MB/sec

Backend 2分析

   Instance 67692cad49234453:8cf581505134c8ce:(1m58s 0.00%)

     Hdfs split stats (:<# splits>/): 0:8/536.87M

      - RowsProduced: 20

     CodeGen:

        - CodegenTime: 5ms

        - CompileTime: 115ms

        - LoadTime: 2ms

        - ModuleFileSize: 34.37 KB

     AGGREGATION_NODE (id=3):(1m58s 1.00%)

        - BuildBuckets: 1.02K

        - BuildTime: 1s144ms

        - GetResultsTime: 21K clock cycles

        - MemoryUsed: 33.44 KB

        - RowsReturned: 20

        - RowsReturnedRate: 0

     HASH_JOIN_NODE (id=2):(1m56s 1.52%)

        - BuildBuckets: 32.77K

        - BuildRows: 43.22K

        - BuildTime: 22ms

        - MemoryUsed: 0.00

        - ProbeRows: 4.11M

        - ProbeTime: 1s750ms

        - RowsReturned: 4.06M

        - RowsReturnedRate: 34.71 K/sec

        EXCHANGE_NODE (id=4):(6s739ms 5.71%)

          - MemoryUsed: 0.00

          - RowsReturned: 43.22K

          - RowsReturnedRate: 6.41 K/sec

     HDFS_SCAN_NODE (id=0):(1m48s 91.78%)

        - BytesRead: 512.00 MB

        - DelimiterParseTime: 5s083ms

        - MaterializeTupleTime: 515ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 4.82 MB/sec

        - RowsReturned: 4.11M

        - RowsReturnedRate: 37.90 K/sec

        - ScanRangesComplete: 8

        - ScannerThreadsReadTime: 1m46s

        - TotalReadThroughput: 4.32 MB/sec

Backend 3分析

   Instance 67692cad49234453:8cf581505134c8cf:(1m31s 0.00%)

     Hdfs split stats (:<# splits>/): 0:8/481.61M

      - RowsProduced: 20

     CodeGen:

        - CodegenTime: 5ms

        - CompileTime: 114ms

        - LoadTime: 6ms

        - ModuleFileSize: 34.37 KB

     AGGREGATION_NODE (id=3):(1m31s 1.15%)

        - BuildBuckets: 1.02K

        - BuildTime: 1s030ms

        - GetResultsTime: 20K clock cycles

        - MemoryUsed: 33.44 KB

        - RowsReturned: 20

        - RowsReturnedRate: 0

     HASH_JOIN_NODE (id=2):(1m30s 1.77%)

        - BuildBuckets: 32.77K

        - BuildRows: 43.22K

        - BuildTime: 21ms

        - MemoryUsed: 0.00

        - ProbeRows: 3.66M

        - ProbeTime: 1s577ms

        - RowsReturned: 3.50M

         - RowsReturnedRate: 38.62 K/sec

       EXCHANGE_NODE (id=4):(6s741ms 7.35%)

          - MemoryUsed: 0.00

          - RowsReturned: 43.22K

          - RowsReturnedRate: 6.41 K/sec

     HDFS_SCAN_NODE (id=0):(1m22s 89.73%)

        - BytesRead: 459.30 MB

        - DelimiterParseTime: 5s587ms

        - MaterializeTupleTime: 426ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 5.64 MB/sec

        - RowsReturned: 3.66M

        - RowsReturnedRate: 44.49 K/sec

        - ScanRangesComplete: 8

        - ScannerThreadsReadTime: 1m21s

        - TotalReadThroughput: 4.99 MB/sec

Plan Fragment 2分析

 Fragment 2:

Backend 4分析

   Instance 67692cad49234453:8cf581505134c8d0:(11K clock cycles 84.11%)

      - RowsProduced: 0

     CodeGen:

        - CodegenTime: 358K clock cycles

        - CompileTime: 24ms

        - LoadTime: 2ms

        - ModuleFileSize: 34.37 KB

     HDFS_SCAN_NODE (id=1):(1K clock cycles 15.89%)

        - BytesRead: 0.00

        - MaterializeTupleTime: 0K clock cycles

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 0.00 /sec

        - RowsReturned: 0

        - RowsReturnedRate: 0

        - ScanRangesComplete: 0

        - ScannerThreadsReadTime: 0K clock cycles

        - TotalReadThroughput: 0.00 /sec

Backend 5分析

   Instance 67692cad49234453:8cf581505134c8d1:(1s026ms 0.01%)

     Hdfs split stats (:<# splits>/): 0:1/23.96M

      - RowsProduced: 11.35K

     CodeGen:

        - CodegenTime: 321K clock cycles

        - CompileTime: 21ms

        - LoadTime: 2ms

        - ModuleFileSize: 34.37 KB

     HDFS_SCAN_NODE (id=1):(1s026ms 99.99%)

        - BytesRead: 22.85 MB

        - DelimiterParseTime: 196ms

        - MaterializeTupleTime: 3ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 8.94 MB/sec

        - RowsReturned: 11.35K

        - RowsReturnedRate: 11.05 K/sec

        - ScanRangesComplete: 1

        - ScannerThreadsReadTime: 2s554ms

        - TotalReadThroughput: 6.53 MB/sec

Backend 6分析

   Instance 67692cad49234453:8cf581505134c8d2:(2s957ms 0.03%)

     Hdfs split stats (:<# splits>/): 0:2/44.22M

      - RowsProduced: 21.19K

     CodeGen:

        - CodegenTime: 766K clock cycles

        - CompileTime: 22ms

        - LoadTime: 6ms

        - ModuleFileSize: 34.37 KB

     HDFS_SCAN_NODE (id=1):(2s957ms 99.97%)

        - BytesRead: 42.17 MB

        - DelimiterParseTime: 358ms

        - MaterializeTupleTime: 6ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 7.09 MB/sec

        - RowsReturned: 21.19K

        - RowsReturnedRate: 7.17 K/sec

        - ScanRangesComplete: 2

        - ScannerThreadsReadTime: 5s948ms

        - TotalReadThroughput: 6.49 MB/sec

Backend 7分析

   Instance 67692cad49234453:8cf581505134c8d3:(790ms 0.03%)

     Hdfs split stats (:<# splits>/): 0:1/22.16M

      - RowsProduced: 10.67K

     CodeGen:

        - CodegenTime: 314K clock cycles

        - CompileTime: 21ms

        - LoadTime: 2ms

        - ModuleFileSize: 34.37 KB

     HDFS_SCAN_NODE (id=1):(790ms 99.97%)

        - BytesRead: 21.13 MB

        - DelimiterParseTime: 219ms

        - MaterializeTupleTime: 3ms

        - MemoryUsed: 0.00

        - PerDiskReadThroughput: 10.67 MB/sec

        - RowsReturned: 10.67K

        - RowsReturnedRate: 13.50 K/sec

        - ScanRangesComplete: 1

        - ScannerThreadsReadTime: 1s981ms

        - TotalReadThroughput: 3.25 MB/sec

查询处理完成，注销服务

I0122 20:12:03.767083  1031 impala-server.cc:1393] close():query_id=67692cad49234453:8cf581505134c8ca

I0122 20:12:03.767102  1031 impala-server.cc:1008]UnregisterQuery(): query_id=67692cad49234453:8cf581505134c8ca

I0122 20:12:03.770297  1031 data-stream-mgr.cc:190]DeregisterRecvr(): fragment_id=67692cad49234453:8cf581505134c8cb, node=5

处理节点1日志

本节点开始处理PlanFragment 1 Backend 0的任务

I0122 20:10:13.667968 26470impala-server.cc:1655] ExecPlanFragment()instance_id=67692cad49234453:8cf581505134c8cc coord=10.28.171.42:22000backend#=0

I0122 20:10:13.668203 26470plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cc

I0122 20:10:13.674814 26470plan-fragment-executor.cc:95] descriptor table forfragment=67692cad49234453:8cf581505134c8cc

I0122 20:10:13.797652  4113 plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8cc

本节点开始处理PlanFragment 2 Backend 4的任务

I0122 20:10:13.800835 26470impala-server.cc:1655] ExecPlanFragment()instance_id=67692cad49234453:8cf581505134c8d0 coord=10.28.171.42:22000 backend#=4

I0122 20:10:13.800878 26470plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8d0

I0122 20:10:13.803349 26470plan-fragment-executor.cc:95] descriptor table for fragment=67692cad49234453:8cf581505134c8d0

I0122 20:10:13.816920  4113 aggregation-node.cc:166]AggregationNode(node_id=3) using llvm codegend functions.

I0122 20:10:13.820927  4113 hash-join-node.cc:157]HashJoinNode(node_id=2) using llvm codegend function for building hash table.

I0122 20:10:13.828485  4118 plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8d0

I0122 20:10:13.830157  4113 hash-join-node.cc:170]HashJoinNode(node_id=2) using llvm codegend function for probing hash table.

查询任务全部处理完成，注销服务

I0122 20:11:45.129060  4113 data-stream-mgr.cc:190]DeregisterRecvr(): fragment_id=67692cad49234453:8cf581505134c8cc, node=4

处理节点2日志

本节点开始处理PlanFragment 1 Backend 1的任务

I0122 20:10:12.942119 22235impala-server.cc:1655] ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8cdcoord=10.28.171.42:22000 backend#=1

I0122 20:10:12.942455 22235plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cd

I0122 20:10:12.949337 22235plan-fragment-executor.cc:95] descriptor table forfragment=67692cad49234453:8cf581505134c8cd

I0122 20:10:13.072387 31215plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8cd

本节点开始处理PlanFragment 2 Backend 5的任务

I0122 20:10:13.073715 22235impala-server.cc:1655] ExecPlanFragment()instance_id=67692cad49234453:8cf581505134c8d1 coord=10.28.171.42:22000backend#=5

I0122 20:10:13.073760 22235plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8ca instance_id=67692cad49234453:8cf581505134c8d1

I0122 20:10:13.076200 22235plan-fragment-executor.cc:95] descriptor table forfragment=67692cad49234453:8cf581505134c8d1

I0122 20:10:13.091857 31215aggregation-node.cc:166] AggregationNode(node_id=3) using llvm codegendfunctions.

I0122 20:10:13.095860 31215hash-join-node.cc:157] HashJoinNode(node_id=2) using llvm codegend function forbuilding hash table.

I0122 20:10:13.098786 31218plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8d1

I0122 20:10:13.105170 31215hash-join-node.cc:170] HashJoinNode(node_id=2) using llvm codegend function forprobing hash table.

查询任务全部处理完成，注销服务

I0122 20:11:57.536412 31215data-stream-mgr.cc:190] DeregisterRecvr():fragment_id=67692cad49234453:8cf581505134c8cd, node=4

处理节点4日志

本节点开始处理PlanFragment 1 Backend 3的任务

I0122 20:10:06.177486  1448 impala-server.cc:1655]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8cfcoord=10.28.171.42:22000 backend#=3

I0122 20:10:06.177809  1448 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cf

I0122 20:10:06.184216  1448 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8cf

tuples:

I0122 20:10:06.306742 15883plan-fragment-executor.cc:194] Open(): instance_id=67692cad49234453:8cf581505134c8cf

本节点开始处理PlanFragment 2 Backend 7的任务

I0122 20:10:06.308400  1448 impala-server.cc:1655]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8d3coord=10.28.171.42:22000 backend#=7

I0122 20:10:06.308454  1448 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8d3

I0122 20:10:06.311002  1448 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8d3

I0122 20:10:06.326146 15883aggregation-node.cc:166] AggregationNode(node_id=3) using llvm codegendfunctions.

I0122 20:10:06.330163 15883hash-join-node.cc:157] HashJoinNode(node_id=2) using llvm codegend function forbuilding hash table.

I0122 20:10:06.333781 15885 plan-fragment-executor.cc:194]Open(): instance_id=67692cad49234453:8cf581505134c8d3

I0122 20:10:06.339367 15883hash-join-node.cc:170] HashJoinNode(node_id=2) using llvm codegend function forprobing hash table.

查询任务全部处理完成，注销服务

I0122 20:11:38.036321 15883data-stream-mgr.cc:190] DeregisterRecvr():fragment_id=67692cad49234453:8cf581505134c8cf, node=4

原创文章，转载请注明出处：http://blog.csdn.net/wind5shy/article/details/8563355