某一个nodemanager退出后，导致 application master中出现大量的如下日志，并且持续很长时间，application master才成功退出。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<code>2016-06-24 09:32:35,596 INFO [ContainerLauncher </code><code>#3] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:35,596 INFO [ContainerLauncher </code><code>#9] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:35,597 INFO [ContainerLauncher </code><code>#7] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,455 INFO [ContainerLauncher </code><code>#8] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,539 INFO [ContainerLauncher </code><code>#5] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,539 INFO [ContainerLauncher </code><code>#1] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,539 INFO [ContainerLauncher </code><code>#6] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,539 INFO [ContainerLauncher </code><code>#2] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,539 INFO [ContainerLauncher </code><code>#0] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,596 INFO [ContainerLauncher </code><code>#4] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,597 INFO [ContainerLauncher </code><code>#3] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 09:32:36,597 INFO [ContainerLauncher </code><code>#9] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206/192.168.1.199:32951. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>.......</code>
<code>2016-06-24 12:57:52,328 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 8 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:57:53,339 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 9 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:04,357 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 0 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:05,367 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 1 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:06,378 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 2 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:07,392 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 3 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:08,399 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 4 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:09,408 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 5 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:10,417 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 6 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:11,425 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 7 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
<code>2016-06-24 12:58:12,434 INFO [Thread-1835] org.apache.hadoop.ipc.Client: Retrying connect to server: dchadoop206</code><code>/192</code><code>.168.1.199:32951. Already tried 8 </code><code>time</code><code>(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)</code>
1）dchadoop206上的nodemanager退出后（由于重启），导致application master持续的去连接之前nodemanager上的container。显然这些container是已经连接不上了。
2）最终经过非常长的时间大概3-4小时后，连接不上的异常才抛出，application master正常结束。
<code><strong style=</code><code>"font-family:'Source Sans Pro', 'Helvetica Neue', Helvetica, Arial, sans-serif;white-space:normal;color:rgb(51,51,51);font-size:1.4em;line-height:1.1;"</code><code>><span style=</code><code>"font-family:'黑体', SimHei;font-size:16px;"</code><code>>二、问题分析<</code><code>/span</code><code>><</code><code>/strong</code><code>><br></code>
这个问题主要涉及hadoop的rpc机制。首先看下面两个配置参数
<code> </code><code>#定义client连接到nodemanager的最大超时时间,不是单次连接，而是经过多少时间连接不上nodemanager，则认为操作失败</code>
<code> </code><code><property></code>
<code> </code><code><name>yarn.client.nodemanager-connect.max-wait-ms<</code><code>/name</code><code>></code>
<code> </code><code><value>15*60*1000<</code><code>/value</code><code>></code>
<code> </code><code><</code><code>/property</code><code>></code>
<code> </code><code># 定义每次尝试去连接nodemanager的时间间隔</code>
<code> </code><code><name>yarn.client.nodemanager-connect.retry-interval-ms<</code><code>/name</code><code>></code>
<code> </code><code><value>10*1000<</code><code>/value</code><code>></code>
根据这两个参数的定义，ApplicationMaster经过15分钟仍然连不上nodemanager的container，会取消try connect。但观察的情况是Application Master 需要等大约30分钟，才取消try connect。主要原因在于hadoop 的rpc机制如下，首先ApplicationMaster 会根据上面的两个参数，构造一个RetryUpToMaximumCountWithFixedSleep的重连策略，这个重连策略会通过以下方式计算
MaximumCount：yarn.client.nodemanager-connect.max-wait-ms/yarn.client.nodemanager-connect.retry-interval-ms=90次
而每次的RPC请求中，Client也有自己的重连策略，就是类似这样的东东:
<code>由两个rpc参数控制，ipc.client.connect.max.retries=10和ipc.client.connect.retry.interval=1000ms控制</code>
所以最终ApplicationMaster 放弃try connect的等待时间是：90*(10+10)=1800s
1）在提交map-reduce/hive sql/hive server2等客户端机器修改yarn-site.xml的以下参数
2）hadoop命令行中通过-D设置该参数
<code> </code><code><value>180000<</code><code>/value</code><code>></code>
这样总的等待时间就是6分钟。
这个修改是不需要做任何重启yarn组件操作的，是一个客户端相关的操作！
本文转自 zouqingyun 51CTO博客，原文链接：http://blog.51cto.com/zouqingyun/1881294，如需转载请自行联系原作者
![Hadoop之运行wordcount[图]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)