spark shuffle服務開啟

在送出了一個spark任務後，發現卡在了這一步，看意思是不打算配置設定資源給這個任務了。看字面意思是RM已經收到了請求，隻是不給它而已，就有兩種可能 一是資源不足，而是沒聯系上AM。

然後下面告訴我們壓根就沒發現nodemanager

在啟動Hadoop之後，發現nodemanager節點并未啟動起來，于是對Hadoop進行初始化并啟動，但發現啟動Hadoop的時候節點正常，nodemanager節點依然會挂掉。

2018-09-19 15:15:57,961 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:261)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:495)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:543)           

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2267)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2291)
... 10 more           

Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2171)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2265)
... 11 more           

這裡顯示是spark使用spark_shuffle時出現了問題，那好 下面就是解決辦法。

<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,mapreduce_shuffle</value>           

<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>           

然後把spark目錄下的yarn/spark-2.2.2-yarn-shuffle.jar hadoop/share/hadoop/yarn/lib/

然後重新開機就可以了。。。