Hadoop wordcount報錯解決方案

文章目錄

• • `Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild`
  • `ERROR:Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster`
  • `其他`

Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild

解決方案：

• 1, 在指令行執行，複制資訊。冒号用逗号代替

hadoop classpath
           

• 2 , 編輯yarn-site.xml，添加資訊：

  </configuration>
  <property>
  	<name>namemapreduce.reduce.env/name</name>										
  	<value>valueHADOOP_MAPRED_HOME=/opt/module/hadoop-		3.1.3/value</value>
  </property>
  
    <property>
      <name>yarn.application.classpath</name>
   
      <value>複制的Hadoop classpath資訊,如
      		/opt/hadoop-3.1.0/etc/hadoop,
              /opt/hadoop-3.1.0/share/hadoop/common/*,
              /opt/hadoop-3.1.0/share/hadoop/common/lib/*,
              /opt/hadoop-3.1.0/share/hadoop/hdfs/*,
              /opt/hadoop-3.1.0/share/hadoop/hdfs/lib/*,
              /opt/hadoop-3.1.0/share/hadoop/mapreduce/*,
              /opt/hadoop-3.1.0/share/hadoop/mapreduce/lib/*,
              /opt/hadoop-3.1.0/share/hadoop/yarn/*,
              /opt/hadoop-3.1.0/share/hadoop/yarn/lib/*
       </value>
    </property>
  </configuration>
             

  設定完畢後用xsync指令分發到其他伺服器上并使用

  start-all.sh

  指令重新開機Hadoop叢集，

  成功運作。

ERROR:Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

解決方案：

• 1，編輯mapred-site.xml，添加以下資訊

  <property>
    <name>yarn.app.mapreduce.am.env</name>
    <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
  </property>
  <property>
    <name>mapreduce.map.env</name>
    <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
  </property>
  <property>
    <name>mapreduce.reduce.env</name>
    <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
  </property>
             

• 2， 設定完畢後用xsync指令分發到其他伺服器上并使用

  start-all.sh

  指令重新開機Hadoop叢集，

  成功運作。

其他

如果仍舊報錯，請參考可成功運作的配置：

mapred-site.xml配置如下

<configuration>

<!-- 指定 MapReduce 程式運作在 Yarn 上 -->
<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>

<property>
	<name>namemapreduce.reduce.env/name</name>
	<value>valueHADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3/value</value>

</property>
<property>
 	 <name>yarn.app.mapreduce.am.env</name>
  	<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
  	<name>mapreduce.map.env</name>
  	<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
  	<name>mapreduce.reduce.env</name>
  	<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>

</configuration>

           

yarn-site.xml 配置如下

<configuration>


<!-- 指定 MR 走 shuffle -->
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>mapreduce_shuffle</value>
</property>
<!-- 指定 ResourceManager 的位址-->
<property>
	<name>yarn.resourcemanager.hostname</name>
	<value>hadoop103</value>
</property>
<!-- 環境變量的繼承 -->
<property>
	<name>yarn.nodemanager.env-whitelist</name>
	<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CO
NF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAP
RED_HOME</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>


<property>
        <name>yarn.application.classpath</name>
        <value>
        /opt/module/hadoop-3.1.3/etc/hadoop,/opt/module/hadoop-3.1.3/share/hadoop/common/lib/*,/opt/module/hadoop-3.1.3/share/hadoop/common/*,/opt/module/hadoop-3.1.3/share/hadoop/hdfs,/opt/module/hadoop-3.1.3/share/hadoop/hdfs/lib/*,/opt/module/hadoop-3.1.3/share/hadoop/hdfs/*,/opt/module/hadoop-3.1.3/share/hadoop/mapreduce/lib/*,/opt/module/hadoop-3.1.3/share/hadoop/mapreduce/*,/opt/module/hadoop-3.1.3/share/hadoop/yarn,/opt/module/hadoop-3.1.3/share/hadoop/yarn/lib/*,/opt/module/hadoop-3.1.3/share/hadoop/yarn/*
   </value>
    </property>
</configuration>

           

運作成功截圖：