環境:剛搭好的CDH5.13.1,叢集5台機器,HIVE底層MR。
hive指令行輸入select count(*)或者insert into 都會報以下錯誤:
日志描述
![](https://img.laitimes.com/img/__Qf2AjLwojIjJCLyojI0JCLiAzNfRHLGZkRGZkRfJ3bs92YsYTMfVmepNHLrVzVhNTNXplc1cFZ5JlMMBjVtJWd0ckW65UbM5WOHJWa5kHT20ESjBjUIF2X0hXZ0xCMx81dvRWYoNHLrdEZwZ1Rh5WNXp1bwNjW1ZUba9VZwlHdssmch1mclRXY39CXldWYtlWPzNXZj9mcw1ycz9WL49zZuBnLzkTO1IjMxkDMxIzMwkTMwIzLc52YucWbp5GZzNmLn9Gbi1yZtl2Lc9CX6MHc0RHaiojIsJye.png)
以下是application日志:http://hadoop01:8088
Jobhistory日志:http://hadoop01:19888/jobhistory
檢視yarn log(指令行輸入如下):
#yarn logs -applicationId application_1552964743608_0001 > logs_20190319.txt
更改過的配置:
發現報錯後,上網各種百度谷歌,改了很多配置,大部分都失敗了,但是中間改過一次mapred-site.xml的address,後來又改回去了反而成功了,是以估計是中間某步改成功了,還好改過的基本都記錄了,主要修改過以下幾個配置檔案,特此記錄一下,也供大家參考。
1.yarn-site.xml
2.mapred-site.xml
3./etc/profile
4.hive-site.xml
1.改yarn-site.xml
配置檔案:
#find / -name yarn-site.xml
查到有以下五個比較像,但不确定是哪個,故各項都同時修改以下五個:
1)/etc/hadoop/conf.cloudera.yarn/yarn-site.xml
2)/etc/hive/conf.cloudera.hive/yarn-site.xml
3)/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/yarn-site.xml
4)/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/hadoop/conf.empty/yarn-site.xml
5)/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/hadoop/conf.pseudo/yarn-site.xml
修改參數:
1)增加yarn.nodemanager.resource.memory-mb,值改為:14510
2)yarn.application.classpath,值改為:
3)yarn.scheduler.minimum-allocation-mb,值改為:2048
4)增加:yarn.nodemanager.aux-services,值改為:mapreduce_shuffle
5)增加:yarn.log-aggregation-enable,值改為:true
2.改mapred-site.xml
配置檔案:
#find / -name mapred-site.xml
同樣的,同時修改以下三個檔案的各參數:
1)/etc/hadoop/conf.cloudera.yarn/mapred-site.xml
2)/etc/spark/conf.cloudera.spark_on_yarn/yarn-conf/mapred-site.xml
3)/etc/hive/conf.cloudera.hive/mapred-site.xml
修改參數:
1)修改mapreduce.map.memory.mb,值改為:2048
2)修改mapreduce.reduce.memory.mb,值改為:2048
3)mapreduce.jobhistory.address,值改為:hadoop01:10020
4)mapreduce.jobhistory.webapp.addres,值改為:hadoop01:19888
5)mapreduce.application.classpath,值改為:
3.改/etc/profile
增加HADOOP_HOME,修改PATH增加HADOOP_HOME
修改如下:
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/hadoop
增加export PATH= P A T H : PATH: PATH:HADOOP_HOME/bin:$HADOOP_HOME/sbin
4.改hive-site.xml
配置檔案:
1)/etc/hive/conf.cloudera.hive/hive-site.xml
2)/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/hive/conf.dist/hive-site.xml
修改參數:
1)hive.insert.into.multilevel.dirs,值改為:true