天天看點

hadoop hdfs 添加資料目錄出錯

由于原來配置的hadoop data目錄快要用滿了,故準備修改配置檔案增加資料目錄,以便擴容,但由于疏忽,把core-site.xml, hdfs-site.xml配置檔案dfs.datanode.data.dir 配置項增加了配置目錄,但未建立實際目錄,重新開機datanode服務時,報如下錯誤:

2014-11-18 08:51:39,128 WARN org.apache.hadoop.hdfs.server.common.Util: Path /data1/cdh/data/hadoop-hdfs/data should be specified as a

URI in configuration files. Please update hdfs configuration.

2014-11-18 08:51:39,129 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool <registe

ring> (Datanode Uuid unassigned) service to namenode.hadoop/192.168.136.177:9000

org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 1, volumes configured: 2, volum

es failed: 1, volume failures tolerated: 0

at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.<init>(FsDatasetImpl.java:194)

at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)

at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)

at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:936)

at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:895)

at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:274)

at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:220)

at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:813)

at java.lang.Thread.run(Thread.java:745)

2014-11-18 08:51:39,129 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (

Datanode Uuid unassigned) service to namenode.hadoop/10.219.136.170:9000

2014-11-18 08:51:39,230 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with

NN

java.lang.Exception: trace

at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)

at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.remove(BlockPoolManager.java:91)

at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:854)

at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:350)

at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:617)

at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:835)

at java.lang.Thread.run(Thread.java:745)

?error

2014-11-18 08:51:06,567 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting

2014-11-18 08:51:06,568 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting

2014-11-18 08:51:06,851 INFO org.apache.hadoop.hdfs.server.common.Storage: Data-node version: -55 and name-node layout version: -55

2014-11-18 08:51:06,954 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled

2014-11-18 08:51:06,955 INFO org.apache.hadoop.hdfs.server.common.Storage: Restored 0 block files from trash.

2014-11-18 08:51:06,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null

2014-11-18 08:51:06,532 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default>

2014-11-18 08:51:06,567 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting

2014-11-18 08:51:06,568 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting

2014-11-18 08:51:06,851 INFO org.apache.hadoop.hdfs.server.common.Storage: Data-node version: -55 and name-node layout version: -55

2014-11-18 08:51:06,954 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled

2014-11-18 08:51:06,955 INFO org.apache.hadoop.hdfs.server.common.Storage: Restored 0 block files from trash.

at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.<init>(FsDatasetImpl.java:194)

at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:34)

at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetFactory.newInstance(FsDatasetFactory.java:30)

at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:936)

at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:895)

java.lang.Exception: trace

at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)

at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.remove(BlockPoolManager.java:91)

at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:854)

java.lang.Exception: trace

at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:143)

at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:856)

2014-11-18 08:51:09,077 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0

2014-11-18 08:51:09,108 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down DataNode at slave1/192.168.36.20

解決方法:修改hdfs-site.xml檔案中dfs.datanode.data.dir内容為如下配置方式

<property>

<name>dfs.datanode.data.dir</name>

<value>

file:///data/cdh/data/dfs/data,file:///data1/cdh/data/hadoop-hdfs/data

</value>

<final>true</final>

</property>

之前是後面添加的目錄配置方式沒有加上file://字首

,重新開機就OK

繼續閱讀