http://trac.nchc.org.tw/cloud/wiki
Content 1. HDFS Shell基本操作
1.1 瀏覽你HDFS目錄
/opt/hadoop$ bin/hadoop fs -ls
1.2 上傳資料到HDFS目錄
/opt/hadoop$ bin/hadoop fs -put conf input
/opt/hadoop$ bin/hadoop fs -ls
/opt/hadoop$ bin/hadoop fs -ls input
1.3 下載HDFS的資料到本地目錄
/opt/hadoop$ bin/hadoop fs -get input fromHDFS
/opt/hadoop$ ls -al | grep fromHDFS
/opt/hadoop$ ls -al fromHDFS
1.4 刪除檔案
/opt/hadoop$ bin/hadoop fs -ls input
/opt/hadoop$ bin/hadoop fs -rm input/masters
1.5 直接看檔案
/opt/hadoop$ bin/hadoop fs -ls input
/opt/hadoop$ bin/hadoop fs -cat input/slaves
1.6 更多指令操作
[email protected]:/opt/hadoop$ bin/hadoop fs
Usage: java FsShell
[-ls <path>]
[-lsr <path>]
[-du <path>]
[-dus <path>]
[-count[-q] <path>]
[-mv <src> <dst>]
[-cp <src> <dst>]
[-rm <path>]
[-rmr <path>]
[-expunge]
[-put <localsrc> ... <dst>]
[-copyFromLocal <localsrc> ... <dst>]
[-moveFromLocal <localsrc> ... <dst>]
[-get [-ignoreCrc] [-crc] <src> <localdst>]
[-getmerge <src> <localdst> [addnl]]
[-cat <src>]
[-text <src>]
[-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
[-moveToLocal [-crc] <src> <localdst>]
[-mkdir <path>]
[-setrep [-R] [-w] <rep> <path/file>]
[-touchz <path>]
[-test -[ezd] <path>]
[-stat [format] <path>]
[-tail [-f] <file>]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-chgrp [-R] GROUP PATH...]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Content 2. 使用網頁Gui瀏覽資訊
- Map/Reduce Administration
- NameNode
Content 3. 更多HDFS shell 的用法
- bin/hadoop fs <args> ,下面則列出 <args> 的用法
- 以下操作預設的目錄在 /user/<$username>/ 下
$ bin/hadoop fs -ls input
Found 4 items
-rw-r--r-- 2 hadooper supergroup 115045564 2009-04-02 11:51 /user/hadooper/input/1.txt
-rw-r--r-- 2 hadooper supergroup 987864 2009-04-02 11:51 /user/hadooper/input/2.txt
-rw-r--r-- 2 hadooper supergroup 1573048 2009-04-02 11:51 /user/hadooper/input/3.txt
-rw-r--r-- 2 hadooper supergroup 25844527 2009-04-02 11:51 /user/hadooper/input/4.txt
- 完整的路徑則是 hdfs://node:port/path 如:
$ bin/hadoop fs -ls hdfs://gm1.nchc.org.tw:9000/user/hadooper/input
Found 4 items
-rw-r--r-- 2 hadooper supergroup 115045564 2009-04-02 11:51 /user/hadooper/input/1.txt
-rw-r--r-- 2 hadooper supergroup 987864 2009-04-02 11:51 /user/hadooper/input/2.txt
-rw-r--r-- 2 hadooper supergroup 1573048 2009-04-02 11:51 /user/hadooper/input/3.txt
-rw-r--r-- 2 hadooper supergroup 25844527 2009-04-02 11:51 /user/hadooper/input/4.txt
-cat
-chgrp
-chmod
-chown
-copyFromLocal, -put
-copyToLocal, -get
-cp
-du
-dus
-expunge
- 清空垃圾桶
$ bin/hadoop fs -expunge
-getmerge
-ls
- 列出文件或目錄的資訊
- 文件名 <副本數> 文件大小 修改日期 修改時間 權限 用戶ID 組ID
- 目錄名 <dir> 修改日期 修改時間 權限 用戶ID 組ID
$ bin/hadoop fs -ls
-lsr
- ls命令的遞迴版本
$ bin/hadoop fs -lsr /
-mkdir
-moveFromLocal
-mv
-rm
-rmr
- 遞迴刪除資料夾(包含在內的所有檔案)
$ bin/hadoop fs -rmr in1
-setrep
-stat
-tail
-test
- 測試檔案, -e 檢查文件是否存在(1=存在, 0=否), -z 檢查文件是否為空(1=空, 0=不為空), -d 檢查是否為目錄(1=存在, 0=否)
- 用法: bin/hadoop fs -test -[ezd] URI
$ bin/hadoop fs -test -e /user/hadooper/input/5.txt
$ bin/hadoop fs -test -z /user/hadooper/input/5.txt
test: File does not exist: /user/hadooper/input/5.txt
$ bin/hadoop fs -test -d /user/hadooper/input/5.txt
test: File does not exist: /user/hadooper/input/5.txt
-text
-touchz