HDFS的shell操作(開發重點)
1.基本文法
bin/hadoop fs 具體指令 OR bin/hdfs dfs 具體指令
dfs是fs的實作類。
2.指令大全
bin/hadoop fs
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] <localsrc> ... <dst>]
[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-count [-q] <path> ...]
[-cp [-f] [-p] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] <path> ...]
[-expunge]
[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-getfacl [-R] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-d] [-h] [-R] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touchz <path> ...]
[-usage [cmd ...]]
3.常用指令實操
(0)啟動Hadoop叢集(友善後續的測試)
sbin/start-dfs.sh
sbin/start-yarn.sh
(1)-help:輸出這個指令參數
hadoop fs -help rm
(2)-ls: 顯示目錄資訊
hadoop fs -ls /
(3)-mkdir:在HDFS上建立目錄
hadoop fs -mkdir -p /sanguo/shuguo
(4)-moveFromLocal:從本地剪切粘貼到HDFS
touch kongming.txt
hadoop fs -moveFromLocal ./kongming.txt /sanguo/shuguo
(5)-appendToFile:追加一個檔案到已經存在的檔案末尾
touch liubei.txt
vi liubei.txt
輸入
san gu mao lu
hadoop fs -appendToFile liubei.txt /sanguo/shuguo/kongming.txt
(6)-cat:顯示檔案内容
hadoop fs -cat /sanguo/shuguo/kongming.txt
(7)-chgrp 、-chmod、-chown:Linux檔案系統中的用法一樣,修改檔案所屬權限
hadoop fs -chmod 666 /sanguo/shuguo/kongming.txt
hadoop fs -chown dev1:dev1 /sanguo/shuguo/kongming.txt
(8)-copyFromLocal:從本地檔案系統中拷貝檔案到HDFS路徑去
hadoop fs -copyFromLocal README.txt /
(9)-copyToLocal:從HDFS拷貝到本地
hadoop fs -copyToLocal /sanguo/shuguo/kongming.txt ./
(10)-cp :從HDFS的一個路徑拷貝到HDFS的另一個路徑
hadoop fs -cp /sanguo/shuguo/kongming.txt /zhuge.txt
(11)-mv:在HDFS目錄中移動檔案
hadoop fs -mv /zhuge.txt /sanguo/shuguo/
(12)-get:等同于copyToLocal,就是從HDFS下載下傳檔案到本地
hadoop fs -get /sanguo/shuguo/kongming.txt ./
(13)-getmerge:合并下載下傳多個檔案,比如HDFS的目錄 /user/dev1/test下有多個檔案:log.1, log.2,log.3,…
hadoop fs -getmerge /user/dev1/test/* ./zaiyiqi.txt
(14)-put:等同于copyFromLocal
hadoop fs -put ./zaiyiqi.txt /user/dev1/test/
(15)-tail:顯示一個檔案的末尾
hadoop fs -tail /sanguo/shuguo/kongming.txt
(16)-rm:删除檔案或檔案夾
hadoop fs -rm /user/dev1/test/jinlian2.txt
(17)-rmdir:删除空目錄
hadoop fs -mkdir /test
hadoop fs -rmdir /test
(18)-du統計檔案夾的大小資訊
hadoop fs -du -s -h /user/dev1/test
2.7 K /user/dev1/test
hadoop fs -du -h /user/dev1/test
1.3 K /user/dev1/test/README.txt
15 /user/dev1/test/jinlian.txt
1.4 K /user/dev1/test/zaiyiqi.txt
(19)-setrep:設定HDFS中檔案的副本數量
hadoop fs -setrep 10 /sanguo/shuguo/kongming.txt
HDFS副本數量如圖