《hadoop数据分析》之hadoop命令
先建个文件夹
(base) localhost:~ XXX$ hadoop fs -mkdir /corpora
2020-01-13 09:47:44,788 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
将”文稿"中的一个txt上传到corpora文件夹中
(base) localhost:~ XXX$ hadoop fs -put`
/Users/XXX/Documents/hadoop_file/shakespeare.txt /corpora/shakespeare.txt
2020-01-13 09:51:45,525 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 09:51:46,827 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
看一下里面的文件夹
(base) localhost:~ XXX$ hadoop fs -ls /
2020-01-13 09:54:25,587 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
drwxr-xr-x - XXX supergroup 0 2020-01-13 09:51 /corpora
drwxr-xr-x - XXX supergroup 0 2020-01-10 17:47 /input
drwxr-xr-x - XXX supergroup 0 2020-01-10 16:23 /output
drwxr-xr-x - XXX supergroup 0 2020-01-10 15:42 /wordcount
用copyFromLocal试一试
(base) localhost:~ XXX$ hadoop fs -copyFromLocal /Users/XXX/Documents/hadoop_file/shakespeare.txt /corpora/shakespeare_copy.txt
2020-01-13 09:56:12,953 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 09:56:14,219 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
在看这个路径：/Users/XXX/Documents/hadoop_file/shakespeare.txt 下面看哈还有没有txt文档
确实已经没有了。
查看txt的内容：
(base) localhost:/ XXX$ hadoop fs -cat /corpora/shakespeare.txt
2020-01-13 10:07:54,912 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 10:07:56,663 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
还可以通过管道传递到less
(base) localhost:/ XXX$ hadoop fs -cat /corpora/shakespeare.txt | less
就会出现下面的界面，输入q以返回到终端
输出后1000个字节，用tail
(base) localhost:/ XXX$ hadoop fs -tail /corpora/shakespeare.txt|less
没有类似的hadoop fs -head，命令来检查文件的前1000字节，不过可以使用hadoop fs -cat并通过管道将文件内容传输到本地shell的head命令
(base) localhost:/ XXX$ hadoop fs -cat /corpora/shakespeare.txt|head
2020-01-13 10:33:12,645 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 10:33:13,865 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
hamlet@0 HAMLET
hamlet@8
hamlet@9
hamlet@10 DRAMATIS PERSONAE
hamlet@29
hamlet@30
hamlet@31 CLAUDIUS king of Denmark. (KING CLAUDIUS:)
hamlet@74
hamlet@75 HAMLET son to the late, and nephew to the present king.
hamlet@131
若想将整个文件从分布式文件系统传输到本地文件系统，可以使用get或copyToLocal，这两个命令是一样的，与之类似的，也可以使用moveToLocal命令，它会将文件从分布式文件系统中删除。get merge命令复制符合给定模式或指定目录下的所有文件，并将其合并成本机的单个文件。如果远程系统上的文件较大，可以在管道传输的时候使用压缩工具：
试了好多次，看正确的格式，本来想着是放到“下载”里面的某个文件夹，因为没有建，直接在命令行中写都不行。
(base) localhost:/ XXX$ hadoop fs -get /corpora/shakespeare.txt /Users/XXX/Downloads/from-remote.txt
2020-01-13 10:54:48,801 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 10:54:50,080 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
使用copyToLocal试试
(base) localhost:/ XXX$ hadoop fs -copyToLocal /corpora/shakespeare.txt /Users/XXX/Downloads/from-remote_copy.txt
2020-01-13 11:00:47,776 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 11:00:49,171 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
使用moveToLocal试试
(base) localhost:/ XXX$ hadoop fs -moveToLocal /corpora/shakespeare_move.txt /Users/XXX/Downloads/from-remote_move.txt
2020-01-13 11:07:00,147 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
moveToLocal: Option '-moveToLocal' is not implemented yet
.#还没有实现？？？没有移动成功
看哈合并的getmerge
(base) localhost:/ XXX$ hadoop fs -getmerge /corpora /Users/XXX/Downloads/local_file.txt
2020-01-13 11:13:44,248 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-01-13 11:13:45,890 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
细水长流～