HADOOP学习_熟悉linux命令+启动hadoop
hadoop学习
- linux命令
-
- mkdir rmdir
- cp
- mv
- rm
- 查看文档内容
- 新建文件
- 找文件
-
- brew和tree的安装方法
- find
- locate
- grep 匹配字符串
- 启动hadoop
linux命令
可以用man查看命令的详细用法
mkdir rmdir
cd ~/desktop
mkdir ./test
mkdir(rmdir)路径必须要一个一个的加，否则会出错
mac@kkkMac desktop % mkdir ./test/a1/a2
mkdir: ./test/a1: No such file or directory
mac@kkkMac desktop % mkdir ./test/a1
mac@kkkMac desktop % mkdir ./test/a1/a2
mac@kkkMac desktop % rmdir ./test/a1/a2
mac@kkkMac desktop % rmdir ./test/a1
cp
test和hadoop是desktop下的两个文件
mac@kkkMac desktop % cp ./test ./hadoop
cp: ./test is a directory (not copied).
发生错误，移动目录的时候加参数 -R
cp -R ./test ./hadoop
如果写成./hadoop /test1 就会在复制的时候自动新建一个test1的目录，然后把test放在test1下面
mv
JAVA是desktop下的文件，-i表示提示
mv -i ./hadoop/test ./JAVA
可以查看一下mv的用法
man mv
EXAMPLES
Rename file foo to bar, overwriting bar if it already exists:
$ mv -f foo bar
COMPATIBILITY
The -n and -v options are non-standard and their use in scripts is not
recommended.
The mv utility now supports HFS+ Finder and Extended Attributes and resource
forks. The mv utility will no longer strip resource forks off of HFS files.
For an alternative method, refer to cp(1).
SEE ALSO
cp(1), rm(1), symlink(7)
STANDARDS
The mv utility is expected to be IEEE Std 1003.2 (“POSIX.2”) compatible.
HISTORY
A mv command appeared in Version 1 AT&T UNIX.
macOS 12.2 March 15, 2013 macOS 12.2
(END)
按q退出查看
rm
mac@kkkMac desktop % rm ./JAVA/test
rm: ./JAVA/test: is a directory
mac@kkkMac desktop % man rm
mac@kkkMac desktop % rm -R ./JAVA/test
通过man mv需要加-R才能移动目录
查看文档内容
查看文件内容除了cat还可以用more
[mac@kkkMac desktop % more ./test/*.txt
this is a test line
A
B
Sd
A
Sd
Fa
F
Asdas
As
As
Das
Asd
Sda
Fg
sdh
Fg
Hd
查看前两行的
mac@kkkMac desktop % head -2 ./test/*.txt
this is a test line
A
查看后两行
mac@kkkMac desktop % tail -2 ./test/*.txt
Fed
Dosa
新建文件
mac@kkkMac desktop % touch ./test/file1.txt
mac@kkkMac desktop % ls ./test
file1.txt testfile.txt
or echo(创建文件时写入东西)
mac@kkkMac desktop % echo ./test/file2.txt
./test/file2.txt
mac@kkkMac desktop % ls ./test
file1.txt testfile.txt
mac@kkkMac desktop % echo 'this is a test line' > ./test/file2.txt
mac@kkkMac desktop % ls ./test
file1.txt file2.txt testfile.txt
找文件
brew和tree的安装方法
这里插入一段安装brew命令的方法(突发奇想想用tree查看树目录)
https://blog.csdn.net/Darling_qi/article/details/122239824
mac@kkkMac desktop % /bin/zsh -c "$(curl -fsSL https://gitee.com/cunkai/HomebrewCN/raw/master/Homebrew.sh)"
开始执行Brew自动安装程序
[[email protected]]
['2022-05-02 10:35:55']['12.2']
https://zhuanlan.zhihu.com/p/111014448
请选择一个下载brew本体的序号，例如中科大，输入1回车。
源有时候不稳定，如果git克隆报错重新运行脚本选择源。
1、中科大下载源
2、清华大学下载源
3、北京外国语大学下载源
4、腾讯下载源
5、阿里巴巴下载源
请输入序号: 1
你选择了中国科学技术大学brew本体下载源
！！！此脚本将要删除之前的brew(包括它下载的软件)，请自行备份。
->是否现在开始执行脚本（N/Y） y
--> 脚本开始执行
Mac os设置开机密码方法：
(设置开机密码：在左上角苹果图标->系统偏好设置->用户与群组->更改密码)
(如果提示This incident will be reported. 在用户与群组中查看是否管理员)
==> 通过命令删除之前的brew、创建一个新的Homebrew文件夹
请输入开机密码，输入过程不显示，输入完后回车
Password:
开始执行
---备份要删除的/usr/local/Homebrew到系统桌面....
---/usr/local/Homebrew 备份完成
-> 创建文件夹 /usr/local/Homebrew
运行代码 ==> /usr/bin/sudo /bin/mkdir -p /usr/local/Homebrew
此步骤成功
运行代码 ==> /usr/bin/sudo /bin/chmod -R a+rwx /usr/local/Homebrew
运行代码 ==> /usr/bin/sudo /usr/sbin/chown mac /usr/local/Homebrew
运行代码 ==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/Homebrew
---备份要删除的/Users/mac/Library/Caches/Homebrew到系统桌面....
---/Users/mac/Library/Caches/Homebrew 备份完成
git version 2.32.0 (Apple Git-132)
下载速度觉得慢可以ctrl+c或control+c重新运行脚本选择下载源
==> 从 https://mirrors.ustc.edu.cn/brew.git 克隆Homebrew基本文件
未发现Git代理（属于正常状态）
Cloning into '/usr/local/Homebrew'...
remote: Enumerating objects: 214404, done.
remote: Total 214404 (delta 0), reused 0 (delta 0), pack-reused 214404
Receiving objects: 100% (214404/214404), 56.95 MiB | 10.07 MiB/s, done.
Resolving deltas: 100% (159236/159236), done.
此步骤成功
--创建Brew所需要的目录
运行代码 ==> /usr/bin/sudo /usr/sbin/chown -R mac:admin /usr/local/Homebrew
运行代码 ==> /usr/bin/sudo /bin/mkdir -p /Users/mac/Library/Caches/Homebrew
运行代码 ==> /usr/bin/sudo /bin/chmod g+rwx /Users/mac/Library/Caches/Homebrew
运行代码 ==> /usr/bin/sudo /usr/sbin/chown -R mac /Users/mac/Library/Caches/Homebrew
--依赖目录脚本运行完成
==> 创建brew的替身
==> 从 https://mirrors.ustc.edu.cn/homebrew-core.git 克隆Homebrew Core
此处如果显示Password表示需要再次输入开机密码，输入完后回车
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core'...
remote: Enumerating objects: 1185470, done.
remote: Total 1185470 (delta 0), reused 0 (delta 0), pack-reused 1185470
Receiving objects: 100% (1185470/1185470), 519.60 MiB | 9.02 MiB/s, done.
Resolving deltas: 100% (828274/828274), done.
Updating files: 100% (6374/6374), done.
此步骤成功
==> 从 https://mirrors.ustc.edu.cn/homebrew-cask.git 克隆Homebrew Cask 图形化软件
此处如果显示Password表示需要再次输入开机密码，输入完后回车
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask'...
remote: Enumerating objects: 687356, done.
remote: Total 687356 (delta 0), reused 0 (delta 0), pack-reused 687356
Receiving objects: 100% (687356/687356), 342.21 MiB | 10.54 MiB/s, done.
Resolving deltas: 100% (492425/492425), done.
此步骤成功
==> 从 https://gitee.com/cunkai/homebrew-services.git 克隆Homebrew services 管理服务的启停
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-services'...
remote: Enumerating objects: 1954, done.
remote: Counting objects: 100% (813/813), done.
remote: Compressing objects: 100% (401/401), done.
remote: Total 1954 (delta 401), reused 802 (delta 392), pack-reused 1141
Receiving objects: 100% (1954/1954), 535.07 KiB | 391.00 KiB/s, done.
Resolving deltas: 100% (885/885), done.
此步骤成功
==> 配置国内镜像源HOMEBREW BOTTLE
sed: /Users/mac/.zprofile: No such file or directory
Brew本体已经安装成功，接下来配置国内源。
请选择今后brew install的时候访问那个国内镜像，例如阿里巴巴，输入5回车。
1、中科大国内源
2、清华大学国内源
3、北京外国语大学国内源
4、腾讯国内源
5、阿里巴巴国内源
请输入序号: 1
你选择了中国科学技术大学国内源
环境变量写入->/Users/mac/.zprofile
此步骤成功
运行代码 ==> /usr/bin/sudo /bin/chmod -R a+rwx /usr/local/Homebrew
运行代码 ==> /usr/bin/sudo /usr/sbin/chown mac /usr/local/Homebrew
运行代码 ==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/Homebrew
==> 安装完成，brew版本
Homebrew 3.4.9-33-g3088bbf-dirty
Homebrew/homebrew-core (git revision 23f83804493; last commit 2022-05-02)
Homebrew/homebrew-cask (git revision 947ffd93cb; last commit 2022-05-02)
Brew前期配置成功
电脑系统版本：12.2
All user-space services OK, nothing cleaned...
==> brew update-reset
==> Fetching /usr/local/Homebrew...
==> Resetting /usr/local/Homebrew...
Updating files: 100% (2580/2580), done.
Branch 'master' set up to track remote branch 'master' from 'origin'.
Reset branch 'master'
Your branch is up to date with 'origin/master'.
==> Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask...
==> Resetting /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask...
Updating files: 100% (4054/4054), done.
Branch 'master' set up to track remote branch 'master' from 'origin'.
Reset branch 'master'
Your branch is up to date with 'origin/master'.
==> Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core...
==> Resetting /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core...
Updating files: 100% (6093/6093), done.
Branch 'master' set up to track remote branch 'master' from 'origin'.
Reset branch 'master'
Your branch is up to date with 'origin/master'.
==> Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-services...
==> Resetting /usr/local/Homebrew/Library/Taps/homebrew/homebrew-services...
Branch 'master' set up to track remote branch 'master' from 'origin'.
Reset branch 'master'
Your branch is up to date with 'origin/master'.
Brew自动安装程序运行完成
国内地址已经配置完成
桌面的Old_Homebrew文件夹，大致看看没有你需要的可以删除。
初步介绍几个brew命令
本地软件库列表：brew ls
查找软件：brew search google（其中google替换为要查找的关键字）
查看brew版本：brew -v 更新brew版本：brew update
安装cask软件：brew install --cask firefox 把firefox换成你要安装的
欢迎右键点击下方地址-打开URL 来给点个赞
https://zhuanlan.zhihu.com/p/111014448
安装成功 但还需要重启终端 或者 运行 source /Users/mac/.zprofile 否则可能无法使用
mac@kkkMac desktop % source /Users/mac/.zprofile
mac@kkkMac desktop % brew install tree
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/tree-2.0.2.monterey.bottle.tar.gz
######################################################################## 100.0%
==> Pouring tree-2.0.2.monterey.bottle.tar.gz
/usr/local/Cellar/tree/2.0.2: 8 files, 151.0KB
==> Running `brew cleanup tree`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
mac@kkkMac desktop % tree ./test
./test
├── file1.txt
├── file2.txt
└── testfile.txt
0 directories, 3 files
mac@kkkMac desktop %
find
mac@kkkMac desktop % find ./test/*.txt
./test/file1.txt
./test/file2.txt
./test/testfile.txt
locate
提示需要新建一个database
mac@kkkMac desktop % locate txt
WARNING: The locate database (/var/db/locate.database) does not exist.
To create the database, run the following command:
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist
Please be aware that the database can take some time to generate; once
the database has been created, this message will no longer appear.
grep 匹配字符串
grep是把管道符前面的输出作为后面的输入
mac@kkkMac desktop % cat ./test/testfile.txt|grep As
Asdas
As
As
Asd
启动hadoop
可以先查看一下环境配置
mac@kkkMac sbin % cat ~/.bash_profile
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home
PATH=$JAVA_HOME/bin:$PATH:.
CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:.
export JAVA_HOME
export PATH
export CLASSPATH
export HADOOP_HOME=/Users/mac/Downloads/hadoop-2.7.3
export PATH=$PATH:$HADOOP_HOME/bin
export HBASE_HOME=/Users/mac/Downloads/hbase-1.4.13
export PATH=$PATH:$HBASE_HOME/bin
export MONGODB_HOME=/Users/mac/Downloads/mongodb-macos-x86_64-4.4.6
export PATH=$PATH:$MONGODB_HOME/bin
export PATH=$PATH:/usr/local/mysql/bin
alias python="/usr/local/bin/python3"
export PATH=$PATH:/usr/local/bin/python3
alias pip="/usr/local/bin/python3"
然后再source使其生效
mac@kkkMac ~ % source ~/.bash_profile
到启动目录下
cd $HADOOP_HOME/sbin
查看目录
mac@kkkMac sbin % ls
distribute-exclude.sh hdfs-config.sh refresh-namenodes.sh start-balancer.sh start-yarn.cmd stop-balancer.sh stop-yarn.cmd
hadoop-daemon.sh httpfs.sh slaves.sh start-dfs.cmd start-yarn.sh stop-dfs.cmd stop-yarn.sh
hadoop-daemons.sh kms.sh start-all.cmd start-dfs.sh stop-all.cmd stop-dfs.sh yarn-daemon.sh
hdfs-config.cmd mr-jobhistory-daemon.sh start-all.sh start-secure-dns.sh stop-all.sh stop-secure-dns.sh yarn-daemons.sh
可以选择启动哪几个，不确定的就start-all.sh
jps查看是否启动成功
发现dataNode没有启动成功,我们先stop-all.sh。
用finder找到该文件查看
这是格式化后namenode中保留的clusterID
这是datanode中保留的clusterID
发现两个id是不一样的，手动将namenode的id复制到datanodeid中去。然后仔重新启动，就发现dataNode可以成功启动了。
mac@kkkMac hadoop-2.7.3 % jps
19539 NodeManager
19139 NameNode
19336 SecondaryNameNode
19225 DataNode
19449 ResourceManager
19578 Jps
//如果觉得这个方法太麻烦了，就把dfs路径下的那几个文件清空，不是删除哈
然后再重新格式化：
hadoop namenode -format
启动成功，下次就不用格式化，直接启动就可以了。
mac@kkkMac hadoop-2.7.3 % jps
20563 NodeManager
20361 SecondaryNameNode
20602 Jps
20474 ResourceManager
20172 NameNode
20255 DataNode
而且，路径下的这几个文件会生成相应的文件
