为Linux集群创建新账户，并配置hadoop集群
之前装python、jdk、hadoop都是用的root账户，这是一个绝对的失策，linux对用户访问权限很严格，新创的hod账户根本无法启动hadoop，而root在hod模式下使用torque是不被建议的，所以只能用hod账户再重新装一遍.
创建用户、设置密码、修改用户、删除用户：
useradd testuser 创建用户testuser
passwd testuser 给已创建的用户testuser设置密码
说明：新创建的用户会在/home下创建一个用户目录testuser
usermod --help 修改用户这个命令的相关参数
userdel testuser 删除用户testuser
rm -rf testuser 删除用户testuser所在目录
上面的几个命令只有root账号才可以使用，如果你不知道自己的系统上面的命令在什么位置可以使用如下命令查找其路径：locate useradd
在root账户下为服务器集群(x101~x156)创建新账户hod.
addhoduser.exp
#!/usr/bin/expect -f
set password 123456
for {set i 1} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
puts "$ip begins!"
spawn ssh $ip
expect "#"
send "useradd hod\r"
expect "#"
send "passwd hod\r"
expect "*New UNIX password*"
send "$password\r"
expect "*Retype new UNIX password*"
send "$password\r"
expect "#"
send "exit\r"
expect eof
puts "$ip end!"
}
close
在hod账户下进行如下的设置：
在每个服务器的hod目录下创建.ssh目录：（错！这部操作被证明是极大的错误，不用自己创建.ssh目录，使用“ssh-keygen -t rsa”会自动创建.ssh，自己创建.ssh目录会由于文件夹权限问题使得ssh无密码访问失效）
mkdirssh.exp(不需要使用)
#!/usr/bin/expect -f
set password 123456
for {set i 1} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
spawn ssh $ip
expect "*password:*"
send "$password\r"
expect eof
expect "#"
send "mkdir /home/hod/.ssh\r"
expect "#"
send "exit\r"
expect eof
}
close
在x101服务器的/home/hod/.ssh目录下创建authorized_keys文件，并设置为644权限，这很关键.(这部操作也就不必了，后面ssh-copy-id -i ~/.ssh/id_rsa.pub hod@x101会自动创建authorized_keys文件，且mod为600，这个权限也是可以的)
为x101~x156配置ssh无密码登录：
wumimadenglu.exp
#!/usr/bin/expect -f
set password 123456
for {set i 1} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
spawn ssh $ip
expect "*password:*"
send "$password\r"
expect eof
expect "#"
send "ssh-keygen -t rsa\r"
expect "*save the key*"
send "\r"
expect "*Enter passphrase*"
send "\r"
expect "*Enter same passphrase again*"
send "\r"
expect eof
expect "#"
send "ssh-copy-id -i ~/.ssh/id_rsa.pub hod@x101\r"
expect "*password:*"
send "$password\r"
expect "#"
send "exit\r"
expect eof
}
close
最后只要将x101下的authorized_keys文件拷贝到x102~x156的.ssh目录即可.这方法会比之前为root账户配置ssh无密码登录快很多很多.
copyauthorized_keystoall.exp
#!/usr/bin/expect -f
set password 123456
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
spawn scp /home/hod/.ssh/authorized_keys hod@$ip:/home/hod/.ssh/
expect "*password:*"
send "$password\r"
expect eof
}
close
这里需要说明一点，在进行scp的时候尽量使用/home/hod/.ssh/authorized_keys这样的绝对目录，使用~/.ssh/authorized_keys可能会由于找不到目录而出错
后续操作：
配置好ssh无密码登录之后，我在hod账户目录下安装了python和jdk，方法都是先拷贝安装包或压缩包再解压安装
copypythontoall.sh
for((i=2;i<57;i++))
do
if [ $i -lt 10 ]
then
scp /home/hod/Python-2.5.1.tgz hod@x10$i:/home/hod/
echo "it is the first if: x10$i"
else
scp /home/hod/Python-2.5.1.tgz hod@x1$i:/home/hod/
echo "it is the first else: x1$i"
fi
doneinstallpythontoall.exp
#!/usr/bin/expect -f
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
spawn ssh $ip
expect "#"
send "tar zxvf /home/hod/Python-2.5.1.tgz\r"
expect "#"
send "cd /home/hod/Python-2.5.1\r"
expect "#"
send "./configure\r"
expect "#"
send "make\r"
expect "#"
send "sudo make install\r"
expect "*Password:*"
send "123456\r"
expect "#"
send "exit\r"
expect eof
puts "$ip completed"
sleep 3
}
closecopyjdktoall.sh
for((i=2;i<57;i++))
do
if [ $i -lt 10 ]
then
scp /home/hod/jdk-7u1-linux-i586.tar.gz hod@x10$i:/home/hod/
echo "it is the first if: x10$i"
else
scp /home/hod/jdk-7u1-linux-i586.tar.gz hod@x1$i:/home/hod/
echo "it is the first else: x1$i"
fi
doneinstalljdktoall.exp
#!/usr/bin/expect -f
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
spawn ssh $ip
expect "#"
send "tar zxvf /home/hod/jdk-7u1-linux-i586.tar.gz\r"
expect "#"
send "exit\r"
expect eof
puts "$ip completed"
sleep 3
}
close
安装hadoop：
installhadooptoall.exp
#!/usr/bin/expect -f
set password 123456
for {set i 2} {$i<57} {incr i} {
if {$i<10} {
set ip x10$i
puts "it is the first if : $ip"
} else {
set ip x1$i
puts "it is the first else : $ip"
}
spawn scp /home/hod/hadoop-0.20.2.tar.gz hod@$ip:/home/hod/
spawn ssh $ip
expect "#"
send "mkdir /home/hod/hadoop\r"
expect "#"
send "cp /home/hod/hadoop-0.20.2.tar.gz /home/hod/hadoop/\r"
expect "#"
send "tar zxvf /home/hod/hadoop/hadoop-0.20.2.tar.gz\r"
expect "#"
send "exit\r"
expect eof
puts "$ip completed"
sleep 3
}
close
tar zxvf /home/hod/hadoop/hadoop-0.20.2.tar.gz会有问题，该操作会把hadoop压缩包解压在/home/hod目录下，应该在这之前先执行cd /home/hod/hadoop操作
在x101按之前root的操作配置好参数：
copypeizhitoall.sh
for((i=2;i<57;i++))
do
if [ $i -lt 10 ]
then
scp -r /home/hod/hadoop/hadoop-0.20.2/conf hod@x10$i:/home/hod/hadoop/hadoop-0.20.2/
scp /home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/hodrc hod@x10$i:/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/
echo "it is the first if: x10$i"
else
scp -r /home/hod/hadoop/hadoop-0.20.2/conf hod@x1$i:/home/hod/hadoop/hadoop-0.20.2/
scp /home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/hodrc hod@x1$i:/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf/
echo "it is the first else: x1$i"
fi
done最后用root用户在/etc/profile文件末尾加上下面语句，并把/etc/profile拷贝覆盖x102~x156
export JAVA_HOME=/home/hod/jdk1.7.0_01 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib:$CLASSPATH export HADOOP_HOME=/home/hod/hadoop/hadoop-0.20.2 export PATH=$HADOOP_HOME/bin:$PATH export CLUSTER_NAME=clus export RM_QUEUE=batch export RM_HOME=/usr/local export PYTHON_HOME=/home/hod/Python-2.5.1 export PATH=$PYTHON_HOME:$PATH export HOD_PYTHON_HOME=/home/hod/Python-2.5.1/python export HOD_CONF_DIR=/home/hod/hadoop/hadoop-0.20.2/contrib/hod/conf export HOD_HOME=/home/hod/hadoop/hadoop-0.20.2/contrib/hod export PATH=$HOD_HOME/bin:$PATH