[转]Setting Up an MPICH2 Cluster in Ubuntu 8.04 LTS Server (Quick-Note)

MpichCluster

Setting Up an MPICH2 Cluster in Ubuntu 8.04 LTS Server (Quick-Note)
(https://wiki.ubuntu.com/MpichCluster )

===========================
1. 更改 /etc/hosts，方便遠端直接快速連結。
===========================

假設有兩個 mpi node，所有 node 的 /etc/hosts 都需修改

master@mpi-1:~$ sudo vi /etc/hosts
master@mpi-2:~$ sudo vi /etc/hosts

10.211.55.5 mpi-1
10.211.55.6 mpi-2


=================================
2. 安裝及設定 NFS，讓各 node 有共享的 master folder。
=================================

master@mpi-1:~$ sudo apt-get install nfs-kernel-server

(在此以 mpi-1 主機作為 master)

設定 NFS 目錄與掛載權限

master@mpi-1:~$ sudo mkdir /mpi
master@mpi-1:~$ sudo vi /etc/exports

/mpi *(rw,sync)

master@mpi-1:~$ sudo chown master:master /mpi
master@mpi-1:~$ sudo /etc/init.d/nfs-kernel-server restart

在其他 node 掛載 mpi-1 master folder

master@mpi-2:~$ sudo mkdir /mpi
master@mpi-2:~$ sudo chown master:master /mpi
master@mpi-2:~$ sudo apt-get install nfs-common
master@mpi-2:~$ sudo mount -t nfs mpi-1:/mpi /mpi

注意：sudo mount -t nfs时一定要写绝对路径，而不能写相对路径，如~/Desktop应该写成/home/XX/Desktop

 

==========================================
3. 安裝 ssh，設定金鑰讓各 node 之間以 master 帳號連結不用輸入密碼。
==========================================

(主要是為了方便往後可在 scripts 裡設定各 node 自動串連)

master@mpi-1:~$ sudo apt-get install ssh
master@mpi-2:~$ sudo apt-get install ssh

master@mpi-1:~$ ssh­-keygen ­-t dsa
master@mpi-1:~$ scp .ssh/id_dsa.pub master@mpi-2:~/mpi-1.pub
master@mpi-2:~$ cat mpi-1.pub >> .ssh/authorized_keys

master@mpi-2:~$ ssh­-keygen ­-t dsa
master@mpi-2:~$ scp .ssh/id_dsa.pub master@mpi-1:~/mpi-2.pub
master@mpi-1:~$ cat mpi-2.pub >> .ssh/authorized_keys

測試登入 (不用輸入密碼)

master@mpi-1:~$ ssh mpi-2
master@mpi-2:~$ ssh mpi-1

============
4. 安裝 MPICH2
============

http://www.mcs.anl.gov/research/projects/mpich2

只要在 mpi-1 master folder 安裝一次即可

master@mpi-1:~$ cd /mpi
master@mpi-1:/mpi $ wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.1.1p1/mpich2-1.1.1p1.tar.gz

master@mpi-1:/mpi $ sudo apt-get install build-essential
master@mpi-1:/mpi $ mkidr mpich2
master@mpi-1:/mpi $ tar xzvf mpich2-­1.1.1p1.tar.gz
master@mpi-1:/mpi $ cd mpich2­-1.1.1p1
master@mpi-1:/mpi/mpich2­-1.1.1p1 $ ./configure --­prefix=/mpi/mpich2
master@mpi-1:/mpi/mpich2­-1.1.1p1 $ make
master@mpi-1:/mpi/mpich2­-1.1.1p1 $ sudo make install

設定 PATH 環境變數 (每個 node 都要設定)

master@mpi-1:~$ vi .bashrc
master@mpi-2:~$ vi .bashrc

export PATH="/mpi/mpich2/bin:$PATH"
export LD_LIBRARY_PATH="/mpi/mpich2/lib:$LD_LIBRARY_PATH"

在各個 node 上測試程式路徑是否正確

master@mpi-1:~$ which mpd
master@mpi-1:~$ which mpiexec
master@mpi-1:~$ which mpirun

master@mpi-2:~$ which mpd
master@mpi-2:~$ which mpiexec
master@mpi-2:~$ which mpirun

============
5. 設定 MPICH2
============

在各個 node 上建立 .mpd.conf 及 mpd.hosts

master@mpi-1:~$ vi mpd.hosts
master@mpi-2:~$ vi mpd.hosts

mpi-1
mpi-2

master@mpi-1:~$ echo MPD_SECRETWORD=xxx-xx > .mpd.conf
master@mpi-1:~$ chmod 600 .mpd.conf
master@mpi-2:~$ echo MPD_SECRETWORD=xxx-xx > .mpd.conf
master@mpi-2:~$ chmod 600 .mpd.conf

啟動各個 node 的 mpi service

master@mpi-1:~$ mpd &
master@mpi-1:~$ mpdboot -n 2
master@mpi-2:~$ mpd &
master@mpi-2:~$ mpdboot -n 2

在各 node 上檢查是否運行成功

master@mpi-1:~$ mpdtrace
mpi-1
mpi-2

master@mpi-2:~$ mpdtrace
mpi-1
mpi-2

完工

=====================
6. 另外可搭配安裝設定 mpiBLAST
=====================
master@mpi-1:~$ sudo apt-get install libncbi6 libncbi6-dev ncbi-data ncbi-tools-bin ncbi-tools-x11 libncbi6-dbg csh

Installation Guide:
http://www.mpiblast.org/Docs/Install

 

 

This guide describes how to building a simple MPICH cluster in ubuntu.

Before , you need an basic knowledge about mpich & clustering.

Here we have 4 nodes running ubuntu 7.04 with these host names: ub0,ub1,ub2,ub3;

 

1. Defining hostnames in etc/hosts/

 

Edit /etc/hosts like these:

 

127.0.0.1 localhost 192.168.133.100 ub0 192.168.133.101 ub1 192.168.133.102 ub2 192.168.133.103 ub3

 

Note that the file shouldn't be like this:

 

127.0.0.1 localhost 127.0.1.1 ub0 192.168.133.100 ub0 192.168.133.101 ub1 192.168.133.102 ub2 192.168.133.103 ub3

 

 

2. Installing NFS

 

To Install NFS just run this in terminal:

 

omid@ub0:~$ sudo apt-get install nfs-kernel-server

 

 

3. Sharing Master Folder

 

Make a folder in all nodes, we'll store our data and programs in this folder.

 

omid@ub0:~$ sudo mkdir /mirror

 

And then we share it over nodes on the master node.

 

omid@ub0:~$ sudo echo  /mirror *(rw,sync) >> /etc/exports

 

Note than we store out data and programs only in master node and other nodes will access them with NFS.

4. Mounting /master in nodes

 

 

omid@ub1:~$sudo mount ub0:/mirror /mirror
omid@ub2:~$sudo mount ub0:/mirror /mirror
omid@ub3:~$sudo mount ub0:/mirror /mirror

 

it's better to change fstab in order to mount it on every boot.

 

5. Defining a user for running MPI programs

 

We define a user with same name and same userid in all nodes with a home directory in /mirror.

Here we name it "mpiu"! Also we change the owner of /mirror to mpiu:

 

omid@ub0:~$ sudo chown mpiu /mirror

 

 

6. Installing SSH Server

 

Run this command in all nodes in order to install OpenSSH Server

 

omid@ub0:~$ sudo apt­-get install openssh­server

 

 

7. Setting up SSH with no pass phrase for communication between nodes

 

First we login with our new user:

 

omid@ub0:~$ su - mpiu

 

Then we generate DSA key for mpiu:

 

mpiu@ub0:~$ ssh­-keygen ­-t dsa

 

Leave passphrase empty.

Next we add this key to authorized keys:

 

mpiu@ub0:~$ cd .ssh
mpiu@ub0:~/.ssh$ cat id_pub.dsa >> authorized_keys

 

As the home directory of mpiu in all nodes is the same (/mirror/mpiu) , there is no need to run these commands on all nodes.

To test SSH run:

 

mpiu@ub0:~$ ssh ub1 hostname

 

It should return remote hostname without asking for passphrase.

 

8. Installing GCC

 

Install build-essential package:

 

mpiu@ub0:~$ sudo apt-get install build-essential

 

 

9.Installing Other Compilers

 

Other prefered compilers should be installed before installing MPICH.

In this step we install other compilers such as Inter Fortran, SGI compiler , ... .

 

10. Installing MPICH2

 

Download MPICH2 source code from http://www-unix.mcs.anl.gov/mpi/mpich .

Extract .tar.bz2 file in /mirror. Also make a folder for MPICH installation.

 

mpiu@ub3:/mirror$ mkidr mpich2
mpiu@ub3:/mirror$ tar xvf mpich2-­1.0.5p3.tar.gz
mpiu@ub3:/mirror$ cd mpich2­-1.0.5p3
mpiu@ub3:/mirror/mpich2­-1.0.5p3$ ./configure --­prefix=/mirror/mpich2 
mpiu@ub3:/mirror/mpich2­-1.0.5p3$ make
mpiu@ub3:/mirror/mpich2­-1.0.5p3$ sudo make install

 

For more information about compilation see README file in source package.

After successfully compiling and installing mpich, add these lines to "/mirror/mpiu/.bashrc/"

 

export PATH=/mirror/mpich2/bin:$PATH
export PATH
LD_LIBRARY_PATH="/mirror/mpich2/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH

 

Next we run this command in order to define MPICH installation path to SSH.

 

mpiu@ub0:~$ sudo echo /mirror/mpich2/bin >> /etc/environment

 

For testing our installation run:

 

mpiu@ub0:~$  which mpd
mpiu@ub0:~$  which mpiexec
mpiu@ub0:~$  which mpirun

 

 

11. setting up MPD

 

Create mpd.hosts in mpiu's home directory with nodes names:

 

ub3
ub2
ub1
ub0

 

and run :

 

mpiu@ub0:~$ echo secretword=something   >> ~/.mpd.conf
mpiu@ub0:~$ chmod 600 .mpd.conf

 

To test MPD run above commands. The output should be the current hostname.

 

mpiu@ub0:~$ mpd &
mpiu@ub0:~$ mpdtrace
mpiu@ub0:~$ mpdallexit

 

After all run mpd daemon:

 

mpiu@ub0:~$ mpdboot ­-n 4
mpiu@ub0:~$ mpdtrace

 

The output should be name of all nodes.

There are some examples in "mpich2-1.0.5/examples", we'll run one :

 

mpiu@ub0:~$ mpiexec -np 4 cpi

 

That's it!

For more information visit:

http://www-unix.mcs.anl.gov/mpi/mpich/