mpich2

Setting Up an MPICH2 Cluster in Ubuntu

Creator: Omid Alemi

omid.alemi@gmail.com

This guide describes how to building a simple MPICH cluster in ubuntu.

Before , you need an basic knowledge about mpich & clustering.

Here we have 4 nodes running ubuntu 7.04 with these host names: ub0,ub1,ub2,ub3;

1. Defining hostnames in etc/hosts/

Edit /etc/hosts like these:

127.0.0.1 localhost
192.168.133.100 ub0
192.168.133.101 ub1
192.168.133.102 ub2
192.168.133.103 ub3

Note that the file shouldn't be like this:

127.0.0.1 localhost
127.0.1.1 ub0
192.168.133.100 ub0
192.168.133.101 ub1
192.168.133.102 ub2
192.168.133.103 ub3

or like this:

127.0.0.1 localhost
127.0.1.1 ub0
192.168.133.101 ub1
192.168.133.102 ub2
192.168.133.103 ub3

otherwise other hosts will try to connect to localhost when they try to reach ub0.

2. Installing NFS

NFS allows us to create a folder on the master node and have it synced on all the other nodes. This folder can be used to store programs. To Install NFS just run this in the master node's terminal:

omid@ub0:~$ sudo apt-get install nfs-kernel-server

3. Sharing Master Folder

Make a folder in all nodes, we'll store our data and programs in this folder.

omid@ub0:~$ sudo mkdir /mirror

And then we share the contents of this folder located on the master node to all the other nodes. In order to do this we first edit the /etc/exports file on the master node to contain the additional line

/mirror *(rw,sync)

This can be done using vim or by issuing this command:

omid@ub0:~$ sudo echo /mirror *(rw,sync) >> /etc/exports

Note than we store out data and programs only in master node and other nodes will access them with NFS.
4. Mounting /master in nodes

Now all we need to do is to mount the folder on the other nodes. This can be done manually each time like this:

omid@ub1:~$sudo mount ub0:/mirror /mirror
omid@ub2:~$sudo mount ub0:/mirror /mirror
omid@ub3:~$sudo mount ub0:/mirror /mirror

But it's better to change fstab in order to mount it on every boot. We do this by editing /etc/fstab and adding this line:

ub0:/mirror /mirror nfs

5. Defining a user for running MPI programs

We define a user with same name and same userid in all nodes with a home directory in /mirror.

Here we name it "mpiu"! Also we change the owner of /mirror to mpiu:

omid@ub0:~$ sudo chown mpiu /mirror

6. Installing SSH Server

Run this command in all nodes in order to install OpenSSH Server

omid@ub0:~$ sudo apt­-get install openssh-server

7. Setting up SSH with no pass phrase for communication between nodes

First we login with our new user:

omid@ub0:~$ su - mpiu

Then we generate DSA key for mpiu:

mpiu@ub0:~$ ssh­-keygen ­-t dsa

Leave passphrase empty.

Next we add this key to authorized keys:

mpiu@ub0:~$ cd .ssh
mpiu@ub0:~/.ssh$ cat id_pub.dsa >> authorized_keys2

As the home directory of mpiu in all nodes is the same (/mirror/mpiu) , there is no need to run these commands on all nodes.

To test SSH run:

mpiu@ub0:~$ ssh ub1 hostname

It should return remote hostname without asking for passphrase.

8. Installing GCC

Install build-essential package:

mpiu@ub0:~$ sudo apt-get install build-essential

9.Installing Other Compilers

Other prefered compilers should be installed before installing MPICH.

In this step we install other compilers such as Inter Fortran, SGI compiler , ... .

10. Installing MPICH2

You can install mpich2 using Synaptic by typing:

sudo apt-get install mpich2

Alternatively it can be installed from source...

Download MPICH2 source code from http://www-unix.mcs.anl.gov/mpi/mpich .

Extract .tar.bz2 file in /mirror. Also make a folder for MPICH installation.

mpiu@ub3:/mirror$ mkidr mpich2
mpiu@ub3:/mirror$ tar xvf mpich2-­1.0.5p3.tar.gz
mpiu@ub3:/mirror$ cd mpich2­-1.0.5p3
mpiu@ub3:/mirror/mpich2­-1.0.5p3$ ./configure --­prefix=/mirror/mpich2
mpiu@ub3:/mirror/mpich2­-1.0.5p3$ make
mpiu@ub3:/mirror/mpich2­-1.0.5p3$ sudo make install

For more information about compilation see README file in source package.

After successfully compiling and installing mpich, add these lines to "/mirror/mpiu/.bashrc/"

export PATH=/mirror/mpich2/bin:$PATH
export PATH
LD_LIBRARY_PATH="/mirror/mpich2/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH

Next we run this command in order to define MPICH installation path to SSH.

mpiu@ub0:~$ sudo echo /mirror/mpich2/bin >> /etc/environment

For testing our installation run:

mpiu@ub0:~$ which mpd
mpiu@ub0:~$ which mpiexec
mpiu@ub0:~$ which mpirun

11. setting up MPD

Create mpd.hosts in mpiu's home directory with nodes names:

ub3
ub2
ub1
ub0

and run :

mpiu@ub0:~$ echo secretword=something >> ~/.mpd.conf
mpiu@ub0:~$ chmod 600 .mpd.conf

To test MPD run above commands. The output should be the current hostname.

mpiu@ub0:~$ mpd &
mpiu@ub0:~$ mpdtrace
mpiu@ub0:~$ mpdallexit

After all run mpd daemon:

mpiu@ub0:~$ mpdboot ­-n 4
mpiu@ub0:~$ mpdtrace

The output should be name of all nodes. If this doesn't succeed try running mpdcheck on all hosts to find possible errors in conf files (they will be marked with **).

There are some examples in "mpich2-1.0.5/examples", we'll run one :

mpiu@ub0:~$ mpiexec -n 4 cpi

That's it!
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
globus@earth:~$ tar xzf mpich2-1.2.1p1.tar.gz
globus@earth:~/mpich2-1.2.1p1$ ./configure --prefix=/usr/local/mpich2-1.2.1/ 2>&1 | tee c.txt
globus@earth:~/mpich2-1.2.1p1$ make 2>&1 |tee d.txt
globus@earth:~/mpich2-1.2.1p1$ make install
globus@earth:~$ sudo apt-get install mpich2



What is MPICH2?

MPICH2 is a high-performance and widely portable implementation of the Message Passing Interface (MPI) standard (both MPI-1 and MPI-2). The goals of MPICH2 are to provide an MPI implementation that efficiently supports different computation and communication platforms including commodity clusters (desktop systems, shared-memory systems, multicore architectures), and high-speed networks (10 Gigabit Ethernet, InfiniBand, Myrinet, Quadrics). To setup a grid/ring, initially one needs to download the MPICH2 package and install it on your home directory. The following steps will guide you to install MPICH2 and then use it thereafter.
Using MPICH2

1. Add a bin subdirectory

setenv PATH "/local/wecn-linux/apps/mpich2/bin:$PATH"

At this point check that everything is in order by doing

which mpd mpiexec mpirun

2. Place in your home directory a file named .mpd.conf containing the line "secretword=" where is a string known only to yourself. Also, make this file readable and writable only by you by entering:

chmod 600 .mpd.conf

3. Check the working of a "ring" using mpd command on the local machine and bring the ring down

mpd & mpdtrace mpdallexit

The output of mpdtrace should be the hostname of the machine you are running on. The mpdallexit causes the mpd daemon to exit.

4. To bring up a ring on a set of machines start a local daemon:

mpd &

5. To make the local daemon print its host and port in the form _, enter:

mpdtrace -l


6. Log into each of the other machines and do:

setenv PATH /local/wecn-linux/apps/mpich2/bin:$PATH

mpd -h -p &

where the hostname and port belong to the original mpd that you started.

7. From each machine, after starting the mpd, you can do mpdtrace to see which machines are in the ring so far. Repeat the above for as many machines you need to set up in a ring. Remember to exit mpd by using mpdallexit once your job is executed.