[1] TO COMPILE MPI PROGRAM:
gcc -I/usr/local/vmi/mpich/include FILENAME.c -L/usr/local/vmi/mpich/lib/gcc \
-lmpich -lvmi -lpthread -ldl
[2] RUN THE PROGRAM:A) Use the following command:
qsub -I -V -l walltime=00:30:00,nodes=2:ppn=2:prod
This command will run an interactive job with a wall clock limit of 30 minutes, using two nodes and two processors per node.
The resource_names required are:
walltime: maximum wall clock time (hh:mm:ss) [default: 10
mins]
nodes: number of 2-processor nodes [should be
specified]
ppn: how many processors per node to use (1 or 2)
[should be specified]
resource: resource to be used (prod) [should be
specified]
B)
After you enter the command, you will have to wait for PBS to start the job. As with any job, your interactive job will wait in the queue until the specified number of nodes is available. If you specify a small number of nodes, the wait will be shorter. Once the job starts, you will see something like this:
qsub: waiting for job 14888.giant.ncsa.uiuc.edu to start qsub: job 14888.giant.ncsa.uiuc.edu ready ---------------------------------------- Begin PBS Prologue Mon Apr 15 13:46:23 CDT 2002 Job ID: 14888.giant.ncsa.uiuc.edu Username: davem Group: aau Nodes: titan155 titan156 End PBS Prologue Mon Apr 15 13:46:23 CDT 2002 ---------------------------------------- [titan156 ~]$
C) Now you are logged into the launch node. At this point, you can use the vmirun command to start your MPI program. For example:
[titan156 ~]$ vmirun ./a.out
Following, the command, you will see this output:
vmirun: running MPI job on 4 processors (2 nodes)
Launch node is titan156
VMI: Loading protocol Shmem from /etc/VMI/Devices/Shmem/libVMIShmem.so.
VMI: Loading protocol GM from /etc/VMI/Devices/GM/libVMIGM.so.
VMI/GM: Max # of Myrinet Nodes: 159, Max Ports: 8. PPP -> 1
VMI: Following protocol drivers were loaded:
[1] Shmem VMI Protocol Driver v0.1
(C) 1999 NCSA, University of Illinois
[2] GM-1.4 (Multi Port) VMI Protocol Driver v0.3
(C) 2001 NCSA, University of Illinois
Started VMI on node titan155(0)
Started VMI on node titan156(3)
Started VMI on node titan155(1)
Started VMI on node titan156(2)
...
...
[3] EXIT:
When you are done with your runs, you can use the exit command to end
the job:
[titan156 ~]$ exit logout qsub: job 14888.giant.ncsa.uiuc.edu completed
Note: You will be charged for the wall clock time used by all requested nodes until you end the job.
Here are the files that you will need to compile and run on the cluster:
batch.sample file #!/bin/csh # # Sample Batch Script for a titan cluster job # # Submit this script using the command: qsub # # Use the "qstat" command to check the status of a job. # # The following are embedded QSUB options. The syntax is #PBS (the # does # _not_ denote that the lines are commented out so do not remove). # # resource limits walltime: maximum wall clock time (hh:mm:ss) # nodes: number of 2-processor nodes # ppn: how many processors per node to use (1 or 2) # (you are always charged for the entire node) # prod: production titan cluster nodes #PBS -l walltime=00:30:00,nodes=4:ppn=2:prod # # queue name #PBS -q standard # # export all my environment variables to the job #PBS -V # # job name (default = name of script file) #PBS -N testjob # # filename for standard output (default =.o ) #PBS -o testjob.out # # filename for standard error (default = .e ) #PBS -e testjob.err # # send mail when the job begins and ends (optional) #PBS -m be # End of embedded QSUB options #set echo # echo commands before execution; use for debugging # Create the scratch directory for the job and cd to it setenv SCR `set_SCR` echo Scratch directory is $SCR if ($SCR != "") cd $SCR mkdir $PBS_JOBID # make subdirectory based on jobID and cd $PBS_JOBID # cd to this directory # Get files su3_rmd-titan and input.8.10 from directory in UniTree #msscmd cd sample,get input.8.10,get su3_rmd-titan #chmod u+x su3_rmd-titan # Since UniTree gives you the same permissions as # what your umask is set to (generally only read # and write), you need to give yourself # executable permission to run su3_rmd-titan #CHANGE cp ~/greetings . # Run the MPI program on all nodes/processors requested by the job # (program reads from input.8.10 and writes to output.8.10) #vmirun "su3_rmd-titan" vmirun greetings # After the job is done, save the output file to UniTree #CHANGE #msscmd put output.8.10 cp $SCR/* ~
makefile
MPILIBS = -L/usr/local/vmi/mpich/lib/gcc -lmpich -lvmi -lpthread -ldl
%.o: %.c
gcc -c -I/usr/local/vmi/mpich/include $<
test: test.o
gcc -o test test.o $(MPILIBS)
greetings: greetings.o
gcc -o greetings greetings.o $(MPILIBS)