Compilation

General remarks on compilation

Use of /tmp by the compilers

By default the IBM compilers use /tmp to store temporary information. If /tmp fills up or information placed there is erased the compilation may fail, quite often with no clear indication on the cause of the problem. For reasons outlined here, users should not normally use /tmp and files placed on /tmp can get deleted by the system. To avoid the compilation failing at random places in case of the login node being very busy, users can set the environment variable TMPDIR to a directory in their own home or workspace. For example in ksh typing

  mkdir ~/MyTemp
  export TMPDIR=~/MyTemp

makes the compilers use the directory MyTemp in your home space.

Compiling sequential code

Serial codes are compiled with:

• xlf_r (FORTRAN 77)

• xlf90_r (Fortran 90)

• xlc_r (C)

• xlC_r (C++)

• javac (Java)

The _r suffix stands for re-entrant and denotes that the compiler should generate thread safe code. This is essential if you are using OpenMP or threads, but it also helps with MPI codes, so it is always recommended. There are several reasons for this:

• They allow MPI-2 features to be utilised.

• They give access to numerical libraries which are more stable and allow both 32- and 64-bit addressing.

• They are not detrimental to a code's performance.

Please note that for Fortran code the -qsuffix flag will be needed if your Fortran files do not have a .f or .F suffix, e.g.

xlf90_r -qsuffix=f=f90 -o mycode -q64 mycode.f90

For Java codes, the -O flag may be used for optimisation. e.g.

javac -O mycode.java

Compiling MPI code

Fortran

To compile Fortran MPI code the basic command is mpxlf90_r, e.g.:

mpxlf90_r -qsuffix=f=f90 -q64 -o mycode mycode.f90

The mp prefix denotes that the compiler is for multi-processor (MPI) programs.

Please note the comment above (Compiling sequential code) for the -qsuffix flag.

C

mpcc_r is used to compile MPI C code, e.g.:

mpcc_r -q64 -o mycode mycode.c

Please have a look at the below remark on the -q64 option.

C++

mpCC_r is used to compile MPI C++ code:

mpCC_r -q64 -o mycode mycode.C

If your code is using the C++ bindings of MPI instead of the standard C bindings, that is utilising functions which start with ``MPI::'' instead of ``MPI_'', you need to define the pre-processor variable _MPI_CPP_BINDINGS. This can be achieved by using the -D compiler flag:

mpCC_r -q64 -D_MPI_CPP_BINDINGS -o mycode mycode.C

Please have a look at the below remark on the -q64 option.

Compiling OpenMP code

Use the -qsmp option to turn on shared-memory parallelisation. You must use the _r versions of the compilers which provide thread-safe versions of the Fortran/C/C++ compilers, e.g.

$ export OMP_NUM_THREADS=2
$ xlf90_r -qsmp=omp -qsuffix=f=f90 -o hello hello.f90
** hello === End of Compilation 1 ===
1501-510 Compilation successful for file hello.f90.
$ ./hello
hello from 0 of 2
hello from 1 of 2

Other useful compiler switches

-q64
By default the xlf90_r compiler uses 32-bit addressing. To change this to use 64-bit addressing the -q64 options can be used. All object files making up the same executable must be compiled and linked with the -q64 flag.

This flag increases the amount of memory that a program can use and removes some of the restrictions on shared memory segments so the 64-bit MPI library has some additional optimisations. This flag is therefore a good choice for most codes. One possible disadvantage is that all pointers will double in size so there may be a performance impact on codes that use a lot of pointers, this mainly concerns codes written in C or C++.

If you want to make your own 64-bit object code libraries using the ar command you will also need to set the environment variable OBJECT_MODE=64 or use the -X 64 flag. Setting OBJECT_MODE=64 is an alternative to specifying -q64 as a linker option.

-qrealsize=8
It is important to note that the default size of variables declare REAL is 4 bytes (on a Cray, for example, it is 8 bytes). The -qrealsize option allows one to specify either 4 or 8 bytes for the deafault size. Please also note that changing this value from 4 to 8 bytes also changes the size of COMPLEX variables to 2 x 8 bytes, DOUBLE COMPLEX variables to 2 x 16 bytes and DOUBLE PRECISION variables from 8 bytes to 16 bytes. Integers (and logicals) are also promoted to 8 bytes. It is up to the user to ensure that the Fortran/C types are consistent with the MPI types, which will not be altered automatically.

Please see the section on `Porting code' for details of Fortran, C and MPI datatypes.

Example Makefiles

Example Makefiles, batch scripts and short test programs are available at

http://www.hpcx.ac.uk/support/FAQ/template.tar

The Makefiles are also shown below.

To compile the examples, just use:

make -f Makefile.MPI.[f|f90|c] (for .[f|f90|c] MPI files)
make -f Makefile.OMP.[f|f90|c] (for .[f|f90|c] OMP files)
make -f Makefile.MIX.[f|f90|c] (for .[f|f90|c] mixed MPI/OMP files)

It is probably best to do a make -f Makefile.X.Y clean beforehand too.

These examples have deliberately been written using 2 separate files (e.g. mpihello.f90 and hello.f90) to ensure the Makefiles are not completely trivial.

MPI fixed format Fortran code

Example Makefile (Makefile.MPI.f) to compile code with two MPI fixed format Fortran source files: mpihello.f and hello.f.

MF=     Makefile.MPI.f

FC=     mpxlf_r
FFLAGS= -q64 -O3 -qarch=pwr4 -qtune=pwr4
LFLAGS= $(FFLAGS)

EXE=    mpihello

SRC= \
        mpihello.f      \
        hello.f


#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .f .o

OBJ=    $(SRC:.f=.o)

.f.o:
        $(FC) $(FFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(FC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

MPI Fortran 90 code

Example Makefile (Makefile.MPI.f90) to compile code with two MPI Fortran 90 source files: mpihello.f90 and hello.f90.

MF=     Makefile.MPI.f90

FC=     mpxlf90_r
FFLAGS= -qsuffix=f=f90 -q64 -O3 -qarch=pwr4 -qtune=pwr4

LFLAGS= $(FFLAGS)

EXE=    mpihello

SRC= \
        mpihello.f90    \
        hello.f90

#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .f90 .o

OBJ=    $(SRC:.f90=.o)

.f90.o:
        $(FC) $(FFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(FC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

MPI C code

Example Makefile (Makefile.MPI.c) to compile code with two C source files: mpihello.c and hello.c

MF=     Makefile.MPI.c

CC=     mpcc_r
CFLAGS= -q64 -O3 -qarch=pwr4 -qtune=pwr4

LFLAGS= $(CFLAGS)

EXE=    mpihello

SRC= \
        mpihello.c      \
        hello.c

#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .c .o

OBJ=    $(SRC:.c=.o)

.c.o:
        $(CC) $(CFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(CC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

OpenMP fixed format Fortran code

Example Makefile (Makefile.OMP.f) to compile code with two OMP fixed format Fortran source files: omphello.f and hello.f.

MF=     Makefile.OMP.f

FC=     xlf_r
FFLAGS= -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp,noauto
LFLAGS= $(FFLAGS)

EXE=    omphello

SRC= \
        omphello.f      \
        hello.f


#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .f .o

OBJ=    $(SRC:.f=.o)

.f.o:
        $(FC) $(FFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(FC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

OpenMP Fortran 90 code

Example Makefile (Makefile.OMP.f90) to compile code with two OpenMP Fortran 90 source files: omphello.f90 and hello.f90

MF=     Makefile.OMP.f90

FC=     xlf90_r
FFLAGS= -qsuffix=f=f90 -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp,noauto
LFLAGS= $(FFLAGS)

EXE=    omphello

SRC= \
        omphello.f90    \
        hello.f90


#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .f90 .o

OBJ=    $(SRC:.f90=.o)

.f90.o:
        $(FC) $(FFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(FC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

OpenMP C code

Example Makefile (Makefile.OMP.c) to compile code with two OMP C source files: omphello.c and hello.c.

MF=     Makefile.OMP.c

CC=     xlc_r
CFLAGS= -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp:noauto

LFLAGS= $(CFLAGS)

EXE=    omphello

SRC= \
        omphello.c      \
        hello.c

#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .c .o

OBJ=    $(SRC:.c=.o)

.c.o:
        $(CC) $(CFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(CC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

Mixed MPI/OpenMP fixed format Fortran code

Example Makefile (Makefile.MIC.f) to compile code with two MPI/OpenMP fixed format Fortran source files: mixhello.f and hello.f.

MF=     Makefile.MIX.f

FC=     mpxlf_r
FFLAGS= -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp,noauto

LFLAGS= $(FFLAGS)

EXE=    mixhello

SRC= \
        mixhello.f      \
        hello.f

#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .f .o

OBJ=    $(SRC:.f=.o)

.f.o:
        $(FC) $(FFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(FC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

Mixed MPI/OpenMP Fortran 90 code

Example Makefile (Makefile.MIX.f90) to compile code with two mixed MPI/OpenMP Fortran 90 source files: mixhello.f90 and hello.f90

MF=     Makefile.MIX.f90

FC=     mpxlf90_r
FFLAGS= -qsuffix=f=f90 -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp,noauto

LFLAGS= $(FFLAGS)

EXE=    mixhello

SRC= \
        mixhello.f90    \
        hello.f90

#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .f90 .o

OBJ=    $(SRC:.f90=.o)

.f90.o:
        $(FC) $(FFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(FC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core

Mixed MPI/OpenMP C code

Example Makefile (Makefile.MIX.c) to compile code with two mixed MPI/OpenMP C source files: mixhello.c and hello.c.

MF=     Makefile.MIX.c

CC=     mpcc_r
CFLAGS= -q64 -O3 -qarch=pwr4 -qtune=pwr4 -qsmp=omp:noauto

LFLAGS= $(CFLAGS)

EXE=    mixhello

SRC= \
        mixhello.c      \
        hello.c

#
# No need to edit below this line
#

.SUFFIXES:
.SUFFIXES: .c .o

OBJ=    $(SRC:.c=.o)

.c.o:
        $(CC) $(CFLAGS) -c $<

all:    $(EXE)

$(EXE): $(OBJ)
        $(CC) $(LFLAGS) -o $@ $(OBJ)

$(OBJ): $(MF)

tar:
        tar cvf $(EXE).tar $(MF) $(SRC)

clean:
        rm -f $(OBJ) $(EXE) core