Q: I seem to run out of memory very quickly. Can you help?
A: Details of how memory can be managed within your LoadLeveler script
are described below:

MEMORY ALLOCATION UNDER LOADLEVELER

The total amount of real memory which can be occupied by a process is:

   26.96 Gb / tasks_per_node

This is called the RSS - the Resident Set Size.  (A node is an LPAR.) 
The RSS is divided into two areas:  stack and data.

In FORTRAN terms, stack is used for: 

  - subprogram calling information,
  - local variables, including arrays, unless they are marked SAVE.

Data is used for:

  - program code
  - static variables, 
       including COMMON variables and variables marked SAVE,
  - memory allocated by ALLOCATE - known as 'heap' variables,
  - buffers allocated by MPI,
  - the I/O system.

(See http://www.nersc.gov/nusers/resources/software/ibm/sp_memory.html
for more details about how memory is managed on IBM machines.)

A program that runs out of either stack or data space will fail.  For
stack space, you will usually get a message like this:

   ERROR: 0031-250  task 1: Segmentation fault

(Unfortunately, there are other things which can also cause this.)  If
you run out of data space, the message will usually be like this:

   1525-108 Error encountered while attempting to allocate a data
            object.  The program will stop.
 
With the new filter, you can specify the amount of memory to be
allocated to these two areas, using the keywords stack_limit and
data_limit.  For example:

  #@ stack_limit = 400mb

will set the stack to 400 Mb.  The following rules apply:

  * If stack_limit is not specified, it defaults to 20 Mb
    (This has been the fixed size until now)
  * If data_limit is not specified, it defaults to RSS - stack_limit
  * If stack_limit + data_limit > RSS, the job will fail

This means that if you specify neither of these, LoadLeveler will
behave as usual.

Notice that the system will not now allow you to have a virtual memory
which is larger than the RSS.  This means that the system should not
normally swap.  If you need more memory per process than is allowed by
these rules, you will need to reduce the number of tasks you have per
node.

You might think that we are forcing users to waste AUs, by obliging
them to decrease the number of tasks per node instead of swapping in
this case.  But swapping a multiprocessor application has serious
performance implications;  your application is likely to waste at
least as much time simply waiting for the swaps.

Also, to access more than 256 Mb of data space per task, users must compile
their codes with -q64 (or set -bmaxdata:1100000000).  Using -q64 changes
the default compiler setting from 32-bit mode to 64-bit mode.  Indeed, we 
recommend that all codes are compiled with -q64.