Next: , Previous: Memory Requirements, Up: Memory Requirements


2.9.1 Single and Multi-file Operators

The multi-file operators currently comprise the record operators, ncra and ncrcat, and the ensemble operators, ncea and ncecat. The record operators require much less memory than the ensemble operators. This is because the record operators operate on one single record (i.e., time-slice) at a time, wherease the ensemble operators retrieve the entire variable into memory. Let MS be the peak sustained memory demand of an operator, FT be the memory required to store the entire contents of all the variables to be processed in an input file, FR be the memory required to store the entire contents of a single record of each of the variables to be processed in an input file, VR be the memory required to store a single record of the largest record variable to be processed in an input file, VT be the memory required to store the largest variable to be processed in an input file, VI be the memory required to store the largest variable which is not processed, but is copied from the initial file to the output file. All operators require MI = VI during the initial copying of variables from the first input file to the output file. This is the initial (and transient) memory demand. The sustained memory demand is that memory required by the operators during the processing (i.e., averaging, concatenation) phase which lasts until all the input files have been processed. The operators have the following memory requirements: ncrcat requires MS <= VR. ncecat requires MS <= VT. ncra requires MS = 2FR + VR. ncea requires MS = 2FT + VT. ncbo requires MS <= 3VT (both input variables and the output variable). ncflint requires MS <= 3VT (both input variables and the output variable). ncpdq requires MS <= 2VT (one input variable and the output variable). ncwa requires MS <= 8VT (see below). Note that only variables that are processed, e.g., averaged, concatenated, or differenced, contribute to MS. Variables which do not appear in the output file (see Subsetting Variables) are never read and contribute nothing to the memory requirements.

ncwa consumes between two and seven times the memory of a variable in order to process it. Peak consumption occurs when storing simultaneously in memory one input variable, one tally array, one input weight, one conformed/working weight, one weight tally, one input mask, one conformed/working mask, and one output variable. When invoked, the weighting and masking features contribute up to three-sevenths and two-sevenths of these requirements apiece. If weights and masks are not specified (i.e., no ‘-w’ or ‘-a’ options) then ncwa requirements drop to MS <= 3VT (one input variable, one tally array, and the output variable).

The above memory requirements must be multiplied by the number of threads thr_nbr (see OpenMP Threading). If this causes problems then reduce (with ‘-t thr_nbr’) the number of threads.