Next: , Previous: Averaging vs. Concatenating, Up: Averaging vs. Concatenating


2.6.1 Concatenators ncrcat and ncecat

Joining independent files together along a record dimension is called concatenation. ncrcat is designed for concatenating record variables, while ncecat is designed for concatenating fixed length variables. Consider five files, 85.nc, 86.nc, ... 89.nc each containing a year's worth of data. Say you wish to create from them a single file, 8589.nc containing all the data, i.e., spanning all five years. If the annual files make use of the same record variable, then ncrcat will do the job nicely with, e.g., ncrcat 8?.nc 8589.nc. The number of records in the input files is arbitrary and can vary from file to file. See ncrcat netCDF Record Concatenator, for a complete description of ncrcat.

However, suppose the annual files have no record variable, and thus their data are all fixed length. For example, the files may not be conceptually sequential, but rather members of the same group, or ensemble. Members of an ensemble may have no reason to contain a record dimension. ncecat will create a new record dimension (named record by default) with which to glue together the individual files into the single ensemble file. If ncecat is used on files which contain an existing record dimension, that record dimension is converted to a fixed-length dimension of the same name and a new record dimension (named record) is created. Consider five realizations, 85a.nc, 85b.nc, ... 85e.nc of 1985 predictions from the same climate model. Then ncecat 85?.nc 85_ens.nc glues the individual realizations together into the single file, 85_ens.nc. If an input variable was dimensioned [lat,lon], it will have dimensions [record,lat,lon] in the output file. A restriction of ncecat is that the hyperslabs of the processed variables must be the same from file to file. Normally this means all the input files are the same size, and contain data on different realizations of the same variables. See ncecat netCDF Ensemble Concatenator, for a complete description of ncecat.

ncpdq makes it possible to concatenate files along any dimension, not just the record dimension. First, use ncpdq to convert the dimension to be concatenated (i.e., extended with data from other files) into the record dimension. Second, use ncrcat to concatenate these files. Finally, if desirable, use ncpdq to revert to the original dimensionality. As a concrete example, say that files x_01.nc, x_02.nc, ... x_10.nc contain time-evolving datasets from spatially adjacent regions. The time and spatial coordinates are time and x, respectively. Initially the record dimension is time. Our goal is to create a single file that contains joins all the spatially adjacent regions into one single time-evolving dataset.

     for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
       ncpdq -a x,time x_${idx}.nc foo_${idx}.nc # Make x record dimension
     done
     ncrcat foo_??.nc out.nc       # Concatenate along x
     ncpdq -a time,x out.nc out.nc # Revert to time as record dimension

Note that ncrcat will not concatenate fixed-length variables, whereas ncecat concatenates both fixed-length and record variables along a new record variable. To conserve system memory, use ncrcat where possible.