Next: , Previous: Retaining Retrieved Files, Up: Common features


3.9 Selecting Output File Format

Availability: ncap2, ncbo, ncea, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-3’, ‘-4
Long options: ‘--3’, ‘--4’, ‘--64bit’, ‘--fl_fmt’, ‘--netcdf4
All NCO operators support (read and write) all three (or four, depending on how one counts) file formats supported by netCDF4. The default output file format for all operators is the input file format. The operators listed under “Availability” above allow the user to specify the output file format independent of the input file format. These operators allow the user to convert between the various file formats. (The operators ncatted and ncrename do not support these switches so they always write the output netCDF file in the same format as the input netCDF file.)

netCDF supports four types of files: CLASSIC, 64BIT, NETCDF4, and NETCDF4_CLASSIC, The CLASSIC format is the traditional 32-bit offset written by netCDF2 and netCDF3. As of 2005, most netCDF datasets are in CLASSIC format. The 64BIT format was added in Fall, 2004.

The NETCDF4 format uses HDF5 as the file storage layer. The files are (usually) created, accessed, and manipulated using the traditional netCDF3 API (with numerous extensions). The NETCDF4_CLASSIC format refers to netCDF4 files created with the NC_CLASSIC_MODEL mask. Such files use HDF5 as the back-end storage format (unlike netCDF3), though they incorporate only netCDF3 features. Hence NETCDF4_CLASSIC files are perfectly readable by applications which use only the netCDF3 API and library. NCO must be built with netCDF4 to write files in the new NETCDF4 and NETCDF4_CLASSIC formats, and to read files in the new NETCDF4 format. Users are advised to use the default CLASSIC format or the NETCDF4_CLASSIC format until netCDF4 is more widespread. Widespread support for NETCDF4 format files is not expected for a few more years, 2010–2011, say. If performance or coolness are issues, then use NETCDF4_CLASSIC instead of CLASSIC format files.

As mentioned above, all operators write use the input file format for output files unless told otherwise. Toggling the long option ‘--64bit’ switch (or its key-value equivalent ‘--fl_fmt=64bit’) produces the netCDF3 64-bit offset format named 64BIT. NCO must be built with netCDF 3.6 or higher to produce a 64BIT file. Using the ‘-4’ switch (or its long option equivalents ‘--4’ or ‘--netcdf4’), or setting its key-value equivalent ‘--fl_fmt=netcdf4’ produces a NETCDF4 file (i.e., HDF). Casual users are advised to use the default (netCDF3) CLASSIC format until netCDF 3.6 and netCDF 4.0 are more widespread. Conversely, operators given the ‘-3’ (or ‘--3’) switch without arguments will (attempt to) produce netCDF3 CLASSIC output, even from netCDF4 input files.

These examples demonstrate converting a file from any netCDF format into any other netCDF format (subject to limits of the format):

     ncks --fl_fmt=classic in.nc foo_3c.nc # netCDF3 classic
     ncks --fl_fmt=64bit in.nc foo_364.nc # netCDF3 64bit
     ncks --fl_fmt=netcdf4_classic in.nc foo_4c.nc # netCDF4 classic
     ncks --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4
     ncks -3 in.nc foo_3c.nc # netCDF3 classic
     ncks --3 in.nc foo_3c.nc # netCDF3 classic
     ncks -4 in.nc foo_4.nc # netCDF4
     ncks --4 in.nc foo_4.nc # netCDF4
     ncks --64 in.nc foo364.nc # netCDF3 64bit

Of course since most operators support these switches, the “conversions” can be done at the output stage of arithmetic or metadata processing rather than requiring a separate step. Producing (netCDF3) CLASSIC or 64BIT files from NETCDF4_CLASSIC files will always work. However, producing netCDF3 files from NETCDF4 files will only work if the output files are not required to contain netCDF4-specific features.

Note that NETCDF4 and NETCDF4_CLASSIC are the same binary format. The latter simply causes a writing application to fail if it attempts to write a NETCDF4 file that cannot be completely read by the netCDF3 library. Conversely, NETCDF4_CLASSIC indicates to a reading application that all of the file contents are readable with the netCDF3 library. As of October, 2005, NCO writes no netCDF4-specific data structures and so always succeeds at writing NETCDF4_CLASSIC files.

There are at least three ways to discover the format of a netCDF file, i.e., whether it is a classic (32-bit offset) or newer 64-bit offset netCDF3 format, or is netCDF4 format. Each method returns the information using slightly different terminology that becomes easier to understand with practice.

First, examine the end of the first line of global metadata output by ‘ncks -M’:

     % ncks -M foo_3c.nc
     Opened file foo_3c.nc: dimensions = 19, variables = 261, global atts. = 4,
       id = 65536, type = NC_FORMAT_CLASSIC
     % ncks -M foo_364.nc
     Opened file foo_364.nc: dimensions = 19, variables = 261, global atts. = 4,
       id = 65536, type = NC_FORMAT_64BIT
     % ncks -M foo_4c.nc
     Opened file foo_4c.nc: dimensions = 19, variables = 261, global atts. = 4,
       id = 65536, type = NC_FORMAT_NETCDF4_CLASSIC
     % ncks -M foo_4.nc
     Opened file foo_4.nc: dimensions = 19, variables = 261, global atts. = 4,
       id = 65536, type = NC_FORMAT_NETCDF4

This method requires a netCDF4-enabled NCO version 3.9.0+ (i.e., from 2007 or later).

Second, query the file with ‘ncdump -k’:

     % ncdump -k foo_3.nc
     classic
     % ncdump -k foo_364.nc
     64-bit-offset
     % ncdump -k foo_4c.nc
     netCDF-4 classic model
     % ncdump -k foo_4.nc
     netCDF-4

This method requires a netCDF4-enabled netCDF 3.6.2+ (i.e., from 2007 or later).

The third option uses the POSIX-standard od (octal dump) command:

     % od -An -c -N4 foo_3c.nc
        C   D   F 001
     % od -An -c -N4 foo_364.nc
        C   D   F 002
     % od -An -c -N4 foo_4c.nc
      211   H   D   F
     % od -An -c -N4 foo_4.nc
      211   H   D   F

This option works without NCO and ncdump. Values of ‘C D F 001’ and ‘C D F 002’ indicate 32-bit (classic) and 64-bit netCDF3 formats, respectively, while values of ‘211 H D F’ indicate the newer netCDF4 file format.