netCDF version 2 was released in 1993.
NCO (specifically ncks) began soon after this in 1994.
netCDF 3.0 was released in 1996, and we were eager to reap the
performance advantages of the newer netCDF implementation.
One netCDF3 interface call (nc_inq_libvers
) was added to
NCO in January, 1998, to aid in maintainance and debugging.
In March, 2001, the final conversion of NCO to netCDF3
was completed (coincidentally on the same day netCDF 3.5 was
released).
NCO versions 2.0 and higher are built with the
-DNO_NETCDF_2
flag to ensure no netCDF2 interface calls
are used.
However, the ability to compile NCO with only netCDF2
calls is worth maintaining because HDF version 4
1
(available from HDF)
supports only the netCDF2 library calls
(see http://hdf.ncsa.uiuc.edu/UG41r3_html/SDS_SD.fm12.html#47784).
Note that there are multiple versions of HDF.
Currently HDF version 4.x supports netCDF2 and thus
NCO version 1.2.x.
If NCO version 1.2.x (or earlier) is built with only
netCDF2 calls then all NCO operators should work with
HDF4 files as well as netCDF files
2.
The preprocessor token NETCDF2_ONLY
exists
in NCO version 1.2.x to eliminate all netCDF3
calls.
Only versions of NCO numbered 1.2.x and earlier have this
capability.
The NCO 1.2.x branch will be maintained with bugfixes only
(no new features) until HDF begins to fully support the
netCDF3 interface (which is employed by NCO 2.x).
If, at compilation time, NETCDF2_ONLY
is defined, then
NCO version 1.2.x will not use any netCDF3 calls
and, if linked properly, the resulting NCO operators will work
with HDF4 files.
The Makefile supplied with NCO 1.2.x is written
to simplify building in this HDF capability.
When NCO is built with make HDF4=Y
, the Makefile
sets all required preprocessor flags and library links to build
with the HDF4 libraries (which are assumed to reside under
/usr/local/hdf4
, edit the Makefile to suit your
installation).
HDF version 5 became available in 1999, but did not support netCDF (or, for that matter, Fortran) as of December 1999. By early 2001, HDF5 did support Fortran90. In 2004, Unidata and NCSA began a project to implement the HDF5 features necessary to support the netCDF API. NCO version 3.0.3 added support for reading/writing netCDF4-formatted HDF5 files in October, 2005. See Selecting Output File Format for more details.
HDF support for netCDF was completed with HDF5 version version 1.8 in 2007. The netCDF front-end that uses this HDF5 back-end was completed and released soon after as netCDF version 4. Download it from the netCDF4 website.
NCO version 3.9.0 added support for all netCDF4 atomic data
types except NC_STRING
in May, 2007.
Support for additional netCDF4 features has been incremental.
We add one netCDF4 feature at a time.
You must build NCO with netCDF4 to obtain this support.
The main netCDF4 features that NCO currently supports are the new
atomic data types, Lempel-Ziv compression, and chunking.
The new atomic data types are NC_UBYTE
, NC_USHORT
,
NC_UINT
, NC_INT64
, and NC_UINT64
.
Eight-byte integer support is an especially useful improvement from
netCDF3.
All NCO operators support these types, e.g., ncks
copies and prints them, ncra averages them, and
ncap2 processes algebraic scripts with them.
ncks prints compression information, if any, to screen.
NCO version 3.9.9 (June, 2009) added support for the
NC_STRING
netCDF4 atomic data type.
Ragged arrays of strings are supported.
NCO version 3.9.1 (June, 2007) added support for netCDF4 Lempel-Ziv deflation. Lempel-Ziv deflation is a lossless compression technique. See Deflation for more details.
NCO version 3.9.9 (June, 2009) added support for netCDF4 chunking. See Chunking for more details.
netCDF4-enabled NCO handles netCDF3 files without change. In addition, it automagically handles netCDF4 (HDF5) files: If you feed NCO netCDF3 files, it produces netCDF3 output. If you feed NCO netCDF4 files, it produces netCDF4 output. Use the handy-dandy ‘-4’ switch to request netCDF4 output from netCDF3 input, i.e., to convert netCDF3 to netCDF4. See Selecting Output File Format for more details.
Use appropriate caution while netCDF4 is beta software. Problems with netCDF4 and HDF libraries are still being fixed. NCO support for netCDF4 atomic types is relatively untested. Binary NCO distributions (RPMs and debs) still use netCDF3.
For now you must build NCO from source to get netCDF4 support.
Typically, one specifies the root of the netCDF4-beta
installation directory. Do this with the NETCDF4_ROOT
variable.
Then use your preferred NCO build mechanism, e.g.,
export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or- cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism
Our short term goal is to track the netCDF4-beta releases, keep the new netCDF4 atomic type support working, and iron out any problems. Our long term goal is to utilize more of the extensive new netCDF4 feature set. The next major netCDF4 feature we are likely to utilize is parallel I/O. We will enable this in the MPI netCDF operators.
[1] The Hierarchical Data Format, or HDF, is another self-describing data format similar to, but more elaborate than, netCDF.
[2] One must link the NCO code to the HDF4 MFHDF library instead of the usual netCDF library. Does ‘MF’ stands for Mike Folk? Perhaps. In any case, the MFHDF library only supports netCDF2 calls. Thus I will try to keep this capability in NCO as long as it is not too much trouble.