Next: , Previous: Chunking, Up: Common features


3.23 Deflation

Availability: ncap2, ncbo, ncea, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncwa
Short options: ‘-L
Long options: ‘--dfl_lvl’, ‘--deflate

All NCO operators that define variables support the netCDF4 feature of storing variables compressed with Lempel-Ziv deflation. The Lempel-Ziv algorithm is a lossless data compression technique. Activate this deflation with the -L dfl_lvl short option (or with the same argument to the ‘--dfl_lvl’ or ‘--deflate’ long options). Specify the deflation level dfl_lvl on a scale from no deflation (dfl_lvl = 0) to maximum deflation (dfl_lvl = 9). Minimal deflation (dfl_lvl = 1) achieves considerable storage compression with little time penalty. Higher deflation levels require more time for compression. File sizes resulting from minimal (dfl_lvl = 1) and maximal (dfl_lvl = 9) deflation levels typically differ by a few percent in size.

To compress an entire file using deflation, use

     ncks -4 -L 0 in.nc out.nc # No deflation (fast, no time penalty)
     ncks -4 -L 1 in.nc out.nc # Minimal deflation (little time penalty)
     ncks -4 -L 9 in.nc out.nc # Maximal deflation (much slower)

Unscientific testing shows that deflation compresses typical climate datasets by 30-60%. Packing, a lossy compression technique available for all netCDF files (see Packed data), can easily compress files by 50%. Packed data may be deflated to squeeze datasets by about 80%.

     ncks  -4 -L 1 in.nc out.nc # Minimal deflation (~30-60% compression)
     ncks  -4 -L 9 in.nc out.nc # Maximal deflation (~31-63% compression)
     ncpdq         in.nc out.nc # Standard packing  (~50% compression)
     ncpdq -4 -L 9 in.nc out.nc # Deflated packing  (~80% compression)

ncks prints deflation parameters, if any, to screen (see ncks netCDF Kitchen Sink).