Availability: All operators Short options: ‘-p’, ‘-l’ Long options: ‘--pth’, ‘--path’, ‘--lcl’, ‘--local’ |
To access a file via an anonymous FTP server, supply the remote file's URL. FTP is an intrinsically insecure protocol because it transfers passwords in plain text format. Users should access sites using anonymous FTP when possible. Some FTP servers require a login/password combination for a valid user account. NCO allows these transactions so long as the required information is stored in the .netrc file. Usually this information is the remote machine name, login, and password, in plain text, separated by those very keywords, e.g.,
machine dust.ess.uci.edu login zender password bushlied
Eschew using valuable passwords for FTP transactions, since .netrc passwords are potentially exposed to eavesdropping software 1.
SFTP, i.e., secure FTP, uses SSH-based security protocols that solve the security issues associated with plain FTP. NCO supports SFTP protocol access to files specified with a homebrew syntax of the form
sftp://machine.domain.tld:/path/to/filename
Note the second colon following the top-level-domain (tld). This syntax is a hybrid between an FTP URL and a standard remote file syntax.
To access a file using rcp or scp, specify the Internet address of the remote file. Of course in this case you must have rcp or scp privileges which allow transparent (no password entry required) access to the remote machine. This means that ~/.rhosts or ~/ssh/authorized_keys must be set accordingly on both local and remote machines.
To access a file on a High Performance Storage System (HPSS) (such as that at NCAR, ECMWF, LANL, DKRZ, LLNL) specify the full HPSS pathname of the remote file. NCO will attempt to detect whether the local machine has direct (synchronous) HPSS access. In this case, NCO attempts to use the Hierarchical Storage Interface (HSI) command hsi get 2.
The following examples show how one might analyze files stored on remote systems.
ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc ncks -l . /ZENDER/nco/in.nc ncks -l . /home/zender/nco/in.nc ncks -l . http://motherlode.ucar.edu:8080/thredds/dodsC/testdods/in.nc
The first example works verbatim if your system is connected to the
Internet and is not behind a firewall.
The second example works if you have sftp access to the
machine dust.ess.uci.edu
.
The third example works if you have rcp or scp
access to the machine dust.ess.uci.edu
.
The fourth and fifth examples work on NCAR computers with
local access to the HPSS hsi get command
3.
The sixth command works if your local version of NCO is
OPeNDAP-enabled (this is fully described in OPeNDAP),
or if the remote file is accessible via wget.
The above commands can be rewritten using the ‘-p input-path’
option as follows:
ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc ncks -p /ZENDER/nco -l . in.nc ncks -p /home/zender/nco -l . in.nc # HPSS ncks -p http://motherlode.ucar.edu:8080/thredds/dodsC/testdods \ -l . in.nc
Using ‘-p’ is recommended because it clearly separates the input-path from the filename itself, sometimes called the stub. When input-path is not explicitly specified using ‘-p’, NCO internally generates an input-path from the first input filename. The automatically generated input-path is constructed by stripping the input filename of everything following the final ‘/’ character (i.e., removing the stub). The ‘-l output-path’ option tells NCO where to store the remotely retrieved file and the output file. Often the path to a remotely retrieved file is quite different than the path on the local machine where you would like to store the file. If ‘-l’ is not specified then NCO internally generates an output-path by simply setting output-path equal to input-path stripped of any machine names. If ‘-l’ is not specified and the remote file resides on the NCAR HPSS system, then the leading character of input-path, ‘/’, is also stripped from output-path. Specifying output-path as ‘-l ./’ tells NCO to store the remotely retrieved file and the output file in the current directory. Note that ‘-l .’ is equivalent to ‘-l ./’ though the latter is recommended as it is syntactically more clear.
[1] NCO does not implement command line options to
specify FTP logins and passwords because copying those data
into the history
global attribute in the output file (done by
default) poses an unacceptable security risk.
[2] The hsi command must be in the user's path in one of
the following directories: /usr/local/bin
or
/opt/hpss/bin
.
Tell us if the HPSS installation at your site places the
hsi command in a different location, and we will add that
location to the list of acceptable paths to search for hsi.
[3] NCO supported the old NCAR Mass Storage System (MSS) until version 4.0.7 in April, 2011. NCO supported MSS-retrievals via a variety of mechanisms including the msread, msrcp, and nrnet commands invoked either automatically or with sentinels like ncks -p mss:/ZENDER/nco -l . in.nc. Once the MSS was decommissioned in March, 2011, support for these retrieval mechanisms was removed from NCO.