Next: , Previous: Specifying Output Files, Up: Common features


3.7 Accessing Remote Files

Availability: All operators
Short options: ‘-p’, ‘-l
Long options: ‘--pth’, ‘--path’, ‘--lcl’, ‘--local
All NCO operators can retrieve files from remote sites as well as from the local file system. A remote site can be an anonymous FTP server, a machine on which the user has rcp, scp, or sftp privileges, or NCAR's Mass Storage System (MSS), or an OPeNDAP server. Examples of each are given below, following a brief description of the particular access protocol.

To access a file via an anonymous FTP server, supply the remote file's URL. FTP is an intrinsically insecure protocol because it transfers passwords in plain text format. Users should access sites using anonymous FTP when possible. Some FTP servers require a login/password combination for a valid user account. NCO allows these transactions so long as the required information is stored in the .netrc file. Usually this information is the remote machine name, login, and password, in plain text, separated by those very keywords, e.g.,

     machine dust.ess.uci.edu login zender password bushlied

Eschew using valuable passwords for FTP transactions, since .netrc passwords are potentially exposed to eavesdropping software 1.

SFTP, i.e., secure FTP, uses SSH-based security protocols that solve the security issues associated with plain FTP. NCO supports SFTP protocol access to files specified with a homebrew syntax of the form

     sftp://machine.domain.tld:/path/to/filename

Note the second colon following the top-level-domain (tld). This syntax is a hybrid between an FTP URL and a standard remote file syntax.

To access a file using rcp or scp, specify the Internet address of the remote file. Of course in this case you must have rcp or scp privileges which allow transparent (no password entry required) access to the remote machine. This means that ~/.rhosts or ~/ssh/authorized_keys must be set accordingly on both local and remote machines.

To access a file on NCAR's MSS, specify the full MSS pathname of the remote file. NCO will attempt to detect whether the local machine has direct (synchronous) MSS access. In this case, NCO attempts to use the NCAR msrcp command 2, or, failing that, /usr/local/bin/msread. Otherwise NCO attempts to retrieve the MSS file through the (asynchronous) Masnet Interface Gateway System (MIGS) using the nrnet command.

The following examples show how one might analyze files stored on remote systems.

     ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
     ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc
     ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc
     ncks -l . /ZENDER/nco/in.nc
     ncks -l . mss:/ZENDER/nco/in.nc
     ncks -l . http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata/in.nc

The first example works verbatim if your system is connected to the Internet and is not behind a firewall. The second example works if you have sftp access to the machine dust.ess.uci.edu. The third example works if you have rcp or scp access to the machine dust.ess.uci.edu. The fourth and fifth examples work on NCAR computers with local access to the msrcp, msread, or nrnet commands. The sixth command works if your local version of NCO is OPeNDAP-enabled (this is fully described in OPeNDAP). The above commands can be rewritten using the ‘-p input-path’ option as follows:

     ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc
     ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc
     ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc
     ncks -p /ZENDER/nco -l . in.nc
     ncks -p mss:/ZENDER/nco -l . in.nc
     ncks -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata \
          -l . in.nc

Using ‘-p’ is recommended because it clearly separates the input-path from the filename itself, sometimes called the stub. When input-path is not explicitly specified using ‘-p’, NCO internally generates an input-path from the first input filename. The automatically generated input-path is constructed by stripping the input filename of everything following the final ‘/’ character (i.e., removing the stub). The ‘-l output-path’ option tells NCO where to store the remotely retrieved file and the output file. Often the path to a remotely retrieved file is quite different than the path on the local machine where you would like to store the file. If ‘-l’ is not specified then NCO internally generates an output-path by simply setting output-path equal to input-path stripped of any machine names. If ‘-l’ is not specified and the remote file resides on the NCAR MSS system, then the leading character of input-path, ‘/’, is also stripped from output-path. Specifying output-path as ‘-l ./’ tells NCO to store the remotely retrieved file and the output file in the current directory. Note that ‘-l .’ is equivalent to ‘-l ./’ though the latter is recommended as it is syntactically more clear.


Footnotes

[1] NCO does not implement command line options to specify FTP logins and passwords because copying those data into the history global attribute in the output file (done by default) poses an unacceptable security risk.

[2] The msrcp command must be in the user's path and located in one of the following directories: /usr/local/bin, /usr/bin, /opt/local/bin, or /usr/local/dcs/bin.