Data Transfer
The NorStore facilities are stand-alone resources. One must explicitly transfer files to/from the NorStore systems.
In general, if you plan to move large quantities data across sites, use a fast file system at both ends. That is, a (parallel) file system for temporary storage of intermediate to large quantities of data. The NorStore resources are equipped with a parallel file system.
There are three basic types of transfers:
- Moving data between NorStore resources. A number of clients are available for high-performance data transfer. You may choose an appropriate client depending on location and size of your transfer.
- Moving data between NorStore and Notur (HPC) resources. Also here, a number of clients are available for high-performance data transfer.
- Moving data between a NorStore resource and an external (or local) machine. Many users will find basic data transfer utilities like sftp and scp useful for most transfers. Clients for high-performance data transfer exist for transferring your data more efficiently and these are highly recommended for large data transfers. The transfer speed that can be achieved depends on the configuration of the external resource (hardware specification, file system and OS-tuning) as well as all the capacities and configurations of the network that lies in between the NorStore and external resource. To achieve high-performance data transfer, both ends usually must be configured. In case there will be frequent or large data transfer between the NorStore and external resource, it will be worthwhile to install the proper transfer protocols and tools also on the external resource. Contact the NorStore support staff for more information.
NorStore gives researchers access to the data from a broad range of
locations and aims to promote group collaboration. The aim is that one
can access the NorStore resources from any location using the same
commands/protocols. In particular, the file systems on the NorStore
resources are
The NorStore resources have 10 Gbit/s connectivity to the academic research network that is operated by UNINETT.
Basic Data Transfer
A number of commonly used protocols are available. Transferring data between your local client machine and a NorStore resource can be done using common file transfer protocols. Currently, the following protocols are supported:
- SCP: Secure Copy
- SCP on wikipedia
- In a Linux environment, type 'man scp' for a list of command-line options for SCP.
- SFTP: SSH File Transfer Protocol
- SFTP on wikipedia
- In a Linux environment, type 'man sftp' for a list of command-line options for SFTP.
- HTTP, HTTPS: Hypertext Transfer Protocol (over Secure Socket Layer)
Access by http(s) can be controlled through the use of passwords or (browser) certificates.
For security reasons, TELNET and FTP are not supported.
High Performance Data Transfer
The above protocols work well but they are not tuned for performance. To achieve high performance for bulk transfer, it is recommended that one of the following methods is used.
- HPN-SSH: a patch for OpenSSH designed to increase performance in an SSH-session over high-speed connections. The patched SSH is fully interoperable with other SSH servers and clients. HPN clients will be able to download faster from non-HPN servers, and HPN servers will be able to receive uploads faster from non-HPN clients. However, the host receiving the data must have a properly tuned TCP/IP stack.
- GRIDFTP: a uniform, secure, high-performance interface to file-based storage systems on the Grid
- Components for Grid Data Management
- GridFTP developed by the Globus Alliance
- GSI-SCP/SFTP - OpenSSH with Grid (GSI) authentication
In general, the GridFTP protocol and SCP/SFTP based on the patched HPN-version of SSH perform well for bulk data transfer.