• Home
• News
• Access
  • Overview
  • Apply
  • Policy
  • Usage Report
• Services
  • Overview
  • Storage Services
  • File Management
  • Data Storage
  • Data Transfer
• Projects
  • Overview
  • 2010
  • 2011
  • 2012
• Docs
• About
  • Overview
  • Hardware
• Support

File Management

The file management on all NorStore resources is kept as homogeneous as possible. It is recommended to familiarize yourself with the characteristics of the available file systems at the locations where your data is (or could be) stored in order to make the best choices with respect to capacity and performance and avoid faulty data transfers and loosing data unnecessary.

Storage space

There are three types of storage allocations on the NorStore resources associated with users and projects. Collaborating researchers and groups can request quota for a project. A project is formed a group of users that share a common research goal, common data collections, etc. Projects are intended for long-term storage of data. Project space in NorStore is shared by all users that belong to the project. Each user also has its own private user space. The user space is meant for private data but is only to be used for the project(s) that the user belongs to.

Users typically control where data resides. Selection of appropriate storage is important for efficient management of job output. Data can be stored in a user's home directory, in the project area, in a temporary location or in archival (mass) storage.

• User space. A user on the NorStore resources can have a home directory where he/she can store private information. Home directories are permanent, but have relatively small quotas (few GB). Home directories are meant for storing information or tools that are private to the user, but that are only used for the purpose of existing projects. Home directiories can for example be used for storing scripts etc. User space is **not** meant to store long-term or temporary data collections. The data in user space is only visible locally. Data in user space is never purged.
• Project space. Project space is the primary category of storage space on the NorStore facilities. Project space is meant for storing data collections. It is intended for archival (mass) storage of data and offers the advantage of long-term retention of (large amounts of) data. The data is in principle accessible from all other resources in the national infrastructure but access restrictions may be imposed if necessary. Data in project space is never purged.
• Scratch space. Scratch space is intended for temporary (short-term) storage only. The scratch area is **not** intended for on-line archival storage. Scratch space is shared with all users on the NorStore resource. No individual disk quotas are enforced on the scratch area. Scratch space is considerably larger than a user's home directory, but the available capacity fluctuates due to the concurrent use by other users and changing project space requirements. Files in scratch space are purged regularly and data will be automatically deleted after 7 days, without prior notice to the owner of the files. It is the responsibility of the user to secure the data that resides on the scratch space, including access control.

Important:

• All users have access to the scratch space for temporary storage of data.
• Project space must not be used for temporary storage of data.
• Scratch space cannot be used for long-term storage of data.
• Once data is purged it cannot be recovered.

Naming conventions

The following naming convention is used on (hypothetical) resource uix.norstore.no:

• User directory: /norstore/user/<username>
• Project directory: /norstore/project/<projectname>
• Scratch space for user: /norstore/scratch/user/<username> or /scratch/<username>
• Scratch space for project: /norstore/scratch/project/<projectname> or /scratch/<projectname>

The user and project is free to choose how to organize data in user and project (and scratch) directories.

Project names are chosen by the applicants. The user name is provided by the user and should be unique across all NorStore and Notur (HPC) resources.

In case the project requires copies (replicas, mirrors) of data on another NorStore resource (e.g., uiy.norstore.no), the copy is available at

• uiy.norstore.no:/backup/uix/project/<projectname>

Users are obviously free to create subdirectories as desired under the user and project space, using the regular Linux naming conventions.

The integrity of user or project data stored under the scratch space is not secured by the system, this is the reponsiblity of the user. No backup, replica or copies are made of the data stored on the scratch space. Data will be purged automatically after 7 days.

Visibility of data

A project can make data available through URL

Contact support@norstore.no to establish a project URL. Access may be restricted by password or certificate.

This location can be accessed using the main data transfer methods (http, scp, etc.) See also the Section on Data Transfer.

Transparency of data

Differences may exist in the underlying storage locations. A number of environment variables are in common use across all sites to provide a common syntax for referring to the location of storage that is available at each site, thereby hiding the underlying differences in their paths. For the moment, no such environment variables are defined.

Note: to move data from one resource to another, explicit paths must be specified rather than using environment variables.

---