NERSC Unveils Global Unified Parallel File System

In a typical high-performance computing (HPC) environment, each large computational system has its own local disk, and access to additional network-attached storage and archival storage servers. Such an environment prevents the consolidation of storage between systems, thus limiting the amount of working storage available on each system to its local disk capacity.

The result? An unnecessary replication of files on multiple systems, an increased workload on users to manage their files, and a burden on the infrastructure to support file transfers between the various systems.

The U.S. Department of Energy’s National Energy Research Scientific Computing Center (NERSC) is implementing a solution using existing and emerging technologies to overcome these inefficiencies. The Global Unified Parallel File System (GUPFS) Project aims to provide a scalable, high-performance, high-bandwidth, shared-disk file system for use by all of NERSC's high-performance production computational systems. GUPFS will provide unified file namespace for these systems and be integrated with the High Performance Storage System (HPSS). Storage servers, accessing the consolidated storage through the GUPFS shared-disk file systems, will provide Hierarchical Storage Management (HSM), backup and archival services. An additional goal is to distribute GUPFS-based file systems to geographically remote facilities as native file systems over the DOE Science Grid.

This environment will eliminate unnecessary data replication, simplify the user environment, provide better distribution of storage resources, and permit the management of storage as a separate entity while minimizing impacts on the computational systems.

The major enabling components of this envisioned environment are a high-performance shared-disk file system and a cost-effective, high-performance Storage Area Network (SAN). These emerging technologies, while evolving rapidly, are not targeted towards the needs of high-performance scientific computing. The GUPFS project intends to encourage the development of these technologies to support HPC needs through both collaborations with other institutions and vendors, and active development.

The GUPFS project is expected to span five years. During the first three years, NERSC plans to test, evaluate and advance shared-disk file systems, SAN technology, and other components of the GUPFS environment. This investigation is expected to include open source and commercial shared-disk file systems, new SAN fabric technologies as they become available, SAN and file system distribution over the WAN, HPSS integration, and file system performance and scaling. During this time NERSC also plans to form collaborations and become active in the shared-disk file system community. At the end of this period, NERSC will assess the feasibility of moving forward with a full implementation in the NERSC production environment.

Provided the assessment is favorable, the last two years of the GUPFS project will focus on implementation. The first step in implementation is to choose a file system based upon the previous evaluation and testing. The next step is building up the SAN infrastructure, while simultaneously starting the full development efforts required for Grid distribution and HSM integration.  Subsequently, there will be a phased rollout of the GUPFS file system on the various production systems, including HPSS. Once adequate SAN infrastructure, production systems, and required software are in place, DOE Science Grid distribution of the shared file systems over the WAN will be initiated.