NERSC's 'Shifter' Scales up to More Than 9,000 Cori KNL Processors

Shifter—the scalable software toolkit that leverages container-based computing to help supercomputer users run a wider range of software more easily and securely—is now better than ever.

Developed at Berkeley Lab's National Energy Research Scientific Computing Center (NERSC) to improve the flexibility and usability of HPC systems for data-intensive workloads by leveraging container-based computing tools such as Docker, Shifter was originally released in 2015. Development of this open-source tool has been ongoing, and in October NERSC released Shifter 16.08.3. The new version, which improves Shifter’s functionality and also fixes some security bugs, is a recommended replacement for all previous releases, according to Doug Jacobsen, an HPC consultant at NERSC and one of the primary developers of Shifter. Jacobsen recently tested Shifter’s scalability using the new version and said the results were better than expected.

“We demonstrated that we can run complex, scientific, Python-based codes on more than 9,000 KNL cores,” he said. “This is significant because this is a tremendously challenging workload to run on large-scale platforms. Shifter’s scalability brings into reach the ability for a large quantity of modern scientific codes to run at scale at NERSC.”

From the get-go, Shifter was designed to allow a user to supply supercomputing centers such as NERSC with a Docker image that can then be converted to a form that can efficiently be distributed and accessed on the compute nodes at scale. Shifter leverages the user interface that Docker provides to allow users to define and create their software "images"—a snap shot of an operating environment for an application or a service that typically contains the required operating system software, libraries, and binaries in addition to anything specific to the application.

“Shifter is a productivity enhancer,” said Shane Canon, a project engineer at NERSC who has also been instrumental in the software package’s development. “It makes it easier for users to develop something locally on their laptop and push to another place. It also enhances scientific reproducibility; being able to take that image and know that you can reliably instantiate it over and over or share it with others is really powerful.”

Shifter leverages parts of the Docker ecosystem, such as Docker registries and Docker Hub, but does not directly use their software internally, Canon emphasized. It is not a replacement for Docker functionality; it is specifically focused on HPC use cases.

“Shifter is strongly focused on the needs and requirements of the HPC workload, which means it can deliver the functionality we are looking for while still meeting the overall performance requirements and constraints that a large-scale HPC platform imposes,” Jacobsen said.

The update also includes the ability to create per-node writable temporary spaces, even if the system lacks a local disk. “The new per-node write cache is particularly useful for data analysis stacks like Spark which often assume a local disk,” Canon explained. Using this capability, NERSC was able to run a Spark job across more than 1,600 nodes on its Cori system.

Over the past year, numerous experimental facilities and academic institutions have found that Shifter makes it much easier for them to run their data-intensive workloads in an HPC environment, Canon noted.

“We continue to see adoption of Shifter not just at NERSC but at other sites,” he said, citing deployments at the Swiss National Supercomputing Centre (CSCS), the Victorian Life Sciences Computation Initiative in Australia and Argonne National Laboratory. Shifter has been particularly popular among the high-energy physics and genomics communities, which often have complex software stacks or use a large range of tools, he added.