Allocation Year Transition: 2019 to 2020

The NERSC 2019 Allocation Year (AY19) ends at 7 a.m. Pacific Time on Tuesday, January 14, 2020. AY20 runs through Monday, January 18, 2021.

Below is information related to the Allocation Year Transition plans. All times listed are Pacific Time.

Changes Prior to the Transition

The following changes will be implemented in preparation for the AY transition. 

Account Creation/Validation Moratorium January 8-14

No user accounts will be created between Wednesday, January 8, 2020 at noon and Tuesday, January 14, 2020 at 08:00.

No More New Project (ERCAP) Requests for AY19 after November 1, 2019

ERCAP requests for AY2019 projects are no longer being accepted.

Allocation Year Transition Process

IRIS Banking and Account Management System Downtime

The IRIS accounting web interface will be unavailable starting at 9 p.m. on Monday, January 13, 2020 and is expected to be back in service by midnight.

Web browser sessions that are active in the IRIS web interface before the downtime must be refreshed by logging out and then logging back in, in order to display accurate information for the new allocation year.

Scheduled System Downtimes

Cori will be unavailable during maintenance from 7 a.m. on Tuesday January 14 to 7 a.m. on Wednesday January 15, 2020. After the maintenance, queued ("idle") jobs associated with non-continuing projects (repos) will be deleted. All “scavenger” jobs will also be deleted from the queue. Held jobs older than 12 weeks will be deleted.

Queued jobs using srun that specifies both --mem and --mem-per-cpu will fail to run (please cancel and resubmit), and new jobs specifying both of these options will fail to submit due to a slight Slurm version upgrade (from 19.05.3 to 19.05.4).

Jupyter will be degraded during the Cori maintenance, 7 a.m. on January 14 to 7 a.m. on January 15.

AY 2020 Charging Begins on January 15

Charging for AY20 will begin on Wednesday, January 15: at midnight (the first moments of the day) if Cori returns from service early, or upon the return of Cori to users later in the day. Any jobs run after the maintenance concludes but before January 15 will not be charged for the time they used on January 14.

Changes for AY20

Major changes are coming in AY20. On Cori, the default programming environments will be updated to new defaults, a Python 3 module will become the default Python module, and the new Community File System will be in production.

New Cori Charge Factors

New charge factors for the KNL and Haswell nodes on Cori will be in effect for AY20 to better reflect the way that the compute nodes are used and to optimize user wait times and machine utilization. The new rates for each kind of node will be:

• Cori Haswell: 140 NERSC Hours per hour used per node     (increased from 90)
• Cori KNL: 80 NERSC Hours per hour used per node     (decreased from 90)

New "overrun" qos to Replace the "scavenger" qos 

The intent of the this QOS is to allow users with a zero or negative balance in one of their projects to continue to run jobs. User should submit to the "overrun" qos in AY20 when they use up the allocation hours.

New Software Defaults and Dynamic Linking Default on Cori

Cray software environment versions from the Cray Developer Toolkit (CDT) version 19.11 will become the defaults on January 14, 2020 after the system maintenance. Dynamic linking becomes the default with the new default CDT version.

The detailed list can be found here. (Note: The Cray CDT contains Cray compilers, the MPI message passing interface, performance and debugging tools, third party libraries, and other software.)

Python 2 retires, Python 3 becomes default

Python 2 will reach its end of life on January 1, 2020, at which point there will be no more development, bug fixes, patches, etc.

Therefore, upon the beginning of the 2020 Allocation Year at NERSC on January 14, the following changes to Python support will occur at NERSC:

• At the AY transition, a Python 3 module (python/3.7-anaconda-2019.10) will become the default Python module.
• The old Python 2 module will remain available but users must specify the version suffix.
• No new installations of Python 2 packages or modules will be performed.
• During the next Cori operating system upgrade, which could occur sometime in 2020, the Python 2 module will be retired.
• NERSC will actively support only Python 3 (or future Python versions should Python 3 become deprecated) on Perlmutter and future systems.

Community File System in Production

The Community File System (CFS) will be deployed on January 21, 2020 and will replace the Project file system. Existing user data will be migrated from Project to CFS, which will require a few days when neither file system will be writable.

Starting with the AY transition (Jan 14, 2020 at 7 a.m.), the Project file system will become read only until users are transitioned to CFS on or about January 21st. At this time, user data will have been copied over and users will be able to access their data on CFS at /global/cfs/cdirs/. The old file paths to Project, /project/projectdirs/, will point to the new CFS directory on Cori until Cori is retired. We encourage users to update their scripts now to use the new /global/cfs/cdirs path.

During this file system transition period, users should place a hold on any jobs that require write access to Project.

New Software Policy

NERSC is developing new software policy for the new Allocation Year.  Our aims in setting these policies are to help us to offer more effective support, and to give better visibility of the current state and future plans for software packages hosted at NERSC. More details are available here. 

Allocations in AY20

AY20 Allocation Awards

The AY20 allocation award letters were emailed to project PIs during the week of December 16. On January 14, 2020, new allocations of CPU time and mass storage will be in effect. Unused allocations from AY19 do not carry over into the new allocation year, except for ALCC projects. (ALCC projects should see the remainder of their 2019/20 ALCC awards reflected in their NERSC 2020 allocation. It may take one or two days for the continuing balance to be updated on our systems. 2019/20 ALCC time expires on June 30, 2020.)

PIs and/or Pi Proxies for renewing projects (repos) need to manually authorize each user in their project that should continue to be a member in AY20 (follow instructions here). Authorizations must be completed by January 8, 2020. Instructions for performing the Continuing User authorization will be sent out during December 2019.

New Projects

When a new project (repo) is created, its initial membership consists only of the PI designated in the ERCAP proposal. The PI (and any users designated as Proxies) may add other users to the project after the new AY starts and the project has been activated. Instructions for adding users to projects can be found in IRIS Guide for PIs. In addition, new users may now request to have themselves added to projects; these user-initiated additions must then be approved by the PI or by a Proxy. See How to Get a NERSC Account. Users will not be able to request accounts in new projects until after the new AY starts.

New Users

Requests for new users for both the renewing and new projects may incur a short delay while their identities are being vetted per DOE requirements.

Discontinued Users

After the new allocation year starts, a user will be considered "discontinued" if that user is not an authorized user in at least one active AY20 project. (Note: Users in 2019 projects (repos) are not automatically authorized for 2020; a PI or PI Proxy must authorize each user for 2020 via the Continuing User process. Instructions for this process will be emailed to PIs/Proxies when it starts.)

All discontinued users will have limited login access to their authorized systems until February 14, 2020. Such users will be unable to submit batch jobs, but will be able to perform limited interactive tasks. The intent of this access is to allow discontinued users the ability to "clean up", e.g., transfer files back to their home institution or archive them in HPSS.

All discontinued users will continue to have write access to HPSS until February 14, 2020, and then read-only and delete access for another five months (until July 14, 2020).