2014 NERSC User Survey Results

Executive Summary

Satisfaction with NERSC remained extremely high in all major categories in 2014, according to 673 users who responded to the 2014 NERSC user survey. The survey satisfaction scores mirrored those achieved in 2013, a year that saw record or near-record scores across the board. The 673 responses was a new high, surpassing the previous mark of 613 in 2013. Those users accounted for 72 percent of all computing hours used at NERSC in 2014.

The overall satisfaction rating of 6.50 on a 7-point scale equaled the best ever recorded and the same as in 2013. The average of all satisfaction scores was 6.33, also the same as the previous year. Only three of the 105 satisfaction questions showed a decrease compared to the previous year.

Survey Format

NERSC conducts its yearly survey of users to gather feedback on the quality of its services and computational resources. The survey helps both DOE and NERSC staff judge how well NERSC is meeting the needs of users and points to areas where NERSC can improve.

The survey is conducted on the web, and in 2014 consists of 105 satisfaction questions that are scored numerically. In addition, we solicit free-form feedback from the users. In December 2014, 5,000 authorized users (those with registered accounts who have signed the computer policy use form) were invited by email to take the 2014 user survey. The survey was open through January 19, 2015.

7-Point Survey Satisfaction Scale

The survey uses a seven-point rating scale, where “1” is “very dissatisfied” and “7” indicates “very satisfied.” For each question the average score and standard deviation are computed. 

3-Point Usefulness and Importance Scale

Questions that asked if a system or service was useful or important used the following scale.

Changes from one year to the next were considered significant if they passed the t-test criteria at the 90% confidence level.

Overall Satisfaction

The average response to the item "Please rate your overall satisfaction with NERSC" was 6.50 on the seven-point satisfaction scale. This was the highest rating ever (to with statistical error) since the survey was created in its current form in 2003.

The following figure shows the overall satisfaction rating from 2003-2014. The red line labeled "Target" is the minimum acceptable DOE target for NERSC.

Overall Satisfaction Questions

Satisfaction was the same as in 2013 in all fives areas surveyed in the first section of the survey: "Overall Satisfaction."  As with the "Overall Satisfaction with NERSC" score, none of the five area have ever received a statistically significant higher rating. (Results shown here include responses from all NERSC users – including those from JGI and PDSF users – for 2012, 2013, and 2014.

Questions Asked on the "Overall Satisfaction" Survey Page

The most common rating in all five categories was "Very Satisfied (7)". The distribution for NERSC Overall is show below.

Other Overall Satisfaction Questions

Genepool596.170.81-

Highest Rated Items

 The 10 highest-rated items (of all 105 satisfaction questions) on the survey involved data storage systems (6 items), consulting and account support (2 items), and NERSC networking and cybersecurity (2). This tells us that users think NERSC takes good care of their data and makes it readily available, and provides excellent consulting, account support, networking, and cybersecurity.

Lowest Rated Items

The 10 lowest-rated items (of all 105 satisfaction questions) on the survey involved the batch queues and the associated wait times (5), and data policies and analysis software (5). Batch wait times on Hopper and Edison were below the NERSC minimum target of 5.25, reflecting the strong demand for access to NERSC systems.

Demographic Responses

Large, Medium, and Small MPP Users

For purpose of survey analysis, users were divided into those who used more than 3 million MPP hours (Large), between 500,000 and 3 million MPP hours (Medium), and less than 500,000 MPP hours (Small). Overall, there was not a great difference among the groups, in contrast to the previous year when larger users expressed greater satisfaction with NERSC.

Satisfaction by NERSC Experience

Users that had been computing at NERSC reported more satisfaction in most areas, the exception being in data resources where new users were the happiest.

Scientific Domains

While users in all scientific domains rated NERSC highly, there were some differences among the groups. The plots below show differences from the average of all user responses. Researchers in accelerators, applied math, chemistry, geosciences, lattice gauge theory, and fusion research ranked NERSC higher than average, while scores were lower users in astrophysics and biosciences. An explanation for these variations are not immediately clear and warrant further investigation.

Satisfaction by Allocation Type

Projects receive allocations via a number of methods. "Production" accounts are allocated by DOE program mangers through the ERCAP allocations process, "ALCC" (ASCR Leadership Computing Challenge) accounts are allocated by DOE's Office of Advanced Scientific Computing, the NERSC Director has a reserve of time to allocate, NERSC awards small "Startup" accounts, and there was a "Data Pilot" program in 2013.

Production accounts make up the vast majority of accounts (and thus survey responses), so their responses define the average and that group shows little variation from it. ALCC and Startup users rated NERSC higher than average in all categories, while Director's Reserve and Data Pilot users had mixed responses.

Hopper

Hopper has been a stable, productive system over the last three years and its satisfaction scores remained high, but down slightly from 2013 . The addition of the production Edison system in 2014 helped relieve some of the demand on Hopper, but the rating for queue wait times still fell to 5.17, but was still above the pre-Edison value of 4.90.

Representative User Comments

" 1. Hopper and Edison were clearly oversubscribed during at least the third quarter of this last calendar year. Turnaround became unworkable for a number of projects. Only more hardware or smaller allocations can really fix this, although the situation has improved following queue adjustments. Given the planned machine move and periods with lower machine availability, some careful consideration should be given to queue structures (etc.) to minimize the crush in the second half of 2015. 2. Some (more) careful expectations management may be needed for Cori/KL. For many applications it might be a lot of work to get performance not significantly better than a similar era Xeon. (Energy usage should be much better, but the allocations are in hours, not joules.) "

" The queue time last year for hopper is quite long, compared to the years before 2013, as far as I remember. This seems odd as more computational resources such as Edison are also available. Maybe some of the queue policies can be further improved. "

" Small to medium jobs are sometimes on the queue for a few consecutive days - and this is extremely inconvenient! "

" Please provide more capabilities for high-throughput jobs. It is very difficult with many users competing for time in a single high-throughput queue (thruput on hopper), especially when small projects need to compete with larger collaborations, which already have their own dedicated resources. I would hope that if NERSC provides dedicated resources to these large collaborations, then the other computing environments would be accessible to smaller projects. Currently, this is not the case for the thruput queue, the only high throughput queue available at NERSC "

" Something needs to be done about the queue wait times on Hopper and Edison. I often have to wait 1 to 2 weeks for a job to start requesting 1000 - 5000 processors. "

" Decrease wait times for short to medium jobs on hopper. "

" Reduce the batch queue wait times on Hopper and Edison, perhaps by altering the priorities or lowering the limits on the run times of jobs (e.g., 12 hrs for jobs with more than 512 cores, etc). Every code should be able to checkpoint. Also, use a 'fair share' system so that users lose priority as they run more jobs. "

" The queue time is very important for me. Now it will take me several days to finish one job on Hopper, compared with the very short waiting time (less than 1 day) in early 2014. "

" Currently, the batch wait times on both Hopper and Edison are terrible. I wish this could be improved going forward in future. "

" I have had an extremely positive experience with NERSC. Edison and hopper have been indispensable for my research over the past 2 years. NERSC's consultants have been extremely helpful and responsive. "

" Excellent programming environment, excellent hardware, good queueing policies on hopper; the higi-priority debug queue is extremely useful in particular (not all supercomputing centers have them, regrettably) "

" I have been very impressed with the stability of the computing systems I have used (Hopper and Edison). Any issues I had were resolved very quickly and thoroughly. Planned outages of systems were communicated well in advance of when they would occur, allowing me to schedule my tasks appropriately. "

" It is very easy to install and run the simulation I need to. Considering I do not have no experience using super computer, it took just few hours to run my case in Hopper. "

Edison

Edison was in preproduction in 2013. Users were not charged for usage, but the system was subject to frequent and unannounced downtimes. Users rated Edison overall higher than any of the other Edison categories. This was likely due to their overall satisfaction with the system, but unhappiness with frequest outages for testing and configuration. During this time, NERSC also experimented with a fair-share batch scheduler, for which many users expressed their dislike to NERSC consultants.

Representative User Comments

"I just wanted to say that I'm really happy with the shift from Hopper to Edison. My code is MPI only and the limited memory/core on Hopper made it really difficult to run the problem sizes I needed to run. It's very difficult to commit the necessary 6 months required to rewrite one's code to target a single fleeting architecture choice. With Edison's increased memory/core, I'm now able to make forward progress on scientific problems for which I'd been fighting with this limitation on Hopper."

"I am not fond at all of the 'fairshare' variable in the batch scheduler on Edison."

"I have mixed feelings about the new 'fair share' policy on Edison."

"Notifications about downtime. Edison uptime in November + December was abysmal. it's very hard to run long-term jobs when uptime is limited to two days at most. Plus the downtimes were far in excess of the planned windows. I can't complain since Edison was free and in 'beta', but it did make work difficult, especially since Hopper ended up being oversubscribed."

NOTE: The fair share component to the scheduler is not being used in production on Edison in 2014.

Carver

All satisfaction scores were up for Carver as it continued to be a reliable resource for users that like a standard LINUX computing environment.

Representative user comments:

"NERSC needs to continue to expand its offerings for data-intensive high-throughput computing. Carver is currently the best NERSC general purpose resource for this and I am concerned that it is going away, leaving only more targeted offerings like PDSF, Genepool, and non-data-intensive HPC offerings like Edison and Hopper."

"I can access my data from carver even if hopper or edison is down."

"There are a few routines only available on Carver that I need, but my simulations are all ran on Hopper and Edison. Global scratch makes it easy to access my simulation results from Carver."

"Data analysis is convenient on Carver."

"My only experience is with Carver and it has been all positive. Jobs enter the queue quickly, maintenance is performed quickly and I can quickly move data to, from and around the environment."

Genepool

Users of Genepool, a bioinfomatics cluster run by NERSC for the Joint Genome Institute, were far more satisfied in 2013 than in 2012. All satisfaction ratings increased by more than one full point. The improvement was expressed by one user, who wrote "I would like to thank the NERSC team for stabilizing both Genepool and our storage to a level that we can spend time thinking about Science rather than the infrastructure."

Representative user comments:

"Very happy with hardware configurations, plenty of compute resources (on genepool) Filesystem performance is excellent especially considering the size and complexity of the configuration."

"I would like to thank the NERSC team for stabilizing both Genepool and our storage to a level that we can spend time thinking about Science rather than the infrastructure. Specifically, Douglas Jacobsen and Kjiersten Fagnan have put in a lot of effort to maintain the open communication so that users can get more involved and informed. It is also obvious that the team can now more proactively manage the system. Thank you and keep up the excellent work!"

"There's a high degree of inconsistency on Genepool - depending on whether you ssh or qlogin, and depending on which node you are on, various commands like 'ulimit' give varying and unpredictable output. This makes it incredibly difficult to develop, test, and deploy software in a way that will assure it can run on any node. Furthermore, forking and spawning processes is impossible to do safely when UGE may nondeterministically kill anything that temporarily exceeds the ulimit, even if it uses a trivial amount of physical memory. As a result, it is difficult to utilize system parallelism - if you know a set of processes will complete successfully when run serially, but UGE may kill them if you try any forking or piping, then the serial version has to go into production pipelines no matter how bad the performance may be. Also, Genepool's file systems are extremely unreliable. Sometimes projectb may yield 500MB/s, and sometimes it may take 30 seconds to read a 1kb file. Sometimes 'ls', 'cd', or 'tab'-autocomplete may just hang indefinitely. These low points where it becomes unusable for anywhere from a few minutes to several hours are crippling. Whether this is caused by a problem at NERSC or the actions of users at JGI, it needs to be resolved."

"Overall metadata servers seem to be a bottleneck for the disk systems I frequently use (global homes and /projectb). In particular, global homes becomes almost unusable at least once per day. For example, vim open and closing can take minutes which make using the system extremely frustrating. The queue configuration on genepool has changed substantially over the past 18 months and while it is substantially improved in terms of utilization, coherency and usability it is still not possible to simply submit a job to the cluster and expect it to run on appropriate resources. Determining qsub parameters needed to get a job to run require detailed knowledge of the hardware and cluster configuration which ideally would not be the case. I would like to be able to submit a job and have it run."

PDSF

Satisfaction scores for PDSF increased significantly in the areas of overall satisfaction and uptime. No areas showed a significant decrease.

User Comments

I need more slots for jobs (nodes) on PDSF.

HPSS

Satisfaction scores for HPSS increased significantly in the areas of data reliability and user interface. No areas showed a significant decrease.

Representative User Comments

"Currently HPSS is entirely local; any chance of remote backup services?"

HPSS SRU distribution across users should be controllable by project managers. It's annoying to track down individual users if their HPSS uploads suddenly saturated the entire repo's allowance. The project managers should be able to modify the % charged to each repo and even purge some data if it's taking up too much time.

"Archival HPSS is one of the things NERSC does well."

"HPSS is pretty good, though it still needs a programatic interface (e.g. python library) rather than just script interfaces."

"Back up and retrieval of data on the HPSS is fast."

"I was also impressed with the Globus Online interface to HPSS, tho originally I may have hit some bumps there."

Project File System

Satisfaction scores for the Project file system improved significantly in system uptime. No areas showed a significant decrease.

User Comments

"Not sure if it's possible, but some sort of disk space explorer for large projectdirs would be great. e.g. I currently run "du" to figure out what directories are taking up a lot of space. Would be nice to have a pie chart showing which root directories on my projectdirs are taking up a lot space, then click that root dir to see which subdir is offending. There are visual tools to show the overall usage of scratch, projectdirs, home, etc as well as tools to find what users are taking up space. But I don't know of a nice disk space finder."

"We need the equivalent of /project optimized for installing collaboration code. Collaboration production code shouldn't live in some postdoc's home directory, and /project/projectdirs/ is optimized for large streaming files, not for installing code and managing modules."

"Could we please get a shorter path to /project/projectdirs/NAME ? Like /proj/NAME maybe or even /p/NAME?"

Software

Users liked NERSC's offering of software overall, especially programming libraries more in 2013 than in 2012. Other software satisfaction scores were unchanged.

Representative User Comments

"What does NERSC do well? Compile software that I need."

"The module system is very good, offers a lot of good and up-to-date software."

"NERSC is extremely well organized. I am never surprised about, e.g., new software, removed deprecated software, uptime or downtime, or anything else."

"I LOVE the broad choice of compilers, rapid response to issues, and excellent documentation. NERSC also does a great job with versioning of software executables and libraries."

"Provides excellent and reliable HPC platforms and relevant software."

"NERSC provides a broad spectrum of software applications to its users."

"Excellent array of scientific libraries and software."

"Support is AMAZING. Systems are rock solid, and software "just works."

"Regards system software upgrades: I understand the need to do this, but always dread it as often my code(s) may have problems. While consultants are always great at overcoming those types of problems, it takes time and I may end up missing compute times for "target dates".

"I hate the module system! It's a huge pain in the neck to use for someone not familiar with it."

"I need a programming environment that is easier to use and get the user's own software running."

"I would like to have detailed information on each software compilation on the nersc website e.g. lammps."

"Archived changes to software, even when "small" changes are made to the version of something, e.g., 2.1.2 to 2.1.3. Small changes that "shouldn't" have an effect sometimes do. When they are not documented, the user is left trying to figure out what they are doing wrong, when the change is what is causing the problem."

Consulting

Consulting satisfaction scores were high and unchanged from 2012.

Representative User Comments

he consultants are amazingly responsive. I am thoroughly impressed by response times and general helpfulness.

The NERSC facility should be the standard that all computing facilities strive for. The communication, consultation, and reliability are the attributes that come to my mind first.

Attention to our needs by the consultants is excellent. They do an excellent job of helping solve our technical issues as well as providing alternate solutions when our needs run into conflict with NERSC security and other policies.

Nersc consultants are the BEST! Timeliness and quality of help is just great.

So far I'm most impressed by the website (excellent beginner documentation) the friendly and helpful atitude of the consultants, and the frequent appearance of improvements.

Computer systems work well, help desk and consultants are amazing (fast and always useful).

I really appreciate the consultants. Being able to ask and get quick answers to questions is worth a lot.

Consultants and their help are AWESOME.

NERSC does very well in providing highly competent technical consulting, and they are very prompt.

My one small gripe is that it sometimes takes 3-4 hours for a new ticket to be assigned to the consultant. If the issue is urgent, this delay is problematic. I have tried contacting consultants with my questions directly, with rather mixed results, so now I either enter a ticket and brace myself for a long wait or do not bother with a ticket and attempt to get my answers through other means.

In the past we have asked about setting up a separate filesystem for software (see NERSC Consulting incident number 40343). This question has lingered unanswered for months.

Accounts and Allocations

Comments

I want to complain about NIM. The new process of adding users to a repo has some annoying bugs. The website can be non-responsive and I don't know if it actually did anything. Numerous times I had to send along a companion email to consult just to make sure my approval of a new user to a repo went in. One user emailed me they had been given the username of 0 (the digit zero)! Put this as a quote on a slide to the DOE program managers as a quote from one of your users: "It is ridiculous for the DOE's flagship scientific computing center to be stuck with such a primitive user management system as NIM. This system needs a serious overhaul or better yet, a replacement."

Communications

Users told us that email and status on the web a were communications methods that are the most useful to them.

Are you adequately informed about NERSC changes?

Yes: 354 (95.7%), No: 16 (4.3%)

Training

NERSC training satisfaction scores were up considerably from 2012.

Web

NERSC's users liked the increasing emphasis on system status information on the web and the ability to communicate with NERSC using mobile devices.</>

Comments

Not sure if it's possible, but some sort of disk space explorer for large projectdirs would be great. e.g. I currently run "du" to figure out what directories are taking up a lot of space. Would be nice to have a pie chart showing which root directories on my projectdirs are taking up a lot space, then click that root dir to see which subdir is offending. There are visual tools to show the overall usage of scratch, projectdirs, home, etc as well as tools to find what users are taking up space. But I don't know of a nice disk space finder.

Better search

Live status should be updated MUCH more frequently.

I have found it very useful to have slides from AIT/HPC meetings for reference on the Genepool Training and Tutorials page. However it would be nice to also have a few take-home points from the meeting on the site.

Navigation around the web site is easy for places I know, but to to find something new, that sometimes is difficult. It would help someone like me if there were an "index page", or "table of contents". I have a much easier time with that type of organization (eg, more words, alpha order, less image driven search).

Everything that is there is very useful to the user. The web page suits my needs PERFECTLY.

A page with a list of papers coming out of nersc computations would be very helpful.

My satisfaction level is extremely high, but: More technical information on designing parallel applications would be useful -- for instance, what kind of consistency guarantees do the file systems offer, or what kinds of interprocess communication are supported. Additionally, workflow management (eg. with Oozie) and web-based management of those workflows could be a useful feature -- SLAC has something similar with its pipeline infrastructure.

This is one of the most useful websites for getting started on a system. Thank you.

I would like to have more in-depth information of what is happening at NERSC available online. I can no longer participate in the NUGEX meetings and feel that I'm less informed than when I could actively participate. Maybe a transcript of the NUGEX conference calls would be useful.

The information that a person might need is almost always there. Sometimes finding that information can be tricky. For example, it seemed like I should have been able to find more quickly the format for acknowledging NERSC in manuscripts submitted for publication.

think "Queues and Scheduling Policies" page for clusters are most important, and then hope it could have a link at home page of clusters.

The software list needs to be updated somewhat. The list is comprehensive, but unfortunately some software requires custom execution environments (like the cluster compatibility modes) that should be better outlined and emphasized. Documentation for HSI and HTAR should be more extensive as well. But I am a power user, so I guess I look at the NERSC site as more of a reference than a tutorial.

Better searching.

It would be nice to have more useful system status information. Often the MOTD and website have not been updated.

Another complaint I have is timeliness in reporting about downtime. I understand that sometimes it may take a few minutes to realize that a system is down, but live status only allows a little bit of information out about what's going on. A little more openness from NERSC on its technical status would be great. How about a status blog we can be pointed to? I understand this can look bad for Cray (well tough for them) sometimes, but it really aggravates users that the updates on live status and MOTD are so terse. Let's have a kindler, gentler NERSC.

A cool thing would be a blog aggregating posts from various NERSC group leads and maybe some of the big users about what they're working on, what changes are coming to NERSC, etc. Might be more work but it could be distributed around...

If the "Queues and Policies" page for each system can be made more easily accessible on the NERSC web site, that would be helpful.

The website is extremely well put together.

I think the website is very well organized with lots of useful information.

So far I'm most impressed by the website (excellent beginner documentation) the friendly and helpful atitude of the consultants, and the frequent appearance of improvements.

A great website, great uptime--great notice of upcoming downtime.

Data Analytics and Visualization

Where to you perform analysis and visualization of data produced at NERSC?

Job Workflows

NERSC's users told us that it is important to them to be able to run both massively parallel jobs as well as ensembles of low ot medium-concurrency jobs.

Question: How important is it to you to be able to run the following types of jobs?

Data

Users top data needs are I/O bandwidth and the amount of storage space available for live access. Close behind is long-term data retention. The least important is analytics and visualization assistance from NERSC.

How important are each of the following to you?

Programming Models

We asked users to let us know if they had experience with the programming models shown in the chart below. By far, users are most familiar with OpenMP. This will be important for programming on next-generation systems, where mixed MPI and OpenMP is expected to be the main programming model, with other languages and paradigs vying for acceptance.

Application Readiness

We asked users if about how ready their codes were for using GPU or Intel Phi processors. Their responses could be divided into the categories show in the table below.

What are your plans for transitioning to a many-core architecture like GPUs or Intel Phi? How much of your code can use vector units or processors?

Number of users responding: 110

How can NERSC help you prepare for manycore architectures?

Number of respondants: 67

Comments

234 users responded to the following questions. Many responses fell into multiple categories, and can be broadly categorized as in the tables below. For the full text of all comments, see the "Full Comments" link in the navigation menu on the left side of this page.

What does NERSC do well?

Does NERSC provide the full range of systems and services you need to meet your scientific goals? If not, what else do you need?