2013 NERSC User Survey Results

Executive Summary

NERSC user satisfaction was up significantly in all major categories in 2013, according to the results of the 2013 annual NERSC user survey. A record number of users, 613, responded to the survey, which was conducted Dec. 6, 2013 to Jan. 31, 2014. This represents a 12.1% response rate from the 5.000 users who had been active in 2013. The 613 users accounted for about 67% of all the raw compute hours used on Hopper, Edison, Carver, Dirac, and Planck in 2013.

The overall satisfaction rating of 6.50 on a 7-point scale equaled the best ever recorded and was an increase of 0.21 points over 2012. The average of all satisfaction scores was 6.34, also the highest ever. For the first time, none of the 100 satisfaction questions showed a decrease compared to the previous year.

Survey Format

NERSC conducts its yearly survey of users to gather feedback on the quality of its services and computational resources. The survey helps both DOE and NERSC staff judge how well NERSC is meeting the needs of users and points to areas where NERSC can improve.

The survey is conducted on the web, and in 2013 consists of 100 satisfaction questions that are scored numerically. In addition, we solicit free-form feedback from the users. On December 6, 2013, 5,000 authorized users (those with registered accounts who have signed the computer policy use form) were invited by email to take the 2013 user survey. The survey was open through January 31, 2014.

7-Point Survey Satisfaction Scale

The survey uses a seven-point rating scale, where “1” is “very dissatisfied” and “7” indicates “very satisfied.” For each question the average score and standard deviation are computed. 

3-Point Usefulness and Importance Scale

Questions that asked if a system or service was useful or important used the following scale.

Changes from one year to the next were considered significant if they passed the t-test criteria at the 90% confidence level.

Overall Satisfaction

The average response to the item "Please rate your overall satisfaction with NERSC" was 6.50 on the seven-point satisfaction scale. This was the second highest average rating since the survey was created in its current form in 2003. In fact, it was not statistically significant different from the 6.54 rating in 2011.

The following figure shows the overall satisfaction rating from 2003-2013. The red line labeled "Target" is the minimum acceptable DOE target for NERSC.

Overall Satisfaction Questions

Satisfaction was up significantly over 2012 in all fives areas surveyed in the first section of the survey: "Overall Satisfaction."  As with the "Overall Satisfaction with NERSC" score, none of the five area have ever received a statistically significant higher rating. (Results shown here include responses from all NERSC users – including those from JGI and PDSF users – for both 2012 and 2013.

Questions Asked on the "Overall Satisfaction" Survey Page

The most common rating in all five categories was "Very Satisfied (7)" as shown in these plots of the numerical distribution for each topic.

Other Overall Satisfaction Questions

* - Edison was in a pre-production early user phase in 2013.

Highest Rated Items

 The 11 highest-rated items (of all 100 satisfaction questions) on the survey involved data storage systems (7 items), consulting and account support (3 items), and NERSC cybersecurity. This tells us that users think NERSC takes good care of their data and makes it readily available, and provides excellent consulting, account support, and cybersecurity.

Lowest Rated Items

 The 10 lowest-rated items (of all 100 satisfaction questions) on the survey involved the pre-production Edison system (5), batch queues and wait times (5), the NX X-Windows accelerator service, and visualization software. This tells us that users want better throughput for their jobs and stable systems (during pre-production, Edison was frequently made unavailable, sometimes on short notice). None of the lowest-rated items were scored lower than in 2012, and two – NX and Hopper queue wait times – showed significant improvements. For the first time, not a single satisfaction score was below the NERSC minimum target of 5.25.

* - Edison was in a pre-production early user phase in 2013.

Demographic Responses

Large, Medium, and Small MPP Users

For purpose of survey analysis, users were divided into those who used more than 3 million MPP hours (Large), between 500,000 and 3 million MPP hours (Medium), and less than 500,000 MPP hours (Small). Overall, larger users expressed greater satisfaction with NERSC. The execption was in software; many Medium MPP users rely on NERSC's pre-built suite of application software.

Satisfaction by NERSC Experience

Users that had been computing at NERSC reported more satisfaction in most areas, the exception being in software where new users were the happiest.

Scientific Domains

While users in all demographic groups rated NERSC highly, there were some differences among the groups. The plots below show differences from the average of all user responses. Researchers in applied math, chemistry, and fusion research ranked NERSC higher than average, while scores were lower users in astrophysics, high energy physics, and biosciences. An explanation for these variations are not immediately clear and warrent further investigation.

Satisfaction by Allocation Type

Projects receive allocations via a number of methods. "Production" accounts are allocated by DOE program mangers through the ERCAP allocations process, "ALCC" (ASCR Leadership Computing Challenge) accounts are allocated by DOE's Office of Advanced Scientific Computing, the NERSC Director has a reserve of time to allocate, NERSC awards small "Startup" accounts, and there was a "Data Pilot" program in 2013.

Production accounts make up the vast majority of accounts (and thus survey responses), so their responses define the average and that group shows little variation from it. ALCC and Startup users rated NERSC higher than average in all categories, while Director's Reserve and Data Pilot users had mixed responses.

Hopper

Hopper has been a stable, productive system over the last two years and its satisfaction scores remained high and largely unchanged. The addition of free pre-production time on Edison in 2013 helped relieve some of the demand on Hopper and the rating for queue wait times jumped significantly to 5.31; however that was still the lowest scoring item on the survey.

Representative User Comments

"Of all the HPC systems I have used (Stampede, Lonestar), Hopper has been the most satisfying experience in terms of user experience. I must admit that the learning curve to familiarize myself with the environment was a little steep in the beginning, but once everything was setup, it was smooth sailing all the way!"

"The only (serious) problem to me is the slow response of basic file commands, especially on Hopper, and especially everything involving X11."

"I run most of my VASP and Turbomole jobs on NERSC (mostly Hopper, slightly Carver). Hopper is vital for my project. I appreciate the professional performance of NERSC as a whole and Hopper particularly."

"Hopper is an excellent Massively Parallel computer; its resources are most adequate for my research."

"I find it impressive how reliable Hopper and Edison are, given their complexity and the wide variety of users they have."

"NERSC maintains a highly reliable cluster (Hopper) on which I can run a variety of molecular dynamics simulations important to my Ph.D. research."

"The turn-around time to run jobs is so slow that it hurts scientific productivity. My post-doc spends most of his week waiting for jobs to get through the queue on Hopper."

"Hopper and Edison have been quite overloaded lately, and it takes quite a while to run a large job job there."

"Queue times on Hopper and Edison make them much less valuable than they would otherwise be. Its bad when you have to wait several hours to run a job that takes less than 1 hour and only uses a handful of nodes."

Edison

Edison was in preproduction in 2013. Users were not charged for usage, but the system was subject to frequent and unannounced downtimes. Users rated Edison overall higher than any of the other Edison categories. This was likely due to their overall satisfaction with the system, but unhappiness with frequest outages for testing and configuration. During this time, NERSC also experimented with a fair-share batch scheduler, for which many users expressed their dislike to NERSC consultants.

Representative User Comments

"I just wanted to say that I'm really happy with the shift from Hopper to Edison. My code is MPI only and the limited memory/core on Hopper made it really difficult to run the problem sizes I needed to run. It's very difficult to commit the necessary 6 months required to rewrite one's code to target a single fleeting architecture choice. With Edison's increased memory/core, I'm now able to make forward progress on scientific problems for which I'd been fighting with this limitation on Hopper."

"I am not fond at all of the 'fairshare' variable in the batch scheduler on Edison."

"I have mixed feelings about the new 'fair share' policy on Edison."

"Notifications about downtime. Edison uptime in November + December was abysmal. it's very hard to run long-term jobs when uptime is limited to two days at most. Plus the downtimes were far in excess of the planned windows. I can't complain since Edison was free and in 'beta', but it did make work difficult, especially since Hopper ended up being oversubscribed."

NOTE: The fair share component to the scheduler is not being used in production on Edison in 2014.

Carver

All satisfaction scores were up for Carver as it continued to be a reliable resource for users that like a standard LINUX computing environment.

Representative user comments:

"NERSC needs to continue to expand its offerings for data-intensive high-throughput computing. Carver is currently the best NERSC general purpose resource for this and I am concerned that it is going away, leaving only more targeted offerings like PDSF, Genepool, and non-data-intensive HPC offerings like Edison and Hopper."

"I can access my data from carver even if hopper or edison is down."

"There are a few routines only available on Carver that I need, but my simulations are all ran on Hopper and Edison. Global scratch makes it easy to access my simulation results from Carver."

"Data analysis is convenient on Carver."

"My only experience is with Carver and it has been all positive. Jobs enter the queue quickly, maintenance is performed quickly and I can quickly move data to, from and around the environment."

Genepool

Users of Genepool, a bioinformatics cluster run by NERSC for the Joint Genome Institute, were far more satisfied in 2013 than in 2012. All satisfaction ratings increased by more than one full point. The improvement was expressed by one user, who wrote "I would like to thank the NERSC team for stabilizing both Genepool and our storage to a level that we can spend time thinking about Science rather than the infrastructure."

Representative user comments:

"Very happy with hardware configurations, plenty of compute resources (on genepool) Filesystem performance is excellent especially considering the size and complexity of the configuration."

"I would like to thank the NERSC team for stabilizing both Genepool and our storage to a level that we can spend time thinking about Science rather than the infrastructure. Specifically, Douglas Jacobsen and Kjiersten Fagnan have put in a lot of effort to maintain the open communication so that users can get more involved and informed. It is also obvious that the team can now more proactively manage the system. Thank you and keep up the excellent work!"

"There's a high degree of inconsistency on Genepool - depending on whether you ssh or qlogin, and depending on which node you are on, various commands like 'ulimit' give varying and unpredictable output. This makes it incredibly difficult to develop, test, and deploy software in a way that will assure it can run on any node. Furthermore, forking and spawning processes is impossible to do safely when UGE may nondeterministically kill anything that temporarily exceeds the ulimit, even if it uses a trivial amount of physical memory. As a result, it is difficult to utilize system parallelism - if you know a set of processes will complete successfully when run serially, but UGE may kill them if you try any forking or piping, then the serial version has to go into production pipelines no matter how bad the performance may be. Also, Genepool's file systems are extremely unreliable. Sometimes projectb may yield 500MB/s, and sometimes it may take 30 seconds to read a 1kb file. Sometimes 'ls', 'cd', or 'tab'-autocomplete may just hang indefinitely. These low points where it becomes unusable for anywhere from a few minutes to several hours are crippling. Whether this is caused by a problem at NERSC or the actions of users at JGI, it needs to be resolved."

"Overall metadata servers seem to be a bottleneck for the disk systems I frequently use (global homes and /projectb). In particular, global homes becomes almost unusable at least once per day. For example, vim open and closing can take minutes which make using the system extremely frustrating. The queue configuration on genepool has changed substantially over the past 18 months and while it is substantially improved in terms of utilization, coherency and usability it is still not possible to simply submit a job to the cluster and expect it to run on appropriate resources. Determining qsub parameters needed to get a job to run require detailed knowledge of the hardware and cluster configuration which ideally would not be the case. I would like to be able to submit a job and have it run."

PDSF

Satisfaction scores for PDSF increased significantly in the areas of overall satisfaction and uptime. No areas showed a significant decrease.

User Comments

I need more slots for jobs (nodes) on PDSF.

HPSS

Satisfaction scores for HPSS increased significantly in the areas of data reliability and user interface. No areas showed a significant decrease.

Representative User Comments

"Currently HPSS is entirely local; any chance of remote backup services?"

HPSS SRU distribution across users should be controllable by project managers. It's annoying to track down individual users if their HPSS uploads suddenly saturated the entire repo's allowance. The project managers should be able to modify the % charged to each repo and even purge some data if it's taking up too much time.

"Archival HPSS is one of the things NERSC does well."

"HPSS is pretty good, though it still needs a programatic interface (e.g. python library) rather than just script interfaces."

"Back up and retrieval of data on the HPSS is fast."

"I was also impressed with the Globus Online interface to HPSS, tho originally I may have hit some bumps there."

Project File System

Satisfaction scores for the Project file system improved significantly in system uptime. No areas showed a significant decrease.

User Comments

"Not sure if it's possible, but some sort of disk space explorer for large projectdirs would be great. e.g. I currently run "du" to figure out what directories are taking up a lot of space. Would be nice to have a pie chart showing which root directories on my projectdirs are taking up a lot space, then click that root dir to see which subdir is offending. There are visual tools to show the overall usage of scratch, projectdirs, home, etc as well as tools to find what users are taking up space. But I don't know of a nice disk space finder."

"We need the equivalent of /project optimized for installing collaboration code. Collaboration production code shouldn't live in some postdoc's home directory, and /project/projectdirs/ is optimized for large streaming files, not for installing code and managing modules."

"Could we please get a shorter path to /project/projectdirs/NAME ? Like /proj/NAME maybe or even /p/NAME?"

Software

Users liked NERSC's offering of software overall, espeicially programming libraries more in 2013 than in 2012. Other software satisfaction scores were unchanged.

Representative User Comments

"What does NERSC do well? Compile software that I need."

"The module system is very good, offers a lot of good and up-to-date software."

"NERSC is extremely well organized. I am never surprised about, e.g., new software, removed deprecated software, uptime or downtime, or anything else."

"I LOVE the broad choice of compilers, rapid response to issues, and excellent documentation. NERSC also does a great job with versioning of software executables and libraries."

"Provides excellent and reliable HPC platforms and relevant software."

"NERSC provides a broad spectrum of software applications to its users."

"Excellent array of scientific libraries and software."

"Support is AMAZING. Systems are rock solid, and software "just works."

"Regards system software upgrades: I understand the need to do this, but always dread it as often my code(s) may have problems. While consultants are always great at overcoming those types of problems, it takes time and I may end up missing compute times for "target dates".

"I hate the module system! It's a huge pain in the neck to use for someone not familiar with it."

"I need a programming environment that is easier to use and get the user's own software running."

"I would like to have detailed information on each software compilation on the nersc website e.g. lammps."

"Archived changes to software, even when "small" changes are made to the version of something, e.g., 2.1.2 to 2.1.3. Small changes that "shouldn't" have an effect sometimes do. When they are not documented, the user is left trying to figure out what they are doing wrong, when the change is what is causing the problem."

Consulting

Consulting satisfaction scores were high and unchanged from 2012.

Representative User Comments

he consultants are amazingly responsive. I am thoroughly impressed by response times and general helpfulness.

The NERSC facility should be the standard that all computing facilities strive for. The communication, consultation, and reliability are the attributes that come to my mind first.

Attention to our needs by the consultants is excellent. They do an excellent job of helping solve our technical issues as well as providing alternate solutions when our needs run into conflict with NERSC security and other policies.

Nersc consultants are the BEST! Timeliness and quality of help is just great.

So far I'm most impressed by the website (excellent beginner documentation) the friendly and helpful atitude of the consultants, and the frequent appearance of improvements.

Computer systems work well, help desk and consultants are amazing (fast and always useful).

I really appreciate the consultants. Being able to ask and get quick answers to questions is worth a lot.

Consultants and their help are AWESOME.

NERSC does very well in providing highly competent technical consulting, and they are very prompt.

My one small gripe is that it sometimes takes 3-4 hours for a new ticket to be assigned to the consultant. If the issue is urgent, this delay is problematic. I have tried contacting consultants with my questions directly, with rather mixed results, so now I either enter a ticket and brace myself for a long wait or do not bother with a ticket and attempt to get my answers through other means.

In the past we have asked about setting up a separate filesystem for software (see NERSC Consulting incident number 40343). This question has lingered unanswered for months.

Accounts and Allocations

Comments

I want to complain about NIM. The new process of adding users to a repo has some annoying bugs. The website can be non-responsive and I don't know if it actually did anything. Numerous times I had to send along a companion email to consult just to make sure my approval of a new user to a repo went in. One user emailed me they had been given the username of 0 (the digit zero)! Put this as a quote on a slide to the DOE program managers as a quote from one of your users: "It is ridiculous for the DOE's flagship scientific computing center to be stuck with such a primitive user management system as NIM. This system needs a serious overhaul or better yet, a replacement."

Communications

Users told us that email and status on the web a were communications methods that are the most useful to them.

Are you adequately informed about NERSC changes?

Yes: 354 (95.7%), No: 16 (4.3%)

Training

NERSC training satisfaction scores were up considerably from 2012.

Web

NERSC's users liked the increasing emphasis on system status information on the web and the ability to communicate with NERSC using mobile devices.</>

Comments

Not sure if it's possible, but some sort of disk space explorer for large projectdirs would be great. e.g. I currently run "du" to figure out what directories are taking up a lot of space. Would be nice to have a pie chart showing which root directories on my projectdirs are taking up a lot space, then click that root dir to see which subdir is offending. There are visual tools to show the overall usage of scratch, projectdirs, home, etc as well as tools to find what users are taking up space. But I don't know of a nice disk space finder.

Better search

Live status should be updated MUCH more frequently.

I have found it very useful to have slides from AIT/HPC meetings for reference on the Genepool Training and Tutorials page. However it would be nice to also have a few take-home points from the meeting on the site.

Navigation around the web site is easy for places I know, but to to find something new, that sometimes is difficult. It would help someone like me if there were an "index page", or "table of contents". I have a much easier time with that type of organization (eg, more words, alpha order, less image driven search).

Everything that is there is very useful to the user. The web page suits my needs PERFECTLY.

A page with a list of papers coming out of nersc computations would be very helpful.

My satisfaction level is extremely high, but: More technical information on designing parallel applications would be useful -- for instance, what kind of consistency guarantees do the file systems offer, or what kinds of interprocess communication are supported. Additionally, workflow management (eg. with Oozie) and web-based management of those workflows could be a useful feature -- SLAC has something similar with its pipeline infrastructure.

This is one of the most useful websites for getting started on a system. Thank you.

I would like to have more in-depth information of what is happening at NERSC available online. I can no longer participate in the NUGEX meetings and feel that I'm less informed than when I could actively participate. Maybe a transcript of the NUGEX conference calls would be useful.

The information that a person might need is almost always there. Sometimes finding that information can be tricky. For example, it seemed like I should have been able to find more quickly the format for acknowledging NERSC in manuscripts submitted for publication.

think "Queues and Scheduling Policies" page for clusters are most important, and then hope it could have a link at home page of clusters.

The software list needs to be updated somewhat. The list is comprehensive, but unfortunately some software requires custom execution environments (like the cluster compatibility modes) that should be better outlined and emphasized. Documentation for HSI and HTAR should be more extensive as well. But I am a power user, so I guess I look at the NERSC site as more of a reference than a tutorial.

Better searching.

It would be nice to have more useful system status information. Often the MOTD and website have not been updated.

Another complaint I have is timeliness in reporting about downtime. I understand that sometimes it may take a few minutes to realize that a system is down, but live status only allows a little bit of information out about what's going on. A little more openness from NERSC on its technical status would be great. How about a status blog we can be pointed to? I understand this can look bad for Cray (well tough for them) sometimes, but it really aggravates users that the updates on live status and MOTD are so terse. Let's have a kindler, gentler NERSC.

A cool thing would be a blog aggregating posts from various NERSC group leads and maybe some of the big users about what they're working on, what changes are coming to NERSC, etc. Might be more work but it could be distributed around...

If the "Queues and Policies" page for each system can be made more easily accessible on the NERSC web site, that would be helpful.

The website is extremely well put together.

I think the website is very well organized with lots of useful information.

So far I'm most impressed by the website (excellent beginner documentation) the friendly and helpful atitude of the consultants, and the frequent appearance of improvements.

A great website, great uptime--great notice of upcoming downtime.

Data Analytics and Visualization

Where to you perform analysis and visualization of data produced at NERSC?

Job Workflows

NERSC's users told us that it is important to them to be able to run both massively parallel jobs as well as ensembles of low ot medium-concurrency jobs.

Question: How important is it to you to be able to run the following types of jobs?

Data

Users top data needs are I/O bandwidth and the amount of storage space available for live access. Close behind is long-term data retention. The least important is analytics and visualization assistance from NERSC.

How important are each of the following to you?

Programming Models

We asked users to let us know if they had experience with the programming models shown in the chart below. By far, users are most familiar with OpenMP. This will be important for programming on next-generation systems, where mixed MPI and OpenMP is expected to be the main programming model, with other languages and paradigs vying for acceptance.

Application Readiness

We asked users if about how ready their codes were for using GPU or Intel Phi processors. Their responses could be divided into the categories show in the table below.

What are your plans for transitioning to a many-core architecture like GPUs or Intel Phi? How much of your code can use vector units or processors?

Number of users responding: 110

How can NERSC help you prepare for manycore architectures?

Number of respondants: 67

Comments

234 users responded to the following questions. Many responses fell into multiple categories, and can be broadly categorized as in the tables below. For the full text of all comments, see the "Full Comments" link in the navigation menu on the left side of this page.

What does NERSC do well?

Does NERSC provide the full range of systems and services you need to meet your scientific goals? If not, what else do you need?