2007/2008 User Survey Results

Response Summary

Many thanks to the 467 users who responded to this year's User Survey. The response rate has significantly increased from previous years:

• 70 percent of users who had used more than 1 million MPP hours when the survey opened responded
• 43 percent of users who had used between 10,000 and 1 million MPP hours responded
• The overall response rate for the 2,804 authorized users during the survey period was 16.3%.

The respondents represent all six DOE Science Offices and a variety of home institutions: see Respondent Demographics.

The survey responses provide feedback about every aspect of NERSC's operation, help us judge the quality of our services, give DOE information on how well NERSC is doing, and point us to areas we can improve. The survey results are listed below.

You can see the 2007/2008 User Survey text, in which users rated us on a 7-point satisfaction scale. Some areas were also rated on a 3-point importance scale or a 3-point usefulness scale.

The average satisfaction scores from this year's survey ranged from a high of 6.71 (very satisfied) to a low of 4.46 (neutral). Across 128 questions, users chose the Very Satisfied rating 9,486 times, and the Very Dissatisfied rating 81 times. The scores for all questions averaged 6.07, and the average score for overall satisfaction with NERSC was 6.3. See All Satisfaction Ratings.

For questions that spanned the 2007/2008 through 2003 surveys, the change in rating was tested for significance (using the t test at the 90% confidence level). Significant increases in satisfaction are shown in blue; significant decreases in satisfaction are shown in red.

Areas with the highest user satisfaction include account support, the NERSC Global Filesystem, the HPSS mass storage system, consulting services, network performance within the NERSC center, and up times for the Jacquard, Seaborg and Bassi systems.

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Areas with the lowest user satisfaction include batch wait times for Bassi and Jacquard, Franklin availability and I/O, training classes and data analysis and visualization services.

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

The largest increases in satisfaction over last year's survey are for the now retired Seaborg IBM POWER3 system, for computer and network operations 24 by 7 support, and for the software available on our systems.

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

The largest decreases in satisfaction over last year's survey are shown below.

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Survey Results Lead to Changes at NERSC

Every year we institute changes based on the previous year survey. In 2007 and early 2008 NERSC took a number of actions in response to suggestions from the 2006 user survey.

1. 2006 user survey: On the 2006 survey four users had concerns about the MOTD not being updated fast enough after status changes or that it was too long.

   NERSC response: The computer and network operations support has streamlined their procedures for managing status changes during outages, thus giving users more current status information both in the MOTD and by email (for users registered to receive status informational emails). The operations staff also increased their knowledge of account support procedures in order to provide better off hours support. The satisfaction score for Computer and network operations (24x7) support had a significant increase of .3 points over last year's score.

2. 2006 user survey: On the 2006 survey a number of users requested longer wall times for the largest machines.

   NERSC response: In January the wall time for Franklin's regular queues was increased from 12 hours to 24, and then to 36 hours in May. The satisfaction score for Franklin queue structure on the 2007/2008 survey was 6.03 out of 7.

3. 2006 user survey: On the 2006 survey a number of users commented on poor reliability for PDSF disks, and the satisfaction score for PDSF disks had the lowest PDSF hardware rating (5.1).

   NERSC response: NERSC has retired over 90 percent of the old NFS disk vaults and has installed new fiber channel based storage with better failover capabilities. In 2007/2008 the satisfaction score for PDSF disks increased to 5.54.

4. 2006 user survey: On the 2006 survey a number of users requested more resources for interactive and debug jobs.

   NERSC response: NERSC now reserves nodes for interactive/debug jobs on weekends; previously this was done Monday through Friday. This change did not change the satisfaction ratings significantly.

5. 2006 user survey: On the 2006 survey users asked that we provide more cycles and Get Franklin online ASAP.

   NERSC response: Franklin was delivered in January and February 2007. Initially installed with Catamount on the compute nodes, NERSC and Cray decided to install, and successfully tested for production use, an early release of Compute Node Linux (CNL). Early users started using Franklin with CNL in July 2007, and all users had access in September. Franklin was accepted in late October, 2007. Since then NERSC and Cray have worked together to improve system stability, I/O performance and the user environment. In July 2008 NERSC and Cray began upgrading Franklin's compute nodes from dual core to quad core, doubling both the number of cores and the amount of memory. We are looking forward to future enhancements, such as integrating the compute nodes with the NERSC Global Filesystem.

Users are invited to provide overall comments about NERSC:

150 users answered the question What does NERSC do well?  

• 103 respondents stated that NERSC gives them access to powerful computing resources without which they could not do their science;
• 59 mentioned excellent support services and NERSC's responsive staff;
• 24 highlighted good software support or an easy to use user environment;
• 17 pointed to good queue management or job turnaround;
• 15 mentioned data services (HPSS, large disk space, purge policy, NGF, data analysis).

Some representative comments are:

The NERSC facility is fantastic. I'm very pleased with the hardware available, the people, the help, and the queues.

NERSC is generally excellent, and has both leadership computing power and good ease of use, increasing productivity. This is most of all because the staff are very responsive to user needs and are effective in making leadership class machines work well for user applications. Additionally, the queue structure is clear and usable, the networking is very good, and the storage resources are adequate for large jobs. The analytics and visualization programs and associated software support are very important.

Good computing. Good storage. We always need more.

What NERSC is best at is the combination of large-scale computing facilities with more flexible queuing policies than in other comparable facilities. Also the existence of "small-scale supercomputers" (Jacquard) is very useful to make tests.

NERSC is excellent. Franklin is a great resource - lots of cores. The waiting of queues for large core runs is very nice. [Obviously there is a wait time for 16384 core run for 36 hours :) ]

NERSC does customer service very well. I am always pleased whenever I deal with NERSC. I would also say that NERSC's infrastructure for users is very helpful.

108 users responded to What should NERSC do differently?.

The top three areas of concern were Franklin stability and performance, job scheduling and resource allocation policies, and the need for more or different hardware resources. NERSC will analyze these comments and implement changes where possible over the next year.

Some of the comments from this section are:

It would be great if NERSC could magically improve the stability of Franklin... Unfortunately, hardware failures increase with the size and complexity of the system.

Need to improve network and load management on the log in nodes for Franklin. At times it is very difficult to get any work done since the response time is so slow.

Providing for long serial queues (~12 hours) and enabling these for applications such as IDL would further improve the usefulness of Franklin in particular. We appreciate your efforts to do this and look forward to finding a solution with you soon.

Less emphasis on INCITE, special users. More emphasis on providing high throughput for production applications.

As computing clusters grow, it would be very interesting/helpful for NERSC to invest in robust queuing systems such as Google's MapReduce model. It seems that all of NERSC's clusters are based upon the premise that failures are abnormal and can be dealt with as a special case. As clusters and job sizes grow, single point failures can really mess up a massively parallel job (Franklin) or a large number of parallel jobs (bad nodes draining queues on PDSF). Companies like Google have succeeded with their computing clusters by starting with the premise that hardware failures will happen regularly and building queuing systems that can automatically heal, rather than relying upon the users to notice that jobs are failing, stop them, alert the help system, wait for a fix, and then resubmit jobs.

I would suggest doing more to discourage single node and small jobs

NERSC's seaborg was a great success because of its reliability and its large amount of per-node memory. That led to the situations that majority of scientific codes ran well on it. The future computer (NERSC6) shall have a configuration with large amount of per-node memory (similar to bassi or larger, but with larger amount CPUs than bassi has).

Bassi has become so busy as to be almost useless to me.

NERSC should tell more about their strategic plans. Hopefully in three years we will be operating differently than we are now (command line submission, manual data management etc.) Is NERSC going to actively help with this, or simply be a resource provider? Is NERSC going to help campaign to get better performance and resiliency tools (fault tolerance) actually put into production vs being left as academic demos?

More disk space to users. The whole point of having a LARGE cluster is to do LARGE simulations. That means LARGE amounts of data. We should get more storage space (individually).

104 users answered the question How does NERSC compare to other centers you have used?   61 users stated that NERSC was an excellent center or was better than other centers they have used. Reasons given for preferring NERSC include its consulting services and responsiveness, its security, and its queue management.

25 users said that NERSC was comparable to other centers or gave a mixed review and 11 said that NERSC was not as good as another center they had used. Some users expressed dissatisfaction with user support, with available disk space or with queue turnaround time.

Here are the survey results:

1. Respondent Demographics
2. Overall Satisfaction and Importance
3. All Satisfaction, Importance and Usefulness Ratings
4. Hardware Resources
5. Software
6. Visualization and Data Analysis
7. HPC Consulting
8. Services and Communications
9. Web Interfaces
10. Comments about NERSC

Respondents by DOE Office and User Role:

Respondents by Organization:

Which NERSC resources do you use?

How long have you used NERSC?

What desktop systems do you use to connect to NERSC?

Web Browser Used to Take Survey:

Operating System Used to Take Survey: