2005 User Survey Results

Response Summary

Many thanks to the 201 users who responded to this year's User Survey. The respondents represent all six DOE Science Offices and a variety of home institutions: see Respondent Demographics.

The survey responses provide feedback about every aspect of NERSC's operation, help us judge the quality of our services, give DOE information on how well NERSC is doing, and point us to areas we can improve. The survey results are listed below.

You can see the 2005 User Survey text, in which users rated us on a 7-point satisfaction scale. Some areas were also rated on a 3-point importance scale or a 3-point usefulness scale.

The average satisfaction scores from this year's survey ranged from a high of 6.73 (very satisfied) to a low of 3.95 (neutral). See All Satisfaction Ratings.

For questions that spanned the 2004 and 2005 surveys the change in rating was tested for significance (using the t test at the 90% confidence level). Significant increases in satisfaction are shown in blue; significant decreases in satisfaction are shown in red.

Areas with the highest user satisfaction include the HPSS mass storage system, HPC consulting, and account support services:

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Areas with the lowest user satisfaction include batch wait times on both Seaborg and Jacquard, Seaborg's queue structure, PDSF disk stability, and Jacquard performance and debugging tools:

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

The largest increases in satisfaction over last year's survey are shown below:

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Only three areas were rated significantly lower this year: PDSF overall satisfaction and uptime, and the amount of time taken to resolve consulting issues. The introduction of three major ssytems in the last year combined with a reduction in consulting staff explain the latter.

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Survey Results Lead to Changes at NERSC

Every year we institute changes based on the previous year survey. In 2005 NERSC took a number of actions in response to suggestions from the 2004 user survey.

1. 2004 user comments: On the 2004 survey 37 users asked us to change the job scheduling policies on Seaborg, 25 requesting more support for midrange jobs.

   NERSC response: In early 2005 NERSC implemented two changes to the queueing policies on Seaborg:

   1. we reduced the scheduling distance between midrange and large jobs
   2. we gave all premium jobs a higher scheduling priority than regular priority large-node jobs (prior to this change the scheduling priority for premium midrange jobs was lower than that for regular priority large-node jobs).

   User satisfaction with Seaborg's batch queue structure increased by .39 points on the 2005 survey.

2. 2004 user comments: On the 2004 survey 25 users requested additional computing resources. In addition, another set of 25 users requested more support for midrange jobs.

   NERSC response: In August 2005 NERSC deployed Jacquard, a Linux cluster with 640 2.2 Ghz Opteron CPUs available for computations and a theoretical peak computational performance of 2.8 Teraflops. This was followed in January 2006 with Bassi, an IBM Power5 system with 888 processors available for computations and a theoretical peak computational performance of 6.7 Teraflops.

   User satisfaction with NERSC's available computing hardware increased by .24 points on the 2005 survey.

3. 2004 user comment: "Faster network connectivity to the outside world. I realize that this may well be out of your hands, but it is a minor impediment to our daily usage."

   NERSC response: During 2005 NERSC upgraded its network infrastructure to 10 gigabits per second. User satisfaction with network connectivity increased by .18 points on the 2005 survey.

4. 2004 user comment: "I want imagemagick on seaborg. Then I could make movies there, and that would complete my viz needs."

   NERSC response: NERSC installed imagemagick on Seaborg; it is also available on Jacquard and DaVinci.

Users are invited to provide overall comments about NERSC:

82 users answered the question What does NERSC do well?   47 respondents stated that NERSC gives them access to powerful computing resources without which they could not do their science; 32 mentioned excellent support services and NERSC's responsive staff; 30 pointed to very reliable and well managed hardware; and 11 said everything. Some representative comments are:

powerful is the reason to use NERSC

65 users responded to What should NERSC do differently?. The areas of greatest concern are the inter-related ones of queue turnaround times (24 comments), job scheduling and resource allocation policies (22 comments), and the need for more or different computational resources (17 comments). Users also voiced concerns about data management, software, group accounts, staffing and allocations. Some of the comments from this section are:

The most important improvement would be to reduce the amount of time that jobs wait in the queue; however, I understand that this can only be done by reducing the resource allocations.

A queued job sometimes takes too long to start. But I think that, given the amount of users, probably there would be no efficient queue management anyway.

Over-allocation is a mistake. Long waits in queues have been a disaster for getting science done in the last few years. INCITE had a negative affect on Fusion getting its science work done.

It's much better to have idle processors than idle scientists/physicists. What matters for getting science done is turnaround time. ...

... Interactive computing on Seaborg remains an issue that needs continued attention. Although it has greatly improved in the past year, I would appreciate yet more reliable availability.

Expand capabilities for biologists; add more computing facilities that don't emphasize the largest/fastest interconnect, to reduce queue times for people who want to runs lots of very loosely coupled jobs. More aggressively adapt to changes in the computing environment.

NERSC needs to expand the IBM-SP5 to 10000 processors to replace the IBM-SP3
Continue to test new machines, including the Cray products

NERSC needs to push to get more compute resources so that scientists can get adequate hours on the machine

51 users answered the question How does NERSC compare to other centers you have used?   Twenty six users stated that NERSC was an excellent center or was better than other centers they have used. Reasons given for preferring NERSC include its consulting services and responsiveness, its hardware and software management and the stability of its systems.

Twelve users said that NERSC was comparable to other centers or gave a mixed review and seven said that NERSC was not as good as another center they had used. The most common reason for finding dissatisfaction with NERSC is the oversubscription for its computational resources and the resulting long wait times. Among PDSF users, the most common dissatisfaction was with disk instability.

Here are the survey results:

1. Respondent Demographics
2. Overall Satisfaction and Importance
3. All Satisfaction, Importance and Usefulness Ratings
4. Hardware Resources
5. Software
6. Visualization and Data Analysis
7. HPC Consulting
8. Services and Communications
9. Web Interfaces
10. Training
11. Comments about NERSC