2004 User Survey Results

Response Summary

Many thanks to the 209 users who responded to this year's User Survey. The respondents represent all six DOE Science Offices and a variety of home institutions: see Respondent Demographics.

The survey responses provide feedback about every aspect of NERSC's operation, help us judge the quality of our services, give DOE information on how well NERSC is doing, and point us to areas we can improve. The survey results are listed below.

You can see the FY 2004 User Survey text, in which users rated us on a 7-point satisfaction scale. Some areas were also rated on a 3-point importance scale or a 3-point usefulness scale.

The average satisfaction scores from this year's survey ranged from a high of 6.74 (very satisfied) to a low of 3.84 (neutral). See All Satisfaction Ratings.

For questions that spanned the 2003 and 2004 surveys the change in rating was tested for significance (using the t test at the 90% confidence level). Significant increases in satisfaction are shown in blue; significant decreases in satisfaction are shown in red.

Areas with the highest user satisfaction include the HPSS mass storage system, HPC consulting, and account support services:

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Areas with the lowest user satisfaction include the IBM SP Seaborg's batch turnaround time and queue structure as well as services used by only small numbers of users (the math and visualization servers, grid services and training classes presented over the Access Grid):

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

The largest increases in satisfaction over last year's survey came from training classes attended in person, visualization services, the HPSS and Seaborg web pages and software bug resolution:

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

The areas rated significantly lower this year include the IBM SP, Seaborg, and available computing hardware:

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Survey Results Lead to Changes at NERSC

Every year we institute changes based on the previous year survey. In 2004 NERSC took a number of actions in response to suggestions from the 2003 user survey.

[Web site] Needs lots of improvement. Most pages cram lots of info in a single page, hard to find what you want, etc. Beyond the home page, the website has an 80's look.

NERSC response: NERSC reorganized its web site last year, merging the previous www.nersc.gov, hpcf.nersc.gov and pdsf.nersc.gov web sites into a newly designed www.nersc.gov web site. The design goals were to have one site that meets the needs of our users, our DOE sponsors, and the general public.

Four of the web interface ratings on this year's survey show increased satisfaction over last year: The HPSS and Seaborg ratings increased by 0.6 and 0.5 points, and the overall web site and software ratings increased by 0.3 points.

IBM's bugfixing is slow!
The compilers and debuggers need to be improved.

NERSC response: In this past year NERSC has established an excellent working relationship with IBM's compiler support group. Representatives of the compiler group now attend the NERSC/IBM quarterly meetings. This has resulted is quicker resolution of bug reports. We think that IBM is now doing an excellent jobs in resolving compiler problems. Also, in the past year compiler upgrades have resulted in compilers that produce better runtime performance. There are currently very few outstanding compiler bugs.

This year's rating for software bug resolution increased by one half point.

totalview is basically unusable.

NERSC response: NERSC has established a good working relationship with Etnus, the vendor that supports TotalView. TotalView upgrades this past year have resulted in more usability features, and Totalview can now debug a wider range of parallel codes and is more stable.

last year we received 8 complaints about TotalView and debugging tools. This year we received two such complaints. User satisfaction with performance and debugging tools on Seaborg increased by 0.3 points and on the PDSF by 0.5 points (these increases were not statistically significant, however).

You need to better develop the floating license approach and make it easier to use for remote users.

NERSC response: During 2004, NERSC finalized consolidation of license management for all visualization software hosted at the Center. The new system, which consists of a set of license servers, also supports remote use of licensed visualization software. See Remote License Services at NERSC.

Please offer more on-line video/on-site courses
It would be nice if NERSC can provide more tutorials.

NERSC response: In 2004 NERSC organized 20 user training lectures in 7 separate events. All were presented via the Access Grid and were captured as streaming videos (using Real Media streaming) so that users can replay them at any time. These lectures have been added to the tutorials page for "one stop" access to training materials. See NERSC Tutorials, How-To's, and Lectures.

more interactive nodes on pdsf
Would prefer faster CPUs at PC farm.
make it faster and bigger diskspace

NERSC response: The PDSF support team has made it possible to run interactively on the batch nodes (there is a FAQ that documents these procedures). They also recently purchased replacement login nodes that are being tested now and should go into production in December 2004. They are top of the line opterons with twice as much memory as the old nodes.

64 3.6 GHz Xeons were added to the PDSF cluster in November 2004. This is 25% more CPU's, and they are almost twice as fast as the older CPU's. We also added about 20 TB additional disk space.

The queue configuration should be returned to a state where it no longer favours jobs using large numbers of processors.
NERSC should move more aggressively to upgrade its high end computing facilities. It might do well to offer a wider variety of architectures. For example, the large Pentium 4 clusters about to become operational at NCSA provide a highly cost effective resources for some problems, but not for others. If NERSC had a greater variety of machines, it might be able to better serve all its users. However, the most important improvement would be to simply increase the total computing power available to users.

NERSC response: NERSC coordinates its scheduling priorities with the Office of Science to accommodate the Office's goals and priorities. This year, the office continued to emphasize capability computing, including large jobs and INCITE jobs. Since NERSC is totally subscribed this means that some other work receives lower priority. In 2004, the Seaborg queue structure still favored large jobs, and was over subscribed more than the previous year. However, several measures have been implemented which should help improve turnaround for all jobs:

 

• Per user run limits were decreased from six to three, and per user idle limits (the number of jobs that are eligible for scheduling) from ten to two. This provides fairer access to Seaborg's processors.
• The OMB (Office of Management and Budget) goal for FY 2005 is that 40 percent of Seaborg's cycled should be delivered to jobs using at least 1/8 of its computational processors (in FY 2004 this goal was 50 percent).
• In early calendar year 2005 NERSC will deploy a new Linux cluster with 640 dual 2.2 Ghz Opteron CPUs available for computations. The target workload for the cluster is jobs that do not naturally scale to 1/8th or more of the computational processors on Seaborg.
• Thanks to additional funding from the Office of Science, NERSC is in the process of procuring additional computational capability for the 06 and 07 allocation years.

The majority of this year's respondents expressed dissatisfaction with Seaborg turnaround time, and about one quarter were dissatisfied with Seaborg's queue policies. Ratings in these two areas dropped by -1.4 and -1 points. The rating for available hardware dropped by 0.5 points. In general, the users who ran smaller concurrency jobs were more dissatisfied than users who ran larger codes.

Users are invited to provide overall comments about NERSC:

118 users answered the question What does NERSC do well?   68 respondents stated that NERSC gives them access to powerful computing resources without which they could not do their science; 53 mentioned excellent support services and NERSC's responsive staff; 47 pointed to very reliable and well managed hardware; 30 claimed that NERSC is easy to use and has a good user environment; and 26 said everything. Some representative comments are:

NERSC supplies a lot of FLOPs reliably, and provides very competent consultants. It is a good place to use parallel codes that scale well on the available machines.

NERSC does a truly outstanding job of supporting both a small community of "power" users as well as a large community of mid-range users. Both are important, and, as a result of NERSC's success in supporting both communities, the Center facilitates an extraordinary amount of scientific productivity.

NERSC has excellent people working there. I'm VERY happy with everyone I've come across. People have been knowledgeable and professional. I compute at NERSC because it's really big. Seriously, the number of processors allows us to do research on problems that we simply cannot do anywhere else. In that regard I consider NERSC a national treasure. One really silly request, how about a NERSC T-Shirt! I'd buy one.

Overall, the services and hardware reliability are excellent. I think that NERSC sets the standard in this regard.

94 users responded to What should NERSC do differently?. The area of greatest concern is Seaborg job turnaround time and queue management policies. Forty five users expressed dissatisfaction with turnaround time and 37 requested a change in Seaborg job scheduling policies, of which 25 expressed concerns with favoring large jobs at the expense of smaller ones. Twenty five users requested newer processors or more computing resources overall. Fifteen expressed dissatisfaction with the allocations process. Some of the comments from this section are:

Change the batch queues so that ordinary jobs execute in days, not weeks.

The queue wait times have been extremely long (about 2 weeks recently), and this has almost completely stalled my research.

The current focus only on jobs which can exhibit high degrees of parallelism is, in my opinion obviously, misguided. Some problems of great scientific interest do not naturally scale to thousands of processors.

NERSC should return to its original mission of providing the production environment which allowed the scientists to maximize their research. That is NERSC should give satisfying the user priority over satisfying the DOE and the OMB.

Given the amount of computer time that I am allocated, I cannot make use of the large number of processors on Seaborg. Unless everyone is allocated enough time to make use of hundreds of processors, NERSC should give more consideration to providing resources for smaller codes.

Also, the job priority system discriminates against smaller jobs (less than 32 nodes) - i.e. MAJORITY of users!

For the last 24 years NERSC has been the place where "I could get things done". With the initiation of the INCITE program that changed. The machine was effectively taken over by the 3 INCITE groups and work at NERSC stopped. After the upgrade my large calculations no longer run at NERSC and I had to move those computations to a p690 in Hannover, Germany.

The computer code I use becomes more complex from day to day to use the best physics you can. However this increases the computing time. The great management and support at NERSC combined with new hardware would be an irresistible package.

NERSC has done an outstanding job of serving the community. In order for this to continue, NERSC needs continued support from the DOE for its staff and the services they provide, and NERSC needs support for a new high end system to replace seaborg.

Similarly, I've described my problems with the ERCAP proposal process. I feel it gives short-shrift to science, and focuses on code optimization to the exclusion of scientific returns.

I think the INCITE program was ill conceived. Betting that the performance of a tiny subset of the scientific community will payoff enormously better than the community as a whole seems to me like trying to time the stock market. It may work once, but the opportunity costs are enormous.

77 users answered the question How does NERSC compare to other centers you have used?   Thirty nine users stated that NERSC was an excellent center (no comparison made) or was better than other centers they have used. Reasons given for preferring NERSC include excellent hardware and software management, good user support and services. Seven respondents said that NERSC was not as good as another center they used. The most common reason for finding dissatisfaction with NERSC is job turnaround time.

Here are the survey results:

1. Respondent Demographics
2. Overall Satisfaction and Importance
3. All Satisfaction, Importance and Usefulness Ratings
4. Hardware Resources
5. Software
6. Security and One Time Passwords
7. Visualization and Data Analysis
8. HPC Consulting
9. Services and Communications
10. Web Interfaces
11. Training
12. Comments about NERSC