2010/2011 User Survey Results

Response Summary

A special thanks to the 411 users who responded to the 2011 survey, which was conducted from June 6-30, 2011. This represents a 13.1 percent response rate from the 3,130 users who had been active in the 12 months prior.  Your responses are important to us because they provide feedback about every aspect of NERSC's operation, help us judge the quality of our services, give DOE information on how well NERSC is doing, and point us to areas we can improve.

The survey strives to be representative of all NERSC users. The hours used by the respondents represent about 71 percent of all MPP hours (used on Hopper, Franklin or Carver) used at the time the survey closed.  MPP respondents were classified according to their usage:

• 60 respondnts had used over 1.5 million hours, generating a response rate of 77% from this community of "large MPP users".
• 131 respondents had used between 100,000 and 1.5 million hours, generating a 38% response rate from the "medium MPP users"
• 149 respondents had used fewer than 100,000 hours, generating a 14% response rate from the "small MPP users".
• 70 respondents were not MPP users - they were either Principal Invesigators or project managers supervising the work of their NERSC users, or they were users of other NERSC resources, such as HPSS, PDSF, Euclid, or Dirac.

On this survey users scored satisfaction on a seven-point scale, where “1” is “very dissatisfied” and “7” indicates “very satisfied.”  The average satisfaction scores from this year's survey ranged from a high of 6.79 to a low of 5.16; the average score was 6.29.

For questions that spanned previous surveys, the change in scoring was tested for significance (using the t test at the 90% confidence level). Significant increases in satisfaction are shown in blue; significant decreases in satisfaction are shown in red.

Areas with Highest User Satisfaction

Areas with the highest user satisfaction are those with average scores of more than 6.5.  NERSC resurces and services with average scores in this range were:

• Global homes, project and scratch
• HPSS mass storage system
• Account support and technical consulting
• Services and Security
• NERSC overall
• Carver
• NERSC's internal network

The top 6 of the 18 questions that scored over 6.5 are shown below.

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Areas with Lowest User Satisfaction

Areas with the lowest user satisfaction are those with average scores of less than 5.5.

Significant Increases in Satisfaction

16 questions scored significantly higher in 2011 compared with 2010.  NERSC has never before had so many increases in satisfaction!

Most of the significant improvements from 2010 were related to the Hopper transition from a small Cray XT5 to a large Cray XE6 (No. 5 on the November 2010 TOP500 list) and NERSC training initiatives.

The two lowest scores on the 2010 survey — Hopper and Franklin batch wait times — improved significantly in 2011, thanks to the new Hopper system.

Significant Decreases in Satisfaction

The largest decrease in satisfaction came from batch wait times on the Carver cluster.  NERSC plans to address this by increasing the size of the Carver system with hardware from the Magellan project, which will conclude in late 2011.

Satisfaction Patterns for Large, Medium and Small MPP Respondents

The MPP respondents were classified as "large" (if their usage was over 1.5 million hours), "medium" (usage between 100,000 and 1.5 million hours) and "small". Satisfaction differences between these three groups are shown in the table below.

The top increases in satisfaction for the large and medium MPP users were for Hopper, analytics, and training.  For the smaller MPP users the three top areas were the PDSF physics cluster, Franklin, and Hopper.

Survey Results Lead to Changes at NERSC

Every year we institute changes based on the previous year survey. In 2010 and early 2011 NERSC took a number of actions in response to suggestions from the 2009/2010 user survey.

On the 2009/2010 survey NERSC training workshops received the third lowest score, with an average satisfaction rating of 5.38 / 7.  In response, NERSC renewed its training efforts in 2010. In additional to its traditional training during the annual NERSC Users Group (NUG) Meeting, NERSC conducted a two-day workshop for Cray XE6 users at its facility in Oakland, joining with members of the Cielo team from Los Alamos National Laboratory and staff from Cray, Inc. Both the NUG training and the XE6 training were concurrently broadcast over the web. In addition, NERSC held a number of web-based training events (webinars) through 2010–2011. In all, NERSC put on eight events for its users from July 1, 2010 to June 30, 2011, with an aggregate attendance of about 375.   

NERSC’s users responded positively to the training classes as indicated by the satisfaction score increase of 0.51 points on the 2010/2011 User Survey. Additional surveys were conducted after each class, with 97.8% of respondents indicating that the training was “useful to me.”

Data analysis and visualization was another area which received  lower satisfaction ratings on the 2009/2010 survey.  In 2010 NERSC hired a new consultant to enhance NERSC’s visualization and data analysis software and services. A significant accomplishment was the robust implementation of an NX server that enabled remote X-Windows based graphical software. NERSC aggressively publicized this new service to its users and held training sessions. NERSC also re-organized the analytics materials on the new web site. As a result of these efforts, users’ satisfaction as measured by the 2010/2011 user survey increased significantly for three data analysis and visualization ratings.

In 2010 NERSC also replaced its data analysis/visualization machine, DaVinci, with a new Sun Sunfire platform, Euclid. User satisfaction with the new system was evident in the user survey results, with Euclid scoring a 6.10/7 satisfaction score, an increase of 0.27 points over DaVinci’s 2009/2010 rating.