2006 User Survey Results

Hardware Resources

Legend:

Hardware Satisfaction - by Score

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Hardware Satisfaction - by Platform

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Hardware Comments:   37 responses

Overall Hardware Comments:   12 responses

Need more resources

Hopefully franklin will fix the long queue wait times.

Please get more computers.

Please assign more space of hardware to its users.

Queue comments

The queue structure could have some improvement, sometimes jobs requiring many nodes make the queue slow but I am sure that you are looking into this.

I run the CCSM model. The model runs a relatively small number of processors for a very long time. For example, we use 248 processors on bassi. On Seaborg, we could potentially get one model year/wallclock day. Since we usually run 130 year simulations, if we had 248 processors continuously, it would take 4.5 months to run the model. We didn't get even close to that. Our last seaborg run took 15 months real time, which is intolerably slow.

Bassi runs faster. On bassi, we get roughly 10 model years/wallclock day, a nice number. So it's cheaper for us to run on bassi, and better. bassi is down more frequently, and I get more machine related errors when running on bassi.

On both machines your queue structure does not give us the priority that we need to get the throughput that we have been allocated. For now it's working because bassi isn't heavily loaded. But as others leave seaborg behind and move onto bassi, the number of slots we get in the queue will go down, and we'll find ourselves unable to finish model runs in a timely fashion again.

Submission of batch jobs is not well documented.

The most unsatisfactory part for me is the confusing policy of queueing the submitted jobs. In an ideal world, it should be first come, first serve with some reasonable constraints. However, I frequently find my jobs waiting for days and weeks without knowing why. Other jobs of similar types or even those with low priority sometimes can jump ahead and run instantaneously. This makes rational planning of the project and account management almost impossible. I assume most of us are not trained as computer scientists with special skills who can find loop holes or know how to take advantages of the system. We only need our projects to proceed as planed.

Good overall

Bassi is great. The good network connectivity within NERSC and to the outside world and the reliability of HPSS make NERSC my preferred platform for post-processing very large scale runs.

Hardware resources at NERSC are the best I have used anywhere. NERSC and in particular Dr. Simon Horst should be congratulated for setting up and running certainly one of the best supercomputing facilities in the world.

Other comments

NERSC could have a more clear and fair computing time reimbursement/refund policy. For example (Reference Number 061107-000061 for online consulting), on 11/07/2006, I had a batch job on bassi interrupted by a node failure. The loadleveler automatically restarted the batch job from beginning, overwritting all the output files before the node failure. Later I requested refund of the 1896 MPP hours wasted in that incident due to the bassi node failure. But my request was denied, which I think is unfair.

I have not done extensive comparison on I/O and network performance. Hopefully, next year I'll be able to provide more useful information here.

every nersc head node should be running grid ftp

every nersc queuing node should be running GT4 GRAM

Comments by Bassi Users:   7 responses

Charge factor

the charge factor of 6 for bassi is absolutely ridiculous compared to jacquard. it performs only half as good as jacquard.

We have consistently found (and NERSC consultants have confirmed) a speedup factor of 2 for Bassi relative to Seaborg on our production code. Because the charge factor is 6, and because we see a speedup of 3 on Jacquard, Bassi is currently not an attractive platform for us, except for extremely large and time-sensitive jobs.

Queue comments

I really like Bassi; however, the availability of Bassi for multiple small jobs is difficult, since only 3 jobs from a user can run at a time; this is difficult to deal with when I have many of these jobs, even when the queues are rather small.

I don't understand why bassi has restriction on using large number of nodes (i.e., > 48 nodes requires special arrangement.)

Disk storage comments

Scratch space is small. My quota is 256GB. Simulation we are currently running are on a 2048^3 and we solve for 3 real variables per grid point giving a total of 96 GB per restart dataset. After 6 hours of running (maximum walltime on bassi), we continue from a restart dataset. But sometimes, we need to do checkpointing (i.e. generate the restart files) half way through the simulation. This amount of being able to hold 3 datasets (initial conditions, half-way checkpoint and at the end) which is not possible. Moreover, for simulations of more scientific interest we solve for 5 variables per grid point. The restart dataset in this case is 160 GB. This means that we cannot run, checkpoint and continue. This quota also prevents fast postprocessing of the data when several realizations of the fields (many datasets) are needed to get reliable statistical results.

I could not unzip library source code on Bassi because it limited the number of subdirectories I could create. That machine is useless to me unless I can get Boost installed. ...

Login problems

Logon behavior to Bassi can be inconsistent with good passwords sometimes being rejected, and then accepted at the next attempt. Molpro does not generally work well on multiple nodes. This is not too much of a problem on Bassi as there are 8 processors per node, but better scaling, with respect to number of nodes, is possible for this code.

Comments by DaVinci Users:   2 responses

I use multiple processors on DaVinci for computations with MATLAB. The multiple processors and rather fast computation are extremely useful for my research projects on climate and ice sheet dynamics. Via DaVinci NERSC has been a huge help to my research program.

... The machine I have been able to effectively use is Davinci because it has Intel compilers. NERSC support has not been helpful at all in getting my software to run on various machines.

Comments by Jacquard Users:   2 responses

On jacquard, it might be nice to make it easier for users who want to submit a large number of single-processor jobs as opposed to a few massively parallel jobs. This is possible but in the current configuration, the user has to manually write code to submit a batch job, ssh to all the assigned nodes, and start the jobs manually. Perhaps that is intentional, but the need does arise, for instance when it is possible to divide a task such that it can be run as 1000 separate jobs which do not need to communicate.

Jacquard is much harder to use than the IBM-SP's...

Comments on Network Performance:   3 responses

we produce output files faster than we can transfer them to our home institution, even using compression techniques. this is usually not an issue, but it has been recently.

Network performance to HPSS seems a bit slower than to resources such as Jacquard. Not sure of how much of a hit this actually is. Just an impression.

It is quite possible that I am unaware of a better alternative, but using BBFTP to transfer files to/from Bassi from/to NSF centers I see data rates of only 30-40 MB/sec. This isn't really adequate for the volume of data that we need to move. For example, I can regularly achieve 10x this rate between major elements of the NSF Teragrid. And that isn't enough either!

Comments about Storage:   3 responses

... The HPSS hardware seems great, but the ability to access it is terrible [see comments in software section].

The largest restriction for us is usually disk and storage; we have been able to work with consulting to make special arrangements for our needs, which have been very helpful.

The low inodes quota is a real pain.

Comments by PDSF Users:   6 responses

Diskservers at PDSF are faring reasonably well, but occasional crashes/outages occur. The move to GFPS has made disk more reliable, but still occasional crashes occur. These sometimes mean that PDSF is unavailable for certain tasks for up to several days (depending on the severity of the crash). This should be an area of continued focus and attention.

My biggest problem with using PDSF has always been that regularly the nodes just freeze, even on something as simple as an ls command. Typically I think this is because some user is hammering a disk I am accessing. This effects the overall useability of the nodes and can be very frustrating. My officemates all use pdsf, and we regularly inform each other about the performance of PDSF to decide whether it is worth trying to connect to the system at all or if it would be better to wait until later.

Interactive use on PDSF is often too slow.

I think that the NGF and the general choice for GPFS is a great improvement over the previous NFS-based systems. I am worried that in recent months we have seen the performance of the PDSF home FS drop significantly.

The switch to GPFS from NFS on PDSF seems to be overall a good thing, but there are now occasional long delays or unavailability of the home disks that I don't like and don't understand...

PDSF has got way too few SSH gateway systems, plus they seem to be selected by round-robin DNS aliasing and thus it is entirely possible to end up on a host with load already approaching 10 while there are still machines available doing absolutely nothing; what I normally do nowadays is look at the Ganglia interface of PDSF and manually log in to the machine with smallest load. There is a definite need for proper load balancing here! Also, it may make sense to separate interactive machines into strict gateways (oriented on minimal latency of connections, with very limited number-crunching privileges) and interactive-job boxes (the opposite).

Comments by Seaborg Users:   3 responses

Seaborg is a little slow, but that is to be expected. The charge factors on the newer, faster machines are dauntingly high.

Our project relies primarily on our ability to submit parallel jobs to the batch queue on Seaborg. To that end, the current setup is more than adequate.

There have been persistent problems with passwords (to seaborg) being reset or deactivated. In one case my password was deactivated but I was not informed (via email or otherwise. This may have been the result of a security breach at our home institute. Several hours are lost trying to regain access to NERSC.