2005 User Survey Results

Visualization and Data Analysis

Where do you perform data analysis and visualization of data produced at NERSC?

Are your data analysis and visualization needs being met? In what ways do you make use of NERSC data analysis and visualization resources? In what ways should NERSC add to or improve these resources?

[Read all 55 responses]

Yes, data analysis needs are being met / positive comments:   22 responses

Yes.

I would like to acknowledge a great deal of help by Cristina Siergerist.

Yes, they are met. Although reading the question, I don't use most of the visualization tools mentioned.

NERSC is fully equipped with all the facilities which a analyst uses.

I use PDSF often to analyze the STAR data. I hope the system at interactive and computing nodes are stable.

Our data analysis needs are based on the ROOT framework

I do mostly batch generation of plots from HDF5 files using GnuPlot.

yes, the data visualization group has been very useful

The visualization packages that I use at NERSC depend on IDL, and the output file processing depends on netcdf (I use NCO to manipulate files). ...

Visualization tools with GEANT4 applications

I use IDL on Seaborg and Jacquard.

Most with IDL, visualization group also try to use the other visualization software. I have got good consulting help from the visualization group. ...

Yes. I use IDL, occasionally on the serial queue if I have a large data set.

I am moving IO to HDF5 format. I will see how good the visualization software is dealing with this format (presumably OK vis-vis IDL)

have previously had accounts on escher for analysis and visualization but mostly used seaborg for access as it seemed the most stable environment. have yet to use DaVinci.

Yes... great to see Visit in use.

I use serial jobs on Seaborg for some trivial post-processing of NERSC-generated data, but otherwise do all analysis and visualization on my local machines.

The consultants have been very helpful with visualization needs

My data analysis and visualization needs are being met very. I use DaVinci and visualization software developed together with the vis group. PLEASE KEEP the vis group running they do a very good job!!!!!!

I use the PDSF queues for processing high energy physics data with my experiment's software. I am satisfied with the speed at which my jobs are processed.

I use IDL on davinci

I use NERSC to visualize some very large datasets. I have some visualization software written within our group, which I sometimes use on Davinci. I need to use NERSC for special rendering options and publication-quality visualization. For this I have used the help of a NERSC consultant and was very satisfied. The constants make lots of good suggestions for viewing the data and ran the software to produce images that we have used several times in publications. They also helped me get started with graphics packages that I could use on my own.

Do vis locally / don't need:   14 responses

Our group does all visualization locally (LINUX / MAC OSX) using open-source programs such as plotmtv and xmgrace.

I don't make use of data analysis and visualization facilities at all. I find it is more expedient to relegate that responsibility to my local machines.

I do all of my visualization elsewhere.

Do not use Data visualization at NERSC

I do my analysis at the local computer

I have not used the data analysis/vis tools- it's something I'd like to try, but haven't gotten around to. I don't know the tools yet, and haven't had the time to learn.

Don't need analysis and visualization at NERSC.

Not needed

the visualization we have used so far this past year has been done at NAVO and ERDC centers (DoD). It was easier for us to keep using these consultants at DoD ....

I don't use NERSC for this

I do not currently use visualization facilities at NERSC. However, I am planning to do so in the future, once I have assembled all the needed pieces at my end.

I currently do not use NERSC analysis and visualization services.

Requests for additional services:   12 responses

Requests for additional software

Our group works with the visualization group and we have a definite need for visualization. Would like to have information visualization software tools such as Spotfire to be available at NERSC. Ideally, it would be good to integrate information visualization with our system.

... It would be helpful to have LaTeX available for a specific post-processing step, but I don't see it anywhere at NERSC.

NERSC response: Please submit a software request.

My data analysis and visualization needs have not been met. Ferret could not be used on seaborg. At least, I would like NERSC adds Ferret on seaborg. ...

NERSC response: The older AIX Ferret binary from the Ferret download site does not work on seaborg (operating system incompatibility). We have contacted the Ferret developers who are working on a current AIX port in conjunction with creating a 64-bit version of Ferret for many different operating system versions. The Ferret developers have not provided a target date when we might expect Ferret to be ready. Meanwhile, we have installed a 32-bit version of Ferret on DaVinci; that version is operational.

not all of the needs are met. using idl for data visualization and simple pre/post analysis. please add 64-bit visualization software if possible

NERSC response: The operating systems on Bassi, DaVinci, Jacquard and Seaborg are all 64-bit; most of the visualization software installed on those systems is a native 64-bit version. On DaVinci, there are a few commercial applications whose vendors have not yet provided a native IA64 port (Matlab, IDL) but fully native 64-bit versions of those applications are installed on other platforms. If there is a specific visualization application you need for your project, please submit a software request.

In order to process data files, I need the NCO operators, which can manipulate netcdf files. If these were available at NERSC, I would do all my analysis there (using NCO, then MATLAB).

NERSC response: netcdf and nco are installed on all NERSC computers (Seaborg, Bassi, Jacquard and Davinci). To access nco (the netcdf operators) first load the module "netcdf". MATLAB is available on Jacquard and DaVinci.

The main data analysis tools I use are CDAT and VCDAT developed at LLNL. I am not sure if are supported now. If they are supported on NERSC platform, I would use them.

NERSC response: Please submit a software request.

No, there are programs that we need that are not on the systems.

Requests for consulting assistance

I have not used NERSC for my visualization needs for several years although I relied heavily on them in years past. I was very satisfied with the computers and consultants at the time. With the large user base of the NIMROD code and its emergence as the defacto standard for computational plasma physics it would be very helpful if NERSC would devote some consultant resources to NIMROD visualization. This would provide a much needed resource to the fusion energy community.

I do all my post-processing and data viz on my Mac, but this is getting out of hand with my recent interest in 3D data. I'll contact NERSC in the near future for help.

I do all my own analysis. I need to figure out what analysis capability NERSC has to offer and how I may use it.

I like to use Matlab for rough graphic purpose. I tried Matlab at nersc. It seemed it didn't work well for me. ...

Other

it seems odd that the 'vis machine' has no graphics pipes. it's faster for us to render our visualizations on our own machines that have OpenGL than it is to use the CPU renderer on davinci.

NERSC response: The Center has future plans to upgrade DaVinci in the 2007 time frame in terms of both processor count as well as the addition of graphics hardware.

Moving data is an inhibitor / disk issues:   7 responses

... Also, large amount of data needs to be moved from seaborg to davinci for analysis. It is very inconvenience. ...

... It is also not convenient if we need to transfer data from seaborg to other nersc machines for data analysis.

... I don't like to transfer data into DaVinci to do the visualization.

Used mostly interactive runs (IDL) on seaborg and struggled with the 30 mins time limit and memory limitations. Recently switched to using DaVinci what solved both these problems. The need to copy files from seaborg to DaVinci is an issue here as well as different endiannes. The common file systems as the one between Jacquard and DaVinci is a great way to go from my point of view.

NERSC response: The NERSC Global Filesystem has been deployed to address the above issues. It is a large, shared filesystem that can be accessed from all the computational systems at NERSC.

Mostly yes, but diskpool storage remains a problematic issue. [PDSF user]

... The diskspace of scratch is not big enough. [Seaborg user]

I wish there were automatic backups for DaVinci

Need more information / training:   4 responses

We might like to use these services, but are mostly unaware of what is available.

NERSC response: The NERSC Visualization services are documented on the web. If the documentation doesn't answer your questions, please contact a NERSC consultant.

A more helpful FAQ on using (especially, the remote fonts) Mathematica (on Jacquard or Davinci) would be extremely helpful. Local system administrator is still unable to configure my Linux machine so that the fonts work correctly over X11. As a result, this somewhat expensive software is unusable. Several of my coworkers, have exactly the same problem; none of them have been able to resolve it.

I think it would be useful to have a greater awareness of what visualization resources are available. Users may not be aware of what is available to them unless they specifically ask a consultant.

I mostly do my data analysis and visualization on local computers. Maintenance of serial queues on Seaborg and other newer platforms (Jacquard, Bassi) is a good idea. Classes on newer visualization software (e.g., Ensight) would be helpful.

Network speed is an inhibitor:   3 responses

Network connectivity between PDSF and LBL building 50 is so poor that data visualization via X is not practical (nor is even an X based editor session such as xemacs). I have to copy all my data to local machines for reasonable visualization.

No. I have mostly done my data analysis on seaborg and davinci. On seaborg, I use IDL and it works very well. On davinci, I use Matlab and it is slow. ... I want to use the software Visit on my windows XP system but it is slow too. I am not sure it is because of the slow network or davinci is slow itself.

The network latency to my work place (East Coast) is too high and makes software with heavy graphical interfaces almost impossible to use. Debugging sessions with Totalview, for example, are very frustrating. IDL is usable as long as the interface stays simple. We do all the intensive visualization locally.

NERSC response: The adverse effect of network latency on interactive remote visualization is a well-known phenomenon. While we can't do anything about reducing network latency, we are evaluating use of an emerging technology for remote desktop sharing (a la "VNC") that will provide some help as it eliminates most of the round-trip Xlib protocol traffic between NERSC and the remote site; such round-trip Xlib traffic significantly contributes to usability and performance issues - the new solution does not eliminate it completely but should result in a noticeable performance improvement. We will post more information when it becomes available later in 2006.

HPC Consulting

Legend:

Satisfaction with HPC Consulting

7=Very satisfied, 6=Mostly satisfied, 5=Somewhat satisfied, 4=Neutral, 3=Somewhat dissatisfied, 2=Mostly dissatisfied, 1=Very dissatisfied

Comments about Consulting:   46 responses

Good service:   20 responses

Nice job!

The consultants do well in a tough job.

NERSC's technical help has been excellent. Each issue I brought to their attention was dealt with quickly, many times within the same day, if not while I was on the phone with them.

good work and thank you.

I've only talked to them about password resets and when some machines are going to be up, but they were helpful.

really very helpful and knowledgeable

I think the consultants at NERSC are doing a great job!

I continue to be impressed with the competence and helpfulness of the NERSC consulting staff.

This group is great. They have been very helpful.

They are good

My experiences with consulting at NERSC have been uniformly positive

In general I find consulting at NERSC superb.

NERSC consultants are always very helpful and prompt at replying. I have accounts on a number of other computing centers and rank NERSC the highest in this regard

Very good

The high-quality online consulting system is one of the most valuable aspects of NERSC.

The consultants are excellent and set NERSC apart from other major computing centers

Great Job, folks. Keep up the good work!

NERSC has by far the best consulting service of all the computing centers that I know.

NERSC has the highest level of consulting support that I can imagine. Consultants are knowledgeable, friendly, and have dedication to customer service. If only everybody else in other parts of my life were as good!

I have found the consultants very useful, prompt to reply, and very helpful with whatever problem I encountered. Thanks!

Mixed evaluation:   3 responses

Iwona is fantastic at responding to and resolving PDSF problems in a timely manner. My experience with non-PDSF support staff has been much worse -- slow response for even simple tasks like rebooting a bad diskvault or compute node. The advertized 4 hour turnaround for online help requests seems to only be true for Iwona.

I'm very happy with the consulting in general. Sometimes (if people are on vacation, on travel, during weekends), there seems to be a lack of experts on call/duty for pdsf. The staff on the 'general' hotline is not necessarily very helpful for pdsfok at) the problems which arise during these times. problems. I don't blame them but there should be somebody in reach to resolve (or at least lo

I mostly use PDSF. When the PDSF consultant is in everything is great. Iwona is a wonderful help to all of us at KamLAND and when she is on travel it is noticeable because the performance of PDSF is often noticeably worse and the response to questions and problems (usually from consultants not specializing in PDSF), while well-intentioned has sometimes not been particularly useful. If there was one complaint I have, its that there does not appear to be enough staff to cover the consulting needs for PDSF. I get the impression that the personnel resources for PDSF are stretched very thin.

NERSC response: For the above three comments: Iwona is a part-time sysadmin for PDSF. The other NERSC consultants do not perform system administrations tasks on the PDSF. NERSC will work to improve the PDSF response time for such issues.

Unhappy:   3 responses

somewhat improve on the C++ knowledge

NERSC response: We recognize the need for more C++ expertise and expect that newly hired consultants will be knowledgeable in C++.

As a followup to my last time, I still have not got a serious answer to the problems. The consultants waited to so long I gave up working on Jacquard for a while. Then they said everything was updated and changed so I should try to solve my problem again and see if it persists. Well, when I have some free time, I'll give it a go, but it seems that if I run into a problem, I might be on my own anyways.

I had bad luck with fortran compilers on Jacquard. I use OpenMP+MPI parallelization and the compilers were not able to handle OMP directives. It is not clear why more robust pg compilers were not used for this linux computer.

NERSC response: For the above two comments: Vendor support of the user environment and software on jacquard is not as tightly integrated as the support for Seaborg and Bassi and resolutions of these issues has taken longer. However, the consultants could have done a better job of at least letting the users know that we have not forgotten the issues, and our goal is to do better in this area.