NERSC Helps Get the Physics out of KamLAND Data

The Standard Model of Particle Physics, which has successfully explained fundamental physics since the 1970s, predicts that neutrinos have no mass and come in three types or "flavors," electron, muon, and tau. But for the past four years, solar neutrino experiments at the Super-Kamiokande Observatory (Super-K) in Japan and the Sudbury Neutrino Observatory (SNO) in Canada have offered compelling evidence that neutrinos do have nonzero mass and oscillate between the three flavors while traveling from the sun to the earth, indicating that the three flavors are actually different states of the same particle. While the evidence was strong, a few physicists had nagging doubts about unexpected interactions between neutrinos and the sun's magnetic field. If the same oscillations could be shown for neutrinos from terrestrial sources-or anti-neutrinos, since anti-matter is the mirror image of matter-the doubts would be dispelled.

For this reason, physicists eagerly awaited the results from KamLAND, the Kamioka Liquid scintillator Anti-Neutrino Detector. Located adjacent to Super-K in a mine on the west coast of Japan's main island of Honshu and supported by an international collaboration (including DOE), KamLAND is the largest low-energy anti-neutrino detector ever built. KamLAND consists of a weather balloon 13 meters (43 feet) in diameter, filled with about a kiloton of liquid scintillator, a chemical soup that emits flashes of light when an incoming anti-neutrino (generated by nuclear reactors in Japan and Korea) collides with a proton. These light flashes are detected by a surrounding array of 1,879 photomultiplier light sensors which convert the flashes into electronic signals that are collected for analysis on computers (see figure).

Looking up at KamLAND's photomultipliers. When a neutrino collides with a proton in the liquid scintillator, a flash of light is emitted that is detected by the photomultipliers and converted into an electronic signal for computer analysis.

KamLAND experiments began generating about 200 gigabytes of data per day in January 2002. That amount of data would swamp the slow network connection between the detector and its research host, Tohuku University, so the data are stored on tapes, then driven by car in a seven-hour trip to the university, where the data are transferred from the tapes and stored for analysis.

After gathering data for six months, U.S. scientists running experiments at KamLAND had 800 tapes containing a vast amount of data, and they wanted to present initial results at conferences in September. But they had not determined how to get the data to the U.S.-Internet bandwidth is inadequate-or where in the U.S. they could store and analyze the data. A further complication was that the data were stored on LTO tapes (Linear Tape Open), a new and not yet widely used format.

Fortunately, the NERSC Center had an LTO system on loan from IBM for the Probe storage research project. NERSC's Mass Storage Group had just finished their evaluation of the system and were preparing to return it to IBM in July 2002 when they learned of the physicists' problem. NERSC reached an agreement with IBM to keep the system a few months longer, and the KamLAND tapes were shipped from Japan to Oakland. NERSC staff had already developed software for the LTO library to interface with our HPSS archive, so the KamLAND group began transferring their data to HPSS.

To avoid interfering with normal NERSC operations, the neutrino group did their data transfers at night and on weekends, increasing the HPSS data traffic by up to 80 percent per day. One weekend they managed to transfer 1.3 terabytes of data in one day. NERSC even reconfigured HPSS to write two data streams for the file size they were using. In all, more than 48 terabytes of data were transferred from the tapes to HPSS.

A second fortunate coincidence was that when the KamLAND data began to pour in, one of the largest PDSF users was idle. With some quick negotiations, the KamLAND group was able to use PDSF to analyze their data (as SNO had done before them), sometimes using the full 400-processor cluster. The KamLAND team was impressed by and grateful for the PDSF staff's efforts to make everything run as smoothly as possible for them.

The NERSC Center's client-oriented services and operational flexibility, together with its large-scale data storage and analysis resources, made possible the timely analysis and announcement of the KamLAND findings, which confirmed the earlier conclusions of the Super-K and SNO researchers. The results were presented at the International Workshop on Neutrinos and Subterranean Science in Washington, DC, September 19-21, 2002, and the 16th International Conference on Particles and Nuclei in Osaka, Japan, September 30-October 4, 2002. The results were also submitted for publication in Physical Review Letters.

NERSC remains poised to play its part in the revision of the Standard Model.