Memorial Gathering Pending for NERSC’s Michael Welcome

A celebration of life is pending for Michael Welcome, a member of NERSC’s Mass Storage Group, who collapsed at work on Thursday, Jan. 30, and subsequently died. Welcome spent his entire career working for computing organizations at Lawrence Berkeley and Lawrence Livermore national laboratories. He was 56.

During his 30-year career, Welcome made significant contributions in the areas of applied mathematics, system administration and improving the efficiency of high performance computing systems.

Welcome was on a six-month assignment with NERSC  at LLNL when DOE made the decision to move the computing center to Berkeley Lab in 1996. Both he and his wife Tammy elected to make the move. In Berkeley, Welcome was a member of the team that administered the new Cray supercomputers, including the Cray T3E, which ushered NERSC into the world of massively parallel computing. Tammy Welcome was leader of NERSC’s Scientific Computing Group, which provided “red carpet” levels of support for NERSC users tackling grand challenges in computational science.

During his career at Berkeley Lab, Welcome also worked for the Future Technologies Group in the Computational Research Division. About seven years ago, Welcome returned to NERSC as a member of the Storage Systems Group.

Although he was primarily a developer for the HPSS archival storage system, Welcome also took on responsibilities on the production side. HPSS, of which NERSC is one of the original consortium members, has two main software components – the core server and the movers. It was the movers, which move the data back and forth in the system, which Welcome worked on. Co-worker Wayne Hurlbert calls the mover “a huge, complicated part of the HPSS code, one of the significant pieces.”

Hurlbert said he came to rely more and more on Welcome for solving production problems. “I’d tell him ‘I’m having this problem’ and he’d snoop around in the code and find the answer.”

In this 1998 photo, Jonathan Carter, Brent Draney and Mike Welcome look over a utilization report for NERSC's Cray T3E supercomputer. Welcome wrote the scheduling software to balance jobs needing fast turnaround with those that had longer time-limits.

But what Hurlbert enjoyed most about working with Welcome was seeing him learn more about the HPSS hardware – tape libraries, IBM servers, disc arrays and the like.

“When he joined the group, he saw himself as a developer and really didn’t want to be involved in the production side. I think he was apprehensive about the hardware,” Hurlbert said. “The great thing is, he came to love it all – he started to enjoy it and learn more and more. Eventually, he even built a substantial HPSS test system from scratch for other members of the consortium to use.”

Hurlbert often ate lunch with Welcome and they talked about things ranging from Hurlbert’s daughter’s struggles with high school calculus to dealing with aging parents.

“I don’t remember a single day when he was in a bad mood,” Hurlbert said. “He was always cheerful and helpful – all that good stuff you want in a colleague.”

Welcome began his career at LLNL when he was hired fresh out of university in 1983 as a member of the SISAL project to develop a new programming language for parallel computing.

“He contributed many innovative ideas to the first implementations of this language, with an emphasis on complier optimization techniques,” said Jim McGraw, who led the SISAL project. “Mike worked very hard and really made a tremendous impression on everyone involved in the project.”

In 1986, Phil Colella, now head of CRD’s Applied Numerical Algorithms Group, was hired at LLNL to create a new Applied Mathematics Group. His first hire was Welcome.

As a member of the Future Technologies Group, Welcome developed a block-structured hydrodynamics code for Unified Parallel C, then applied the code to the UPC logo, with this image being the result.

“Mike was the software architect for the first few generations of our Adaptive Mesh Refinement codes, including what eventually became the BoxLib framework written in C++“ Colella said. “Mike’s work was one of the first demonstrations that C++ could be a useful programming language for high-performance scientific computing.”

In 1995, Welcome expressed a desire to learn about computer security and received a six-month system admin training assignment, working for NERSC computer security lead John Allen.

“So his time with NERSC at LLNL was a six-month training assignment, then off to NERSC at Berkeley, where he was hired as the first non-management member of the LBNL NERSC system admin group,” Allen recalled. “I certainly remember him, as a smart guy, a quick study and a lot of fun to work with.”

With NERSC’s move to Berkeley Lab and the deployment of the Cray T3E, Welcome wrote the scripts that allowed the Computational Systems Group to schedule short-time limit jobs in the daytime, then move them out of the system to run the longer-time limit jobs overnight. This “time-slicing” application allowed the center to balance jobs needing faster turnaround with those that didn’t, improving overall efficiency. He eventually become the project lead for the T3E and in recognition of his work on the T3E and other projects at NERSC, he received three outstanding performance awards.

Tina Declerck worked with Welcome in the systems group and recalled him as someone who would take on new challenges with enthusiasm, was very detail-oriented and had a great sense of humor and a wonderful laugh—even when the joke was on him.

“In general, systems people don't like accounting so when he was working on the CUB (Cray Unified Banker) real-time accounting system, he was frequently given a hard time by others who worked in the same office,” Declerck said. “They would tell him he was ‘contaminating’ the office with accounting bits. One day after we received some new J90s, he came into the office to find a large box surrounding his chair and computer. It was labeled CCC (CUB Containment Cube). Being a good sport, he climbed through the submarine type doorway and worked in there (at least for a little while).”

While a member of the Future Technologies Group, Welcome worked with Associate Laboratory Director Kathy Yelick, who was then leader of the group. He worked on GASnet, the communications layer for the Unified Parallel C (UPC) language, which Cray used in its implementation of UPC on the Cray XT3 and XT4 systems. Welcome also used his AMR experience to create a block-structured hydrodynamics code in UPC.

“Mike was known for his ability to tackle complex software projects and design and build production-quality solutions,” Yelick wrote to staff in announcing Welcome’s death. “But even more importantly, he was a great colleague and friend to many in the Berkeley Lab community. Mike was the consummate team player -- always taking on the part of the project that most needed to be done and offering thoughtful advice to others in the team.”

Outside the Workplace

On the personal side, Mike enjoyed cooking, exercise, and taking care of his African Grey parrot, Pookie.

Bicycling was a particular pleasure for him – he commuted daily by bicycle from his home off of Park Blvd. and the Warren Freeway to the Oakland Scientific Facility. On vacations to Sea Ranch, a favorite outing would be to cycle with some of his friends the eight miles into the hills to a local winery, while others would follow behind in a car with a picnic lunch, along with Pookie.

Welcome, who grew up in South Milwaukee, earned his bachelor’s degree in mathematics at the University of Wisconsin, Parkside, and his master’s in mathematics from the University of Illinois, Urbana-Champaign. He was enrolled in a graduate program in computer science at Illinois, but ultimately decided not to finish the program. It was also at UIUC that he met his wife, Tammy, who has also been a long-time staff member at the two labs.

A Final Thought

Perhaps Jeff Long, who knew Welcome quite well during their time at LLNL, summarized things best with this thought: “I remember Mike as someone who enjoyed life, with an easy laugh and personality befitting his surname. He worked hard, immersing himself in the challenge du jour, and he played hard, enjoying the moment and the people around him.”