Brian Friesen

The newest NERSC supercomputer Cori is a Cray XC40 system consisting of 2,388 Intel Xeon Haswell nodes and 9,688 Intel Xeon‐Phi “Knights Landing” (KNL) nodes. Compared to the Xeon‐based clusters NERSC users are familiar with, optimal performance on Cori requires consideration of KNL mode settings; process, thread, and memory affinity; fine‐grain parallelization; vectorization; and use of the high‐bandwidth MCDRAM memory. This paper describes our efforts preparing NERSC users for KNL through the NERSC Exascale Science Application Program, Web documentation, and user training. We discuss how we configured the Cori system for usability and productivity, addressing programming concerns, batch system configurations, and default KNL cluster and memory modes. System usage data, job completion analysis, programming and running jobs issues, and a few successful user stories on KNL are presented.

We present optical spectra of the nearby Type Ia supernova SN 2011fe at 100, 205, 311, 349 and 578 d post-maximum light, as well as an ultraviolet (UV) spectrum obtained with the Hubble Space Telescope at 360 d post-maximum light. We compare these observations with synthetic spectra produced with the radiative transfer code phoenix. The day +100 spectrum can be well fitted with models that neglect collisional and radiative data for forbidden lines. Curiously, including these data and recomputing the fit yields a quite similar spectrum, but with different combinations of lines forming some of the stronger features. At day +205 and later epochs, forbidden lines dominate much of the optical spectrum formation; however, our results indicate that recombination, not collisional excitation, is the most influential physical process driving spectrum formation at these late times. Consequently, our synthetic optical and UV spectra at all epochs presented here are formed almost exclusively through recombination-driven fluorescence. Furthermore, our models suggest that the UV spectrum even as late as day +360 is optically thick and consists of permitted lines from several iron-peak species. These results indicate that the transition to the ‘nebular’ phase in Type Ia supernovae is complex and highly wavelength dependent.

Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent `offline' analysis. We demonstrate this approach in the compressible gasdynamics/N-body code Nyx, a hybrid MPI+OpenMP code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own ('in situ'). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other 'sidecar' group, which post-processes it while the simulation continues ('in-transit'). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.

NVRAM-based Burst Buffers are an important part of the emerging HPC storage landscape. The National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory recently installed one of the first Burst Buffer systems as part of its new Cori supercomputer, collaborating with Cray on the development of the DataWarp software. NERSC has a diverse user base comprised of over 6500 users in 700 different projects spanning a wide variety of scientific computing applications. The use-cases of the Burst Buffer at NERSC are therefore also considerable and diverse. We describe here performance measurements and lessons learned from the Burst Buffer Early User Program at NERSC, which selected a number of research projects to gain early access to the Burst Buffer and exercise its capability to enable new scientific advancements. To the best of our knowledge this is the first time a Burst Buffer has been stressed at scale by diverse, real user workloads and therefore these lessons will be of considerable benefit to shaping the developing use of Burst Buffers at HPC centers.

SN 2012dn is a super-Chandrasekhar mass candidate in a purportedly normal spiral (SAcd) galaxy, and poses a challenge for theories of type Ia supernova diversity. Here we utilize the fast and highly parametrized spectrum synthesis tool, SYNAPPS, to estimate relative expansion velocities of species inferred from optical spectra obtained with six facilities. As with previous studies of normal SN Ia, we find that both unburned carbon and intermediate-mass elements are spatially coincident within the ejecta near and below 14 000 km s⁻¹. Although the upper limit on SN 2012dn's peak luminosity is comparable to some of the most luminous normal SN Ia, we find a progenitor mass exceeding ∼1.6 M_⊙ is not strongly favoured by leading merger models since these models do not accurately predict spectroscopic observations of SN 2012dn and more normal events. In addition, a comparison of light curves and host-galaxy masses for a sample of literature and Palomar Transient Factory SN Ia reveals a diverse distribution of SN Ia subtypes where carbon-rich material remains unburned in some instances. Such events include SN 1991T, 1997br, and 1999aa where trace signatures of C III at optical wavelengths are presumably detected.

We use observed UV through near-IR spectra to examine whether SN 2011fe can be understood in the framework of Branch-normal Type Ia supernovae (SNe Ia) and to examine its individual peculiarities. As a benchmark, we use a delayed-detonation model with a progenitor metallicity of Z_⊙/20. We study the sensitivity of features to variations in progenitor metallicity, the outer density profile, and the distribution of radioactive nickel. The effect of metallicity variations in the progenitor have a relatively small effect on the synthetic spectra. We also find that the abundance stratification of SN 2011fe resembles closely that of a delayed-detonation model with a transition density that has been fit to other Branch-normal SNe Ia. At early times, the model photosphere is formed in material with velocities that are too high, indicating that the photosphere recedes too slowly or that SN 2011fe has a lower specific energy in the outer ≈0.1 M_⊙ than does the model. We discuss several explanations for the discrepancies. Finally, we examine variations in both the spectral energy distribution and in the colours due to variations in the progenitor metallicity, which suggests that colours are only weak indicators for the progenitor metallicity, in the particular explosion model that we have studied. We do find that the flux in the U band is significantly higher at maximum light in the solar metallicity model than in the lower metallicity model and the lower metallicity model much better matches the observed spectrum.

We present near-infrared synthetic spectra of a delayed-detonation hydrodynamical model and compare them to observed spectra of four normal Type Ia supernovae ranging from day +56.5 to day +85. This is the epoch during which supernovae are believed to be undergoing the transition from the photospheric phase, where spectra are characterized by line scattering above an optically thick photosphere, to the nebular phase, where spectra consist of optically thin emission from forbidden lines. We find that most spectral features in the near-infrared can be accounted for by permitted lines of Fe II and Co II. In addition, we find that [Ni II] fits the emission feature near 1.98 μm, suggesting that a substantial mass of ⁵⁸Ni exists near the center of the ejecta in these objects, arising from nuclear burning at high density.

SN 2011fe was the nearest and best-observed type Ia supernova in a generation, and brought previous incomplete datasets into sharp contrast with the detailed new data. In retrospect, documenting spectroscopic behaviors of type Ia supernovae has been more often limited by sparse and incomplete temporal sampling than by consequences of signal-to-noise ratios, telluric features, or small sample sizes. As a result, type Ia supernovae have been primarily studied insofar as parameters discretized by relative epochs and incomplete temporal snapshots near maximum light. Here we discuss a necessary next step toward consistently modeling and directly measuring spectroscopic observables of type Ia supernova spectra. In addition, we analyze current spectroscopic data in the parameter space defined by empirical metrics, which will be relevant even after progenitors are observed and detailed models are refined.

In supernova (SN) spectroscopy relatively little attention has been given to the properties of optically thick spectral lines in epochs following the photosphere's recession. Most treatments and analyses of post-photospheric optical spectra of SNe assume that forbidden-line emission comprises most if not all spectral features. However, evidence exists that suggests that some spectra exhibit line profiles formed via optically thick resonance-scattering even months or years after the SN explosion. To explore this possibility, we present a geometrical approach to SN spectrum formation based on the "Elementary Supernova" model, wherein we investigate the characteristics of resonance-scattering in optically thick lines while replacing the photosphere with a transparent central core emitting non-blackbody continuum radiation, akin to the optical continuum provided by decaying 56Co formed during the explosion. We develop the mathematical framework necessary for solving the radiative transfer equation under these conditions and calculate spectra for both isolated and blended lines. Our comparisons with analogous results from the Elementary Supernova code SYNOW reveal several marked differences in line formation. Most notably, resonance lines in these conditions form P Cygni-like profiles, but the emission peaks and absorption troughs shift redward and blueward, respectively, from the line's rest wavelength by a significant amount, despite the spherically symmetric distribution of the line optical depth in the ejecta. These properties and others that we find in this work could lead to misidentification of lines or misattribution of properties of line-forming material at post-photospheric times in SN optical spectra.