We present an algorithm and implementation for the parallel computation of exact exchange in Quantum ESPRESSO (QE) that exhibits greatly improved strong scaling. QE is an open-source software package for electronic structure calculations using plane wave density functional theory, and supports the use of local, semi-local, and hybrid DFT functionals. Wider application of hybrid functionals is desirable for the improved simulation of electronic band energy alignments and thermodynamic properties, but the computational complexity of evaluating the exact exchange potential limits the practical application of hybrid functionals to large systems and requires efficient implementations. We demonstrate that existing implementations of hybrid DFT that utilize a single data structure for both the local and exact exchange regions of the code are significantly limited in the degree of parallelization achievable. We present a band-pair parallelization approach, in which the calculation of exact exchange is parallelized and evaluated independently from the parallelization of the remainder of the calculation, with the wavefunction data being efficiently transformed on-the-fly into a form that is optimal for each part of the calculation. For a 64 water molecule supercell, our new algorithm reduces the overall time to solution by nearly an order of magnitude.

The Feynman-Hellmann theorem can be derived from the long Euclidean-time limit of correlation functions determined with functional derivatives of the partition function. Using this insight, we fully develop an improved method for computing matrix elements of external currents utilizing only two-point correlation functions. Our method applies to matrix elements of any external bilinear current, including nonzero momentum transfer, flavor-changing, and two or more current insertion matrix elements. The ability to identify and control all the systematic uncertainties in the analysis of the correlation functions stems from the unique time dependence of the ground-state matrix elements and the fact that all excited states and contact terms are Euclidean-time dependent. We demonstrate the utility of our method with a calculation of the nucleon axial charge using gradient-flowed domain-wall valence quarks on the Nf=2+1+1 MILC highly improved staggered quark ensemble with lattice spacing and pion mass of approximately 0.15 fm and 310 MeV respectively. We show full control over excited-state systematics with the new method and obtain a value of gA=1.213(26) with a quark-mass-dependent renormalization coefficient.

We present a determination of nucleon-nucleon scattering phase shifts for ℓ≥0. The S, P, D and F phase shifts for both the spin-triplet and spin-singlet channels are computed with lattice Quantum ChromoDynamics. For ℓ>0, this is the first lattice QCD calculation using the Lüscher finite-volume formalism. This required the design and implementation of novel lattice methods involving displaced sources and momentum-space cubic sinks. To demonstrate the utility of our approach, the calculations were performed in the SU(3)-flavor limit where the light quark masses have been tuned to the physical strange quark mass, corresponding to mπ=mK≈800 MeV. In this work, we have assumed that only the lowest partial waves contribute to each channel, ignoring the unphysical partial wave mixing that arises within the finite-volume formalism. This assumption is only valid for sufficiently low energies; we present evidence that it holds for our study using two different channels. Two spatial volumes of V≈(3.5 fm)3 and V≈(4.6 fm)3 were used. The finite-volume spectrum is extracted from the exponential falloff of the correlation functions. Said spectrum is mapped onto the infinite volume phase shifts using the generalization of the Lüscher formalism for two-nucleon systems.

The Roofline Performance Model is a visually intuitive method used to bound the sustained peak floating-point performance of any given arithmetic kernel on any given processor architecture. In the Roofline, performance is nominally measured in floating-point operations per second as a function of arithmetic intensity (operations per byte of data). In this study we determine the Roofline for the Intel Knights Landing (KNL) processor, determining the sustained peak memory bandwidth and floating-point performance for all levels of the memory hierarchy, in all the different KNL cluster modes.We then determine arithmetic intensity and performance for a suite of application kernels being targeted for the KNL based supercomputer Cori, and make comparisons to current Intel Xeon processors. Cori is the National Energy Research Scientific Computing Center’s (NERSC) next generation supercomputer. Scheduled for deployment mid-2016, it will be one of the earliest and largest KNL deployments in the world.

We extract pixel-level masks of extreme weather patterns using variants of Tiramisu and DeepLabv3+ neural networks. We describe improvements to the software frameworks, input pipeline, and the network training algorithms necessary to efficiently scale deep learning on the Piz Daint and Summit systems. The Tiramisu network scales to 5300 P100 GPUs with a sustained throughput of 21.0 PF/s and parallel efficiency of 79.0%. DeepLabv3+ scales up to 27360 V100 GPUs with a sustained throughput of 325.8 PF/s and a parallel efficiency of 90.7% in single precision. By taking advantage of the FP16 Tensor Cores, a half-precision version of the DeepLabv3+ network achieves a peak and sustained throughput of 1.13 EF/s and 999.0 PF/s respectively.

We investigate the OpenMP parallelization and optimization of two novel data classification algorithms. The new algorithms are based on graph and PDE solution techniques and provide significant accuracy and performance advantages over traditional data classification algorithms in serial mode. The methods leverage the Nystrom extension to calculate eigenvalue/eigenvectors of the graph Laplacian and this is a self-contained module that can be used in conjunction with other graph- Laplacian based methods such as spectral clustering.We use performance tools to collect the hotspots and memory access of the serial codes and use OpenMP as the parallelization language to parallelize the most timeconsuming parts. Where possible, we also use library routines. We then optimize the OpenMP implementations and detail the performance on traditional supercomputer nodes (in our case a Cray XC30), and test the optimization steps on emerging testbed systems based on Intel’s Knights Corner and Landing processors. We show both performance improvement and strong scaling behavior. A large number of optimization techniques and analyses are necessary before the algorithm reaches almost ideal scaling.

The newest NERSC supercomputer Cori is a Cray XC40 system consisting of 2,388 Intel Xeon Haswell nodes and 9,688 Intel Xeon-Phi “Knights Landing” (KNL) nodes. Compared to the Xeon-based clusters NERSC users are familiar with, optimal performance on Cori requires consideration of KNL mode settings; process, thread, and memory affinity; fine-grain parallelization; vectorization; and use of the high-bandwidth MCDRAM memory. This paper describes our efforts preparing NERSC users for KNL through the NERSC Exascale Science Application Program, Web documentation, and user training. We discuss how we configured the Cori system for usability and productivity, addressing programming concerns, batch system configurations, and default KNL cluster and memory modes. System usage data, job completion analysis, programming and running jobs issues, and a few successful user stories on KNL are presented.

We report on a lattice QCD calculation of the nucleon axial charge, $g_A$, using M\"{o}bius Domain-Wall fermions solved on the dynamical $N_f=2+1+1$ HISQ ensembles after they are smeared using the gradient-flow algorithm. The calculation is performed with three pion masses, $m_\pi\sim\{310,220,130\}$ MeV. Three lattice spacings ($a\sim\{0.15,0.12,0.09\}$ fm) are used with the heaviest pion mass, while the coarsest two spacings are used on the middle pion mass and only the coarsest spacing is used with the near physical pion mass. On the $m_\pi\sim220$ MeV, $a\sim0.12$ fm point, a dedicated volume study is performed with $m_\pi L \sim \{3.22,4.29,5.36\}$. Using a new strategy motivated by the Feynman-Hellmann Theorem, we achieve a precise determination of $g_A$ with relatively low statistics, and demonstrable control over the excited state, continuum, infinite volume and chiral extrapolation systematic uncertainties, the latter of which remains the dominant uncertainty. Our final determination at 2.6\% total uncertainty is $g_A = 1.278(21)(26)$, with the first uncertainty including statistical and systematic uncertainties from fitting and the second including model selection systematics related to the chiral and continuum extrapolation. The largest reduction of the second uncertainty will come from a greater number of pion mass points as well as more precise lattice QCD results near the physical pion mass.

We report on salient features of a mixed lattice QCD action using valence Möbius domain-wall fermions solved on the dynamical Nf=2+1+1 highly improved staggered quark sea-quark ensembles generated by the MILC Collaboration. The approximate chiral symmetry properties of the valence fermions are shown to be significantly improved by utilizing the gradient-flow scheme to first smear the highly improved staggered quark configurations. The greater numerical cost of the Möbius domain-wall inversions is mitigated by the highly efficient QUDA library optimized for NVIDIA GPU accelerated compute nodes. We have created an interface to this optimized QUDA solver in Chroma. We provide tuned parameters of the action and performance of QUDA using ensembles with the lattice spacings a≃{0.15,0.12,0.09} fm and pion masses mπ≃{310,220,130} MeV. We have additionally generated two new ensembles with a∼0.12 fm and mπ∼{400,350} MeV. With a fixed flow time of tgf=1 in lattice units, the residual chiral symmetry breaking of the valence fermions is kept below 10% of the light quark mass on all ensembles, mres0.1×ml, with moderate values of the fifth dimension L5 and a domain-wall height M5≤1.3. As a benchmark calculation, we perform a continuum, infinite volume, physical pion and kaon mass extrapolation of FK±/Fπ± and demonstrate our results are independent of flow time and consistent with the FLAG determination of this quantity at the level of less than one standard deviation.