Illuminating the Dark Universe with the Sequoia Supercomputer

Pavlos Vranas (13-ERD-023)

Abstract

We set out to study whether dark matter might be a composite—that is, made out of more elementary constituents analogous to ordinary nuclear matter. We discovered that a theory almost identical to quantum chromodynamics (which is the experimentally proven theory of the strong nuclear force) provides for a composite dark matter particle, which we named "stealth dark matter." This particle is made out of “dark quarks” that have ordinary electric charge and are tightly bound together by an as-yet unobserved strong force. During the course of our research, we examined composite Higgs theories and the phase transition of quantum chromodynamics. In addition, we completed a study of nucleon–nucleon scattering from first-principles quantum chromodynamics. For our research, we received Tier-1 Grand Challenge computational awards with about one-billion Blue Gene/Q-Vulcan supercomputer core hours annually. We were also one of six international finalists in the 2013 Gordon Bell competition as a result of this work. Additionally, during this project, we became involved with and helped influence the development of the DOE CORAL collaboration between the NNSA's Advanced Simulation and Computing Program and the Office of Science’s Advanced Scientific Computing Research Program. Finally, our work helped advance the exascale supercomputing effort at Lawrence Livermore.

Background and Research Objectives

Our project used LLNL's two petaflops Vulcan supercomputer, along with lattice gauge theory, to simulate theories that could explain the nature of dark matter and lead to experiments that will detect it. Approximately 83% of all matter in the universe does not interact directly with the electromagnetic or strong nuclear force—light does not bounce off it and ordinary matter goes through it with only the feeblest of interactions. Essentially invisible, it has been termed dark matter, yet its interactions with gravity produce striking effects on the movement of galaxies, leaving little doubt of its existence. In the time it takes to read this page, an astonishing amount of this material, about a billion particles, can pass through the human body. Our project involved numerical simulations of models using the methods of lattice gauge theory to convert continuous space–time into a regular four-dimensional grid of points called the lattice, which maps naturally onto the grid of compute nodes of a massively parallel supercomputer such as LLNL’s Vulcan.

A unique confluence of events has been occurring that made this research urgent: (1) strong international interest is producing even more sensitive dark-matter-detection experiments; (2) the upgraded Large Hadron Collider particle collider, built by the European Organization for Nuclear Research and located near Geneva, Switzerland, may produce dark matter particles in the laboratory; (3) theories with strong-force dynamics have been reaching a calculational maturity; and (4) the availability of massively parallel supercomputers, such as the Vulcan machine, which are capable of carrying out the numerical simulations required to study the properties of these theories.

The main objective of this project was to explore this unique opportunity for LLNL to contribute decisively to the understanding of dark matter. In doing so, we also pursued several other objectives including assisting with the start-up of new Livermore Blue Gene/Q machines (Sequoia and Vulcan) and planning for future Lawrence Livermore supercomputers. We also studied the closely related theory of quantum chromodynamics (the experimentally proven theory of the strong nuclear force) both at zero temperature and at the transition temperature of its plasma phase, not only to provide guidance for the theory of dark matter under consideration, but also on its own right. Finally, we studied the possibility that the Higgs sector is an effective theory produced by a higher energy theory similar to quantum chromodynamics.

This project met all its proposed objectives. We postulated the theory of stealth dark matter and simulated it numerically to examine its viability. We found that a theory almost identical to quantum chromodynamics can form a composite dark matter particle made out of dark quarks that have ordinary electric charge and are tightly bound together by an as-yet unobserved strong force. This stealth dark matter particle is stable over cosmic timescales, electrically neutral, has mass larger than 200 times the proton mass, and yet is invisible to all current dark matter direct-detection experiments (see Figure 1). In addition, this particle has a mass heavier than about 700 times the proton mass, and will interact so weakly with ordinary matter that even the most-sensitive future detectors will be unable to detect it.1–4

Figure 1. the new theory of stealth dark matter gives robust predictions for the “stealthy” interactions between dark matter and ordinary nuclear matter (purple diagonal band). the various possible values of the dark matter mass are given in the <em>x</em>-axis, while its strength of interaction with ordinary nuclear matter is given in the <em>y</em>-axis. different collaborations of experimentalists have started looking at this interaction and will be able to rule out or confirm this new theory. the grey a
Figure 1. The new theory of stealth dark matter gives robust predictions for the “stealthy” interactions between dark matter and ordinary nuclear matter (purple diagonal band). The various possible values of the dark matter mass are given in the x-axis, while its strength of interaction with ordinary nuclear matter is given in the y-axis. Different collaborations of experimentalists have started looking at this interaction and will be able to rule out or confirm this new theory. The grey area and the blue area have already been explored by experiments without signs of dark matter, the orange area cannot be investigated with current or future direct-detection experiments, while the white area has not yet been explored by direct-detection experiments, but it is possible to do so in future experiments.

Scientific Approach and Accomplishments

All of our work was performed with numerical simulations on the massively parallel Blue Gene/Q Vulcan supercomputer using the methods of lattice gauge theory. This theory discretizes space and time into a four-dimensional grid of points (lattice) and assigns the fields of the relativistic quantum field theory on the sites and links of this lattice. It then uses variants of molecular dynamics evolution to evolve the fields and produce a statistical sampling of vacuum field configurations. Various physical quantities are then calculated as functions of these fields and analyzed accordingly.

Because our simulations involve dynamic fermions (elementary particles with a half-integer spin), these simulations include fermion vacuum loops. Numerical simulations of fermions are notoriously difficult and involve the inversion of a very large but sparse matrix using variants of the conjugate-gradient algorithm. Straightforward compilation of existing codes and use of the faster Vulcan supercomputer would not have been enough to achieve all the results mentioned above. We had to implement highly efficient code that sustained about 25% of peak, and very importantly, invent new physics methods that dramatically improved our time to solution. In one case, our efforts resulted in an overall factor of 400 improvement over the previous state of the art. With this result, we entered the 2013 Gordon Bell Prize competition and were selected as one of the six international finalists.5

We performed ground-breaking simulations of quantum chromodynamics in the high-temperature regime at the quark–gluon plasma phase transition and in the zero temperature regime that interfaces with nuclear physics.6–10 We also studied the possibility that Higgs physics is the result of a higher-energy composite theory governed by a new strong force. The results may be predicting possible findings during the upgraded run of the Large Hadron Collider in 2016.11–14

In addition, we have continued to help influence the future of supercomputing at LLNL by being actively involved in the CORAL-Sierra nonrecurring engineering contract, as well as DOE's Fast Forward and Design Forward exascale computing effort.

This work has provided an international high-end scientific profile to LLNL with important scientific discoveries. We have recruited postdoctoral researchers who are leaders in the field, three of whom completed their term during this project and are now LLNL staff members. Furthermore, our theoretical work has served the Laboratory's scientific experimental efforts in dark matter, the Large Hadron Collider, and heavy ions. In addition, the groundwork has been laid for studying low-energy nuclear properties directly from quantum chromodynamics. Integral to our approach has been the creation of LLNL-centered national collaborations of physicists across the United States. We are leading three national collaborations with about 15 members each: (1) the Lattice Strong Dynamics collaboration, which was formed to pursue non-perturbative studies of strongly interacting theories likely to produce observable signatures at the Large Hadron Collider; (2) the CalLat (California Lattice) collaboration with Lawrence Berkeley National Laboratory and the University of California, Berkeley, which is focused on constructing a controlled theory of nuclear structure and reactions, and linking that theory directly to lattice quantum chromodynamics; and (3) the HotQCD collaboration, which focuses on lattice gauge calculations in quantum-chromodynamics thermodynamics with an emphasis on calculations that pertain to experimental data from heavy-ion collisions.

Impact on Mission

Our research has added a strong theoretical component to LLNL’s premier experimental dark matter detection program and is helping establish LLNL as a leader in dark matter theory. The project supports the Laboratory’s strategic focus area in stockpile stewardship science with study of the structure and interactions of nuclear particles, which is directly relevant to the physics of light-ion reactions that occur in high-energy-density environments. Our lattice simulations using Livermore supercomputers is strongly aligned to the Laboratory’s core competency in high-performance computing, simulation, and data science.

Conclusion

Our results met all of our project objectives and helped create a strong foundation for theoretical nuclear and particle physics at LLNL with an international profile. With the national collaborations we have established, we have laid the foundation to continue this research. In the CalLat collaboration, for example, we expect to continue the efforts of connecting low-energy nuclear physics with quantum chromodynamics. In addition, we plan to further pursue the study of stealth dark matter and make predictions for possible discovery at the Large Hadron Collider or via direct detection and cosmic observations. We also plan to continue our studies in high-temperature quantum chromodynamics, focusing on the freeze-out regime of a heavy-ion collision. These efforts have gained support from DOE's Scientific Discovery through Advanced Computing program, and we plan to further pursue additional support from the DOE Office of Science. Supercomputing is at the center of our efforts, and we plan to continue to participate in the CORAL-Sierra effort as well as the effort to built an exascale supercomputer in the early 2020s via the Fast Forward and Design Forward DOE programs. For each year of our project, we were awarded a Tier-1 Grand Challenge allocation and ran for about one-billion core hours on Vulcan annually. Finally, during the early days of Sequoia and Vulcan, we also helped identify faulty nodes and power supplies.

References

  1. Appelquist, T., et al., “Stealth dark matter: Dark scalar baryons through the Higgs portal.” Phys. Rev. D 92, 075030 (2015). LLNL-JRNL-667446.
  2. Appelquist, T., et al., “Composite bosonic baryon dark matter on the lattice: SU(4) baryon spectrum and the effective Higgs interaction.” Phys. Rev. D 89, 094508 (2014). LLNL-JRNL-650612. http://dx.doi.org/10.1103/PhysRevD.89.094508
  3. Appelquist, T., et al., "Lattice calculation of composite dark matter form factors." Phys. Rev. D 88, 014502 (2013). LLNL-JRNL-608695. http://dx.doi.org/10.1103/PhysRevD.88.014502
  4. Appelquist, T., et al., “Direct detection of stealth dark matter through electromagnetic polarizability.” Phys. Rev. Lett. 115, 171803 (2015). LLNL-JRNL-667121.
  5. Boyle, P., et al., The origin of mass. Supercomputing 2013, Denver, CO, Nov. 17–22, 2013. LLNL-PROC-641527.
  6. Soltz, R., et al., Lattice QCD thermodynamics with physical quark masses. arXiv:1502.02296 [hep-lat] (2015). LLNL-JRNL-671601.
  7. Appelquist, T., et al., "Equation of state in (2+1)-flavor QCD." Phys. Rev. D 90, 094503 (2014). LLNL-JRNL-657376. http://dx.doi.org/10.1103/PhysRevD.90.094503
  8. Bhattacharya, T., et al., “The QCD phase transition with physical-mass, chiral quarks.” Phys. Rev. Lett. 113, 082001 (2014).
  9. Buchoff, M. I., et al., “The QCD chiral transition, U(1)A symmetry and the Dirac spectrum using domain wall fermions.” Phys. Rev. D 89, 054514 (2014). LLNL-JRNL-642513.
  10. Berkowitz, E., et al., Two-nucleon higher partial-wave scattering from lattice QCD. arXiv:1508.00886 [hep-lat] (2015). LLNL-JRNL-674381.
  11. Appelquist, T., et al., Strongly interacting dynamics and the search for new physics at the LHC. arXiv:1601.04027 (2016). LLNL-JRNL-680732.
  12. Appelquist, T., et al., "Lattice simulations with eight flavors of domain wall fermions in SU(3) gauge theory." Phys. Rev. D 90, 114502 (2014). LLNL-JRNL-665913. http://dx.doi.org/10.1103/PhysRevD.90.114502
  13. Brower, R. C., et al., "Maximum-likelihood approach to topological charge fluctuations in lattice gauge theory." Phys. Rev. D 90, 014503 (2014). LLNL-JRNL-650193. http://dx.doi.org/10.1103/PhysRevD.90.014503
  14. Appelquist, T., et al., “Two-color gauge theory with novel infrared behavior.” Phys. Rev. Lett. 112, 111601 (2014). LLNL-JRNL-652296. http://dx.doi.org/10.1103/PhysRevLett.112.111601

Publications and Presentations

  • Aoki, Y., et al., Lattice study of the scalar and baryon spectrum in many-flavor QCD. Origin of Mass and Strong Coupling Gauge Theories (SCGT15), Nagoya, Japan, Mar. 36, 2015. LLNL-PROC-676444.
  • Appelquist, T., et al., “Composite bosonic baryon dark matter on the lattice: SU(4) baryon spectrum and the effective Higgs interaction.” Phys. Rev. D 89, 094508 (2014). LLNL-JRNL-650612. http://dx.doi.org/10.1103/PhysRevD.89.094508
  • Appelquist, T., et al., “Direct detection of stealth dark matter through electromagnetic polarizability.” Phys. Rev. Lett. 115, 171803 (2015). LLNL-JRNL-667121.
  • Appelquist, T., et al., "Equation of state in (2+1)-flavor QCD." Phys. Rev. D 90, 094503 (2014). LLNL-JRNL-657376. http://dx.doi.org/10.1103/PhysRevD.90.094503
  • Appelquist, T., et al., "Lattice calculation of composite dark matter form factors." Phys. Rev. D 88, 014502 (2013). LLNL-JRNL-608695. http://dx.doi.org/10.1103/PhysRevD.88.014502
  • Appelquist, T., et al., "Lattice simulations with eight flavors of domain wall fermions in SU(3) gauge theory." Phys. Rev. D 90, 114502 (2014). LLNL-JRNL-665913. http://dx.doi.org/10.1103/PhysRevD.90.114502
  • Appelquist, T., et al., “Stealth dark matter: Dark scalar baryons through the Higgs portal.” Phys. Rev. D 92, 075030 (2015). LLNL-JRNL-667446.
  • Appelquist, T., et al., “Two-color gauge theory with novel infrared behavior.” Phys. Rev. Lett. 112, 111601 (2014). LLNL-JRNL-652296. http://dx.doi.org/10.1103/PhysRevLett.112.111601
  • Bazavov, A., et. al., “The equation of state in (2+1)-flavor QCD.” Phys. Rev D 90, 094503 (2014). LLNL-JRNL-657376.
  • Berkowitz, E., et al., Nucleon–nucleon scattering in multiple partial waves from lattice QCD. (2015). LLNL-POST-676169.
  • Bhattacharya, T., et al., "QCD phase transition with chiral quarks and physical quark masses." Phys. Rev. Lett. 113, 082001 (2014). LLNL-JRNL-650194. http://dx.doi.org/10.1103/PhysRevLett.113.082001
  • Boyle, P., et al., The origin of mass. Supercomputing 2013, Denver, CO, Nov. 17–22, 2013. LLNL-PROC-641527.
  • Boyle, P., et al., The origin of mass update. Supercomputing 2013, Denver, CO, Nov. 17–22, 2013. LLNL-PROC-644577.
  • Brower, R. C., et al., "Maximum-likelihood approach to topological charge fluctuations in lattice gauge theory." Phys. Rev. D 90, 014503 (2014). LLNL-JRNL-650193. http://dx.doi.org/10.1103/PhysRevD.90.014503
  • Buchoff, M. I., et al., “The QCD chiral transition, U(1)A symmetry and the Dirac spectrum using domain wall fermions.” Phys. Rev. D 89, 054514 (2014). LLNL-JRNL-642513. http://dx.doi.org/10.1103/PhysRevD.89.054514
  • Schroeder, C. R., The chiral phase transition from lattice QCD with physical pion masses and domain wall fermions. CPOD 2013, 8th Intl. Workshop Critical Point and Onset of Deconfinement, Napa, CA, Mar. 11–15, 2013. LLNL-PROC-645080.
  • Soltz, R., et al., Lattice QCD thermodynamics with physical quark masses. (2015). LLNL-JRNL-671601.
  • Vranas, P. M., The origin of mass. Supercomputing 2013, Denver, CO, Nov. 17–22, 2013. LLNL-PRES-646694.