Learning-Based Predictive Models: A New Approach to Integrating Large-Scale Simulations and Experiments

Brian Spears | 18-SI-002

Project Overview

Predictive simulations are constantly challenged and improved by comparison with experimental data. The experimental observations and simulations of those observations have grown data-rich and often include collections of images, vector-valued data, scalars, and other modalities. However, traditional approaches used to compare simulation and experiment omit much of this data, leaving predictive models less accurate than they could be.

In this effort, we used modern deep-learning techniques to develop improved predictive models by first creating models that simultaneously predict all modalities, and then, in turn, can exploit all available experimental data to improve themselves. These models, now widely known as Cognitive Simulation (CogSim) models, begin with deep neural networks that are trained on the entire collection of simulated observations, in this particular case using inertial confinement fusion implosions as a testbed. The simulation-trained network is then updated to reflect experimental data trends from the National Ignition Facility at Lawrence Livermore National Laboratory. The re-trained model, or elevated model, produces more accurate predictions of new, as-yet unseen, experiments. We drove this model elevation process using a state-of-the-art Manifold and Cyclic Consistency (MaCC) network to use a wide range of image data along with traditional scalar measurements simultaneously. At the same time, our efforts in uncertainty quantification (UQ) have offered two new approaches to provide statistically meaningful confidence intervals on key scientific observables. To support our CogSim deep learning efforts, we developed a novel asynchronous workflow system (Merlin) to steer both extremely high-volume computation and machine learning on the results on high-performance computing (HPC) systems, with the aim of feeding MaCC.

We used this capability to produce an open-source, scientific, machine learning data set composed of 100 million simulations and almost 5 billion scientific images. Ultimately, this enormous data resource drove significant improvements in the Livermore Big Artificial Neural Network (LBANN) framework, which are already being exploited in a number of other efforts, most notably the COVID-19 drug discovery project. In particular, we demonstrated the ability to train an enormous MaCC neural network on billions of images using novel model parallelism on the world's second largest supercomputer, Sierra, which is located at the Laboratory.

Mission Impact

Our CogSim learning framework satisfies a longstanding need in DOE missions, especially NNSA's stockpile stewardship mission, to more effectively exploit experimental data to improve model predictions. We developed CogSim as a central thrust across the Laboratory's mission research challenges. The deep learning methods we developed provide larger, more flexible models that allow use of data without aggregation or reduction. This capability allows predictive and UQ analyses to explore a much larger number of observable quantities—hundreds or thousands—whereas prior techniques were limited to a handful. At the same time, the new techniques allow these many observables to span multiple data types. This capability is especially important because it allows us to include complete image data, making imaging experiments far more valuable and, ultimately, enabling better use of historical data and better design of future experiments.

Publications, Presentations, and Patents

Ahn, D. H., et al. 2018. "Overcoming Scheduling Challenges for Exascale Workflows." WORKS, Dallas, TX, Nov. 2018. LLNL-CONF-756663

——— 2020. “Flux: Overcoming scheduling challenges for exascale workflows.” Future Gener. Comput. Syst., 110:202–213, 2020. LLNL-PRES-814589

Anirudh, R., et al. 2019a. “Mimicgan: Robust projection onto image manifolds with corruption mimicking.” International Journal of Computer Vision (VISI). LLNL-JRNL-774117

——— 2019b. “Exploring generative physics models with scientific priors in inertial confinement fusion.” Proc. Machine Learning for Physical Sciences Workshop at NeurIPS 2019. LLNL-TR-790162

Castillo, V. 2019a. "Enhancing Experimental Design and Understanding with Deep Learning/AI." LLNL-PRES-748201

——— 2019b. "Machine Learning for Better Understanding and Control of Complex Processes." LLNL-PRES-769927

Castillo, V. and Y. Mubarka. 2019. "Enhancing Experimental Design with A0.I." LLNL-PRES-759174

Castillo, V., et al. 2019a. "Asynchronous Method for Active Learning on HPC for Efficient Exploration of Complex Systems." LLNL-POST-754786

——— 2019b. "Developing Fast-running Simulations Models for Manufacturing Using Deep Learning." LLNL-PRES-769382

Jacobs, S. 2019. "LTFB: Training Neural Networks at Sierra Scale." GPU Technology Conference. LLNL-ABS-774862

Jacobs, S., et al. 2019a. "Parallelizing Training of Deep Generative Models on Massive Scientific Datasets." The IEEE Cluster Conference. doi:10.1109/CLUSTER.2019.8891012. LLNL-CONF-776577

——— 2019b. "Parallelizing Training of Deep Models on Massive Scientific Datasets." LLNL-PRES-791164

Koning, J., et al. 2019. "Merlin Workflow: Machine Learning Driven Ensembles." LLNL-PRES-769884

Kustowski, B. 2019. "Correcting Predictions of a Computer Simulation to Match the Experiments Using Transfer Learning." LLNL ASC Machine Learning Workshop, 2019. LLNL-PRES-790119

——— 2020. “Correcting simulation bias in multi-modal data using transfer learning.” Extreme Physics, Extreme Data, Lorentz Center Workshop, Leiden, Netherlands, 2020. LLNL-PRES-801153.

Kustowski, B., et al. 2018a. “Transfer learning for the calibration of the ICF simulations.” DSI Workshop. LLNL-ABS-753684

——— 2018b. “Transfer learning for the calibration of the inertial confinement fusion simulations.” Annual Meeting of the APS Division of Plasma Physics. LLNL-PRES-760734

——— 2019a. "Correcting Predictions of a Deficient Simulation Code Using Transfer Learning." Research Challenges at the interface of Machine Learning and Uncertainty Quantification. LLNL-PRES-782122

——— 2019b. "Early Prediction of the Simulation Outputs." APS DPP 2019. LLNL-PRES-793722

——— 2019c. "Transfer Learning as a Tool for Reducing Simulation Bias: Application to Inertial Confinement Fusion." IEEE Trans. on Plasma Phys. LLNL-PROC-770955

——— 2019d. "Transfer Learning for the Calibration of the ICF Simulations." LLNL-POST-755064

——— 2019e. "Transfer Learning for the Calibration of the Inertial Confinement Fusion Simulations." APS-DPP, 2018. LLNL-PRES-760734

Liu, S., et al. 2018. Topology-driven analysis and exploration of high-dimensional models. Proc. Workshop Research Challenges and Opportunities at the interface of Machine Learning and Uncertainty Quantification. LLNL-ABS-749353

Peterson, J. 2018a. “Aiding computational workflows with machine learning.” JOWOG34 ACS, 2. LLNL-PRES-745444

——— 2018b. “Machine Learning Aided Discovery of a new NIF Design.” Data Science Institute Workshop. LLNL-PRES-755427

——— 2019a. "Aiding Computational Workflows with Machine Learning." LLNL-PRES-745444

——— 2019b. "Machine Learning Aided Discovery of a New NIF Design." LLNL-PRES-755427

——— 2019c. "Fail Fast and Often! And Take Down Other People Too!" LLNL-PRES-782657

——— 2019d. "Learning-Based Predictive Models: A New Approach to Integrating Large-Scale Simulations and Experiments." Machine Learning for Computational Fluid and Solid Dynamics. LLNL-ABS-767192

——— 2020. “Attack of the Killer Neurons: Next-Gen Algorithms and High-Performance Computing.” Lorentz Center, Netherlands. LLNL-PRES-801317

Spears, B. 2017. "Ensemble Simulations of Inertial Confinement Fusion Implosions." Stat. Anal. Data Min., 10, 230–237. doi:10.1002/sam.11344. LLNL-JRNL-694784

——— 2018. "Deep Learning: A Guide for Practitioners in the Physical Sciences." Phys. Plasmas, 25. doi:10.1063/1.5020791. LLNL-JRNL-743970

——— 2019a. "Learning-Based Predictive Models: A New Approach to Integrating Large-Scale Simulations and Experiments." LLNL-PRES-735160

——— 2019b. "Learning-Based Predictive Models: A New Approach to Integrating Large-Scale Simulations and Experiments." Intl Conf. on Data Driven Plasma Sci. LLNL-PRES-756925

——— 2019c. "Making inertial confinement fusion models more predictive.” Phys. Plasmas, 26. LLNL-JRNL-770914

——— 2019d. "Parallelizing Training of Deep Generative Models on Massive Scientific Datasets." IEEE INT C CL COMP. LLNL-CONF-677443

——— 2020a. "Deep learning for NLTE spectral opacities." Phys. Plasmas, 27. LLNL-JRNL-805050

——— 2020b. "Surrogates in inertial confinement fusion with manifold and cycle consistencies." P. Natl. Acad. Sci, 117, 9741-9746. LLNL-JRNL-790997

——— 2020c. "Scalable Topological Data Analysis and Visualization for Evaluating Data-Driven Models in Scientific Applications." IEEE T. Vis. Comput. Gr., 26, 291-300. LLNL-JRNL-770822

——— 2020d. "Transfer Learning to Model Inertial Confinement Fusion Experiments." IEEE T. Plasma Sci., 48, 61-70. LLNL-CONF-764063

Thiagarajan, J., et al. 2019a. “Unsupervised dimension selection using a blue noise graph spectrum.” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). LLNL-CONF-767904

——— 2019b. “Understanding deep neural networks through input uncertainties.” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019. LLNL-PROC-767884

——— 2020. “Designing accurate emulators for scientific processes using calibration-driven deep models.” Nature Communications. LLNL-JRNL-808697

Van Essen, B. 2019a. "Cognitive Simulation: Intersection of Large Scale Machine Learning and Scientific Computing." LLNL-PRES-778998

——— 2019b. "Learning-Based Predictive Models: A New Approach to Integrating Large-Scale Simulations and Experiments." LLNL-PRES-757656