Roger Pearce | 19-FS-009
Overview
The volume of data generated today by security applications is growing at an unprecedented rate. Data streams may come from a variety of sources that can be fused together to form a large graph. For example, network sensors and host-based sensors often simultaneously gather data to monitor a large enterprise network. When a deep historical analysis is required, the data scales often require heavy triage or filtering that can impede deep analysis. The promise of using exascale computing for such analysis is that a unified picture of a large distributed dataset is possible. However, tools to tackle enterprise-level datasets are still in research. With the impending use of accelerators on upcoming leadership-class supercomputers, the need for scalable graph analytics is fundamental to fully utilize next-generation, high-performance computing (HPC) systems.
Our project investigated new techniques for mapping graph analytics to leadership-class HPC systems using benchmarks— Graph500 and GraphChallenge —established by the HPC graph community. Specifically, we investigated the scalability of distributed graph analytics on Sierra-class supercomputers being deployed on the road to exascale. On both the benchmarks, the team demonstrated new capabilities by processing graphs larger than prior art.
Impact on Mission
Our project leveraged and advanced Lawrence Livermore National Laboratory's HPC, simulation, and data science core competencies. Our results enhance Laboratory capabilities to support large-scale graph analytics vital to mission-critical security applications including cybersecurity, space, intelligence, and cyber-physical security.
Publications, Presentations, Etc.
Pearce, R., et al. 2019. "One Quadrillion Triangles Queried on One Million Processors." 2019 IEEE High Performance Extreme Computing Conference, HPEC, September 2019. LLNL-CONF-789648.
Steil, T. and R. Pearce. 2019. "Graph500 on Sierra: Using NVRAM at Scale." 2019 Non-Volatile Memory Workshop, NVMW, March 2019. LLNL-POST-769061.
