Milton Smith (14-ERD-039)
We focused on two algorithms to improve active and hyperspectral imaging sensor performance for remote sensing applications. We examined algorithms to mitigate scene-dependent structured noise from mercury–cadmium–telluride focal plane arrays, and optimization algorithms to improve coverage rate and target sensitivity. For the first focus, we developed an algorithm that specifically addressed the noise source, which is unique to conventional approaches applied as part of the normal data processing chain. This algorithm is currently being used to address sensor performance issues. For the second focus, we utilized spectral and spatial scales and sampling to provide coverage rate improvement of fifty times for a single detector and up to a hundred times under specific circumstances.
Background and Research Objectives
Algorithms are needed that improve performance and extend capabilities of spectral laser sensors to detect scarce materials or gases for a wide variety of remote sensing, surveillance, and reconnaissance applications. Likewise, algorithms that improve performance of remote sensors already in service, as well as for next-generation sensor systems currently under development, are also desired. The information processing chain for sensors is assumed independent of the hardware, having relative minor impact on system performance. It is generally assumed that differences in performance are mostly because of hardware differences between systems and that the processing chain is generalized to yield optimal performance to all systems for all conditions. We selected two algorithm domains important to existing sensor systems that addressed two performance issues, namely coverage rates for active systems and structured noise in mercury–cadmium–telluride focal-plane imaging arrays. Changing the operation and processing paradigm of these two problems can realize a minimum of tenfold performance improvement for each sensor type, significantly increasing the utility of these sensors. These solutions were experimentally validated in the laboratory using archived data sets.
Hyperspectral imaging collects and processes information from across the electromagnetic spectrum, with the goal of obtaining the spectrum for each pixel in the image of a scene. This kind of imaging is used to find objects, identify materials, or detect processes. Livermore hyperspectral imaging sensors were not performing as predicted and not operating at the noise levels measured in the laboratory. For the first focus of our project, our research objectives were to determine why sensor operational noise was not matching laboratory noise and to create an algorithm that could improve performance given knowledge of the noise source. We determined that the noise was sensor scene-dependent, associated with the sensor focal plane arrays. We developed an algorithm that specifically addressed this noise source, which is unique to conventional approaches (e.g., detector nonuniformity correction, noise spike filters, and data whitening and sphering) that are applied as part of the normal data processing chain. This algorithm continues to be used to address sensor performance issues.
The second focus dealt with the lack of use of LLNL active sensors. The drawback in using these point sensor systems was the lack of area coverage. Our objective for this second focus was to develop algorithms that exploit performance based on available versatility in sensor concept of operations. While this objective initially focused on algorithms applied to LLNL active sensors, it also expanded to include passive single-detector hyperspectral imaging sensors. Current compressive image algorithms were not appropriate for application because of target sparsity. Our approach utilized spectral and spatial scales and sampling to provide coverage rate improvement of 50 times for a single detector and up to 100 times under specific circumstances.
Scientific Approach and Accomplishments
Hyperspectral imaging is a remote sensing modality that allows for the spectroscopic identification of materials at standoff distances, of importance to the nonproliferation mission of the DOE and other missions of national significance. Although there exists in the literature a suite of data exploitation methods designed for hyperspectral target detection and identification, such methods are often attended by a variety of limitations in practice. Our research effort focused on the characterization of departures from ideal theoretical models exhibited by hyperspectral imaging systems in practical settings, as well as novel algorithms to enable new hyperspectral sensor design concepts. Together, these emphases were aimed at improving detection performance and increasing detector coverage rates through optimized sampling techniques.
One significant component of this work was our analysis of structured (i.e., scene-dependent) noise in hyperspectral imaging sensors. Structured noise was previously observed to be an important contributor to false alarm rates in practical hyperspectral imaging sensor systems that far exceeded theoretical predictions; at the same time, it was a very poorly understood phenomenon. We employed a "spectral mixture replacement" algorithm, which allowed for detector focal-plane array elements to be labeled as exhibiting anomalous behavior when the measured spectral component does not lie within the modeled subspace spanned by plausible end-member spectra. With ground truth provided by the algorithm indicating the occurrence of structured noise, exploratory data analysis shed light on a number of key aspects of this problematic noise source (see Figure 1).First, we determined that some instances of structured noise are apparently induced by factors inherent in the detector elements themselves. Some focal plane array elements exhibited structured noise in 90% of measurements; likewise, some detectors were more likely to exhibit such noise over a wide variety of observed scenes. Second, our investigation ruled out the persistence phenomenon noted within the astrophysical imaging community as a basis for structured noise. While it was plausible that a similar detector saturation effect could be taking place for hyperspectral focal plane arrays, we did not observe either the time constant or the power-law form characteristic of decay of the persistence phenomenon. Third, we found that sufficient radiance contrast in the scene was necessary to elicit structured noise and that it was then largely concentrated within the highest- and lowest-radiance regions of the scene. Finally, for all the focal plane arrays evaluated, structured noise was more prevalent near the edges of the detector array, possibly because of nonuniform cooling.
In addition to studying detector phenomenology, we further developed and evaluated a detection strategy based on the recursive dyadic partition, with recursive partitioning being a statistical method for multivariable analysis (see Figure 2). Based on the group-testing framework of compressive sensing, albeit without the drawbacks that lead conventional compressive sensing techniques to falter on hyperspectral detection problems, recursive dyadic partitioning is an enabling concept for next-generation hyperspectral imaging sensors. Because recursive dyadic partitioning significantly reduces the measurement rate needed for hyperspectral target detection, it enables single-detector systems that hold promise for avoiding the noise and nonuniformity issues endemic to sensors based on focal plane arrays. Similarly, it can be used to increase the areal coverage rate of sensor systems. We conducted an extensive theoretical analysis of recursive dyadic partitioning, beginning by determining the theoretical baseline performance of the underlying adaptive coherence estimator metric (an algorithm for detection of point targets). An analysis of the likelihood of target detection by recursive dyadic partitioning, assuming independent detection at each spatial scale, confirmed that very good detection performance of a single-pixel target is expected with measurement budgets of at least 5% (versus a raster scan) and that near-perfect detection is expected for 2-pixel target spreads with at least 2% measurement budgets. Having substantiated the expected detection performance of the recursive dyadic partition algorithm, we conducted a sensitivity analysis as well. Sensitivity of the algorithm to target-like clutter showed that performance was degraded once target-like clutter occupied 1% of the scene pixels, as compared with 5% for the baseline detector. This amount of degradation was deemed tolerable, provided targets are spatially sparse and the target spectrum is sufficiently separable from the background subspace. Similarly, recursive dyadic partitioning was theoretically and empirically demonstrated to be resilient against target-colored noise. Finally, the algorithm was shown to inherit the resilience of the underlying adaptive coherence estimator detector related to background covariance misestimation, which is expected in heterogeneous or statistically nonstationary background environments.
The recursive dyadic partition algorithm provides spatial compression for hyperspectral sensing—measurement rates can be further reduced by also performing spectral compression. One means of spectral compression discussed in the literature (e.g., for the Coded Aperture Snapshot Spectral Imager design) is to employ a coded-aperture Hadamard spectrometer.1–3 A Hadamard sensing matrix measures combinations of spectral bands, and a reconstruction method such as the two-step iterative shrinkage/thresholding (TwIST) approach is used to recover the higher-resolution spectrum.4 Baseline recursive dyadic partitioning, utilizing only spatial compression, was compared against a hybrid algorithm combining recursive dyadic partitioning and TwIST, which also provides spectral compression as described above. Synthetic injection trials were conducted, in which a broad-featured target spectrum was injected with a 20% linear mixing ratio into random locations within a scene, with a 14.9-dB signal-to-interference-plus-noise ratio. The Hadamard sensing matrix used in the simulation compressed down to 13% of the original number of spectral bands. The simulation demonstrated that the hybrid sensing approach performed well over a variety of operating parameters, including for single-pixel targets. Counterintuitively, the hybrid algorithm including spectral compression often outperformed the baseline algorithm, even performing better at higher levels of compression than at lower levels. The advantage conferred by spectral compression was most notable for a detection scenario such as was modeled here, with relatively high noise power and a broad-featured target spectrum. For hyperspectral detection of solid target materials, such conditions often prevail. Finally, for both baseline recursive dyadic partitioning and recursive dyadic partitioning with TwIST, detection performance was enhanced when target spatial extents exceeded a single pixel.
While the Hadamard spectrometer construct is appealing, its application to hyperspectral imaging sensor systems is not always straightforward. Constraints needed to make the Hadamard mask physically realizable induce a bias. Moreover, it can be difficult to produce a quickly changing mask to cover different spatial scales. An alternative is to perform spectral compression by imaging through a filter bank. Because infrared spectra of solid materials tend to be relatively smooth, a filter bank of around 25 narrow-band filters is sufficient to reproduce long-wave infrared spectra of solid materials sufficiently well for the purposes of target spectral identification. We developed two approaches to facilitate the use of such filter banks for hyperspectral imaging sensors. The first was an adaptive spectral band selection method that allows for adaptively selecting a subset of possible filters with the objective of maximizing the signal-to-interference-plus-noise ratio under the utilized detection metric and prevailing statistics. Band selection is a difficult problem for which a wide variety of computationally-intensive heuristic strategies are discussed in the literature.5 Here, the way the problem is posed yields a mixed-integer quadratic program that is tractable for practical systems. Different allocations of spectral measurement budgets over the different recursive dyadic partition stages were assessed for various classes of target spectrum and target-background contrast, resulting in an optimal spectral measurement budget allocation. We also developed a super-resolution recovery technique utilizing smoothness priors applicable for spectra of solids. This technique leverages mutual information captured in overlapping filter bands, which alternative techniques like TwIST falter on.
We also investigated other aspects impacting areal coverage rates for novel sensor designs, such as single-detector systems. Such systems may not have “look-ahead” or “look-back” capabilities, in which case adaptive sensing decisions must be made in a strictly serial fashion. We developed a strictly serial variant of the recursive dyadic partition algorithm, which is feasible for hyperspectral detection, although it is more difficult to reliably constrain the measurement rate without additional adaptive thresholding a simple method of image segmentation that can create binary images from grayscale images. We also evaluated the effects of spectral mis-registration. Because of sensor platform motion and sensor pointing, hyperspectral sensors that acquire spectral bands sequentially over time will sample different bands over different fields of view. Using simulated inhomogeneous scenes to determine the expected impact of this effect, we found that even a relatively straightforward recovery approach suffices to compensate for the expected degree of spectral mis-registration inherent in anticipated novel sensor designs.
Finally, we investigated the impact of multiple sources of down-welling radiance, which is the thermal energy radiated onto the ground by all objects in a hemisphere surrounding it, including topography and atmospheric gases and aerosols. It has been observed that in addition to down-welling atmospheric radiance, other sources of side-welling radiance, which measures the gross spectral characteristics of the background, can significantly alter the at-aperture radiance sensed from the target. An end-to-end hyperspectral imaging sensor modeling code we developed was used to evaluate the effectiveness of the recursive dyadic partition algorithm over a range of down- and side-welling radiance conditions. As expected, when a significant fraction of cold sky was occluded by the temperature-matched side-welling source in simulation, loss of thermal contrast led to significantly reduced detection performance (see Figure 2). More surprising, however, was that better detection performance was observed in the case of partial occlusion of the sky than when a steradian view was modeled. This may indicate a tradeoff between the loss of thermal contrast and the reduction of sharply featured atmospheric down-welling for single-detector sensor systems, a possibility that could be explored further in future study.
Impact on Mission
Improvements in sensor performance are central to the Laboratory's strategic focus area in cyber security, space, and intelligence. In addition, this research has applications in a wide range of national security missions, and is applicable to LLNL's core competency in nuclear, chemical, and isotopic science and technology.
This project started partly to respond to hyperspectral imaging sensor user concerns, who thought sensor performance issues were limited to sensors designed and built by LLNL. To a certain degree, this project changed that perception in the community, and these sensors are now in high demand, resulting in new designs. The project had a positive impact on two DOE projects: the Conversion Venture and Hard Solids Ventures. Two new projects will utilize parts of our study—predicting hyperspectral imaging sensor performance for a government sponsor, and multivariate analysis of sensor data, which is a joint effort of the National Nuclear Security Administration and the U.K. Atomic Weapons Establishment. Our project gives Livermore a new framework for hyperspectral imaging sensor data analysis that aids in isolating complex issues surrounding exploit performance of hyperspectral imaging sensors.
The DOE real-time optimization spectral spatial sensor project will support our development of a single-detector hyperspectral imaging sensor, which will incorporate the area coverage strategies developed here. We are also beginning a collaboration with Raytheon in developing a better predictive tool for examining hyperspectral imaging sensor design trades. Our domain knowledge extends to sensor performance limitations in the field as opposed to the laboratory. At the beginning of this project, our knowledge was also limited by what we knew from laboratory measurements. We now can provide expertise for operational sensor performance to provide an end-to-end systems perspective that includes the impact of the operational environment.
- Busuioceanu, M., et al., "Evaluation of the CASSI-DD hyperspectral compressive sensing imaging system." Proc. SPIE Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XIX 8743 (2013). http://dx.doi.org/10.1117/12.2015445
- Wagadarikar, A., et al., "Single disperser design for coded aperture snapshot spectral imaging." Appl. Optic. 47(10), B44 (2008). https://doi.org/10.1364/AO.47.000B44
- Arguello, H., and G. R. Arce, "Code aperture optimization for spectrally agile compressive imaging," J. Opt. Soc. Am. A 28(11), 2400 (2011). https://doi.org/10.1364/JOSAA.28.002400
- Bioucas-Dias, J., and M. A. T. Figueiredo, "A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image reconstruction." IEEE Trans. Image Process. 16(12), 2992 (2007). https://doi.org/10.1109/TIP.2007.909319
- Wang, Y., et al., Research advance on band selection-based dimension reduction of hyperspectral remote sensing images. 2012 2nd Intl. Conf. Remote Sensing, Environment and Transportation Engineering, Nanjing, Jiangsu, China, June 1–3, 2012. https://doi.org/10.1109/RSETE.2012.6260684
Publications and Presentations
- Smith, M., and B. Beauchamp, Predicting and removing scene dependent noise from LWIR hyperspectral images. 2016 Military Sensing Symposia Passive Sensors Conf., Gaithersburg, MD, Oct. 31–Nov. 4, 2016. LLNL-CONF-705517.
- Beauchamp, B., and M. Smith, Validation of a novel approach to predicting and isolating factors limiting LWIR HSI exploit performance. 2016 Military Sensing Symposia Passive Sensors Conf., Gaithersburg, MD, Oct. 31–Nov. 4, 2016. LLNL-CONF-704470.