imaging applications of large-format variable acuity superpixel imagers
TRANSCRIPT
Paper 6206-35 Presented at SPIE DSS Orlando, 2006
Imaging applications of large-format variable acuity superpixel imagers
Mark A. Massie Jon Paul Curzan
Nova Sensors, 320 Alisal Rd, Suite 104, Solvang, CA 93463 (805) 693-9600
Paul L. McCarley Nicholas I. Rummelt
Air Force Research Laboratory AFRL/MNGI
ABSTRACT
A wide variety of imaging applications exist for 1K x 1K midwave infrared (MWIR) imagers and Nova’s Variable Acuity Superpixel Imager (VASI™) technology1,2 has now progressed to this image format. This paper will demonstrate a variety of imagery from MWIR cameras using this large format “LVASI” device; the in-pixel processing used by the LVASI cameras represents the state-of-the-art for image size, total field of view, high frame rates, low data bandwidths and real-time spatial reprogrammability of focal plane arrays (FPAs).
Using these devices, imaging systems may now be implemented that permit the operator to “zoom in” to regions of interest with very high spatial resolution, while covering the remainder of the total field of view (TFOV) at conventional resolutions. The bandwidth compression attainable using these sensors helps to make possible systems that can transmit their high resolution imagery through wireless interconnected networks.
We present recent infrared image data that highlight numerous applications including missile detection/tracking, search/rescue and remote surveillance applications
Keywords: Variable acuity, 1K x 1K, superpixels, FPA, programmable, MWIR, infrared
1. INTRODUCTION Nova’s development of the “Variable Acuity Superpixel Imaging” (VASI™) technology was intended to provide an on-chip means to reduce the amount of image data from a focal plane array (FPA) required to be delivered by the device, and still retain the essential information content in the image. Operating much like that of the human visual system, the “most important” information is sampled at high spatial resolution, and “less interesting” image content is sampled at lower spatial resolution after performing on-chip spatial “binning” operations. This guarantees that:
• Important image information content is preserved. • The total field of view is always monitored, in case other “interesting” objects appear in the periphery,
and • The highest possible frame rates are produced for a fixed bandwidth, or alternately, • The lowest possible transmission bandwidth is required for a fixed output frame rate.
1 P. L. McCarley, Mark A. Massie, and J. P. Curzan, “Large format variable spatial acuity superpixel imaging: visible and infrared systems applications”. SPIE Proc., Infrared Technology and Applications XXX, vol. 5406, pp. 361-369, Orlando, 2004. 2 M. A. Massie, J. P. Curzan, R. A. Coussa, “Operational and performance comparisons between conventional and foveating large format infrared focal plane arrays”, Infrared Technology and Applications XXXI, vol. 5783, pp. 260-271, Orlando, 2005.
This general methodology could be applied to the use of multispectral detector arrays as well. These capabilities are now possible through the use of “READIN” commands to the FPA, used to program the characteristics of individual pixels in the array. Figure 1 demonstrates the concept of “FPA READIN Programmability”, resulting in a user-defined effective distribution of pixels. Also indicated in the figure is that the readout state for each pixel may be programmed individually through the use of the READIN command.
Figure 1. User-supplied commands are “read in” to each pixel, resulting in unique superpixel and readout
state attributes. The READIN word multiplexed into each pixel of this VASI™ will be used in each pixel to set its unique superpixel and readout state. Notice that ANY spatial configuration for the VASI™ would be possible; only two such configurations are shown in the figure. This capability would also permit a variety of spatial convolutions to be performed, opening up an entirely new area of advanced on-FPA image processing to the technical community. The superpixel configuration of each pixel is used to configure switch closures in an on-chip network that interconnects neighboring pixels. In effect, the combined integration capacitance for a given superpixel will be the parallel capacitance of all connected unit cells. The combined photocharge accumulated on the effective integration capacitance produces a voltage that represents the signal on this effective superpixel. In order to achieve a high frame rate, only one value per superpixel need be multiplexed off-chip. In addition, the device permits every standard pixel to be multiplexed out (even though it may be part of a superpixel) for diagnostic and other imaging purposes. In concept, as the row select signal ripples down a column of the device during the readout operation, only those pixels which have been previously programmed to produce their output signal will do so. Nova’s VASI™ design also incorporates a digital output channel that produces a “read-back” of the currently-programmed superpixel state, such that it can be recorded with the gray-level imagery produced by the device. This feature makes it possible for the reconstruction of the superpixel-based image data at a later time. A display processor with this information and the superpixel gray-level values will be able to reconstruct a display representing the appropriate spatial information in the scene. Figure 2 and Figure 3 illustrate the advantages of the VASI™ readout design over conventional on-chip data reduction techniques of pixel dilution and region-of-interest (ROI) windowing. With superpixels, only one output channel is required to obtain significant improvements in frame rate while not giving up any scene
information. And having one analog output channel rather than two, four, or sometimes even 32 channels greatly simplifies off-chip A/D and re-vectoring electronics.
N:1 Dilution- One out of N pixels read out- Fill factor reduced by (N-1)/N
MxM Super-Pixels- One out of N pixels read out (N=M2)- Fill factor remains unchanged
Read out pixel
Skipped pixel
Pixel boundary Figure 2 Dilution modes throw away pixel data in exchange for higher frame rates to maintain total FOV
(left), while a comparable superpixel configuration maintains both FOV and fill factor since all skipped pixels share photocharge with read out neighbors.
Windowing- Sub-frame regions read out successively- IFOV reduced to increase frame rate- Size and position usually limited by the
number of address bits- Rectangular only
Super-Pixels- High acuity and low acuity regions read out- IFOV remains unchanged with increased
frame rate- Any shape and location permitted
Read out pixel
Skipped pixel
Pixel boundary
Figure 3. Windowing modes reduce instantaneous FOV (IFOV) while increasing frame rate (left), while a comparable superpixel configuration can read out both the high-resolution ROI as well as the low
resolution peripheral pixel regions with little additional clocking overhead.
Nova has produced numerous operating versions of VASI™ devices operating with monolithic visible phototransistors in each unit cell (30 micron unit cell pitch) as demonstrated in Figure 4. And 320 x 256 pixel versions (30 micron pixel pitch) have been produced that use indium antimonide (InSb) detectors for midwave infrared (MWIR, 3.0 to 5.0 micron response, Figure 5) and indium gallium arsenide (InGaAs) detectors for near
infrared (NIR, 1.0 to 1.5 micron response, Figure 6). In addition, a 1024 x 1024 device (19.5 micron pixel pitch) has now been demonstrated with MWIR response.
Figure 4 320 x 256 visible VASI image showing region of higher spatial resolution around the eye.
Figure 5 320 x 256 MWIR VASI imagery with various spatial configurations programmed into the FPA.
Figure 6 InGaAs has been hybridized to the 320 x 256 VASI device for NIR spectral response.
2. APPLICATIONS OF MEDIUM AND LARGE FORMAT MWIR FOVEATING FPAS Foveating FPAs are becoming recognized for the imaging speed and bandwidth reduction properties that they possess. Rummelt has developed an efficient technique for performing real-time nonuniformity correction and bad pixel replacement for such foveating devices3, and members of the staff at 21st Century Technologies are incorporating VASI sensors into their continuing work relating to airborne surveillance and determination of three-dimensional structure from motion4. Full Resolution Night-Vision Imaging Mode Figure 7 was produced by the 1K x 1K MWIR VASI FPA operating in full resolution mode; in this condition, the FPA operates at a maximum frame rate of approximately 60 Hz (four output channels). In this mode, the device operates as a conventional imager and the large quantity of on-chip pixels limits the maximum effective frame rate of the system.
Figure 7 1K x 1K full resolution MWIR imagery. This spatial configuration allows for a frame rate of 60
frames/second operating with four output channels from the device.
3 Rummelt, Nicholas I., “A Combined Non-Uniformity and Bad Pixel Correction Method for Superpixelated Infrared Imagery”, Submitted for publication to Infrared Technology and Applications XXXII, SPIE Defense and Security Symposium, April, 2006, Orlando, FL. 4T. Coffman, B.L. Evans, A.C. Bovik, "Halftoning-Inspired Methods for Foveation in Variable-Acuity Superpixel Imager Cameras," Proc. 2005 IEEE Asilomar Conference on Signals, Systems, and Computers, Oct 30- Nov 2, 2005.
Figure 8 Another example of 1K x 1K resolution courtesy the Santa Barbara Sheriff’s Department and
VASI full-resolution imaging.
High Resolution Center, Lower Resolution Peripheral Imaging Figure 9 was made with a spatial configuration on the FPA such that the center 256 x 256 pixels were operating at their highest spatial resolution (i.e., a “fovea” was placed in this region), while the remainder of the image was “superpixelated” with radially-decreasing spatial resolution. The effect of the superpixellation was to increase the frame rate for this condition to approximately 100 Hz, in comparison to the 60 Hz full-resolution rate.
Figure 9 1K x 1K MWIR VASI imagery programmed with high spatial resolution in the center 256 x 256
pixels and radially-decreasing spatial resolution out to the periphery. This spatial configuration allows for frame rates of approximately 100 Hz.
Another example is shown in Figure 10 in which a hovering helicopter was recorded at a frame rate of approximately 300 Hz by placing a 64 x 64 fovea on the center of the image, with 16 x 16 superpixels populating the remainder of the image. This spatial configuration allows for a frame rate of approximately 300 Hz and allows for high-speed imaging of the rotation of the rotor blades and combustion exhaust dynamics.
Figure 10 1K x 1K MWIR VASI imagery of a helicopter, where the center 64 x 64 pixels are a high
resolution fovea and the remainder of the image is composed of 16 x 16 superpixels.
Target Tracking Tracking vehicles is a natural application for foveal sensors because characteristics of the vehicles may be used to key on where representative foveae should be placed in sequential image frames so as to provide high frame rate operation and still recover the required information on objects of interest. An example of this is shown below in Figure 11. This sequence of images shows how the frame rate is affected with varying levels of spatial superpixellation; as the size of the superpixels grows, the frame rate increases for a fixed number of high resolution foveal pixels. Figure 12 shows the relationships between resulting frame rate and superpixel size for two fovea used to track objects of interest in a 320 x 256 portion of the overall 1K x 1K pixel space. As the superpixel size grows, the number of high-resolution pixels in the foveae begins to dominate and control the maximum frame rate. If, in surveillance applications, the foveae can be kept small, the frame rates can grow to many thousands of frames per second.
Figure 11 Automobiles tracked with foveae from remote location show frame rate advantage using foveal
tracking at different levels of superpixellation.
Frame Rate vs. Superpixel Size
Two Foveae
0.0
200.0
400.0
600.0
800.0
1000.0
1200.0
0 5 10 15 20
Superpixel Dimension
Fram
e Ra
te (H
z)
Two 64 x 64 FoveaeTwo 32 x 32 Foveae
Figure 12 Frame rates as a function of superpixel size using two foveae.
Long-Range, Wide Field of View Imaging In many cases, most pixels in a 1K x 1K image contain information that is of limited or no use. As an example, Figure 13 captures the image of a flying helicopter in a small foveal region and the rest of the background imagery is basically sky or ground. This sequence was captured at 283 frames per second because the rest of the image, outside of the central 64 x 64 pixel fovea, is superpixelated at radially-decreasing spatial resolution, thereby permitting a high frame rate for the 1K x 1K field of view.
Figure 13 High resolution on the object of interest is maintained while continuing to monitor the wide
field of view at 283 frames/second.
High-Speed Dynamics High frame rates permit the recording and/or real-time processing of image data such that temporal features in the data may be extracted. As an example of this, the long-range, high foveal spatial and temporal resolution data of Figure 13 may be analyzed to measure the angular speed of the main and tail rotors of the long-range hovering helicopter. The large pixel count of the 1K x 1K VASI imager and its small pixel size (19.5 µm pixel pitch) produces an instantaneous field of view (IFOV) of 0.78 milliradians when using a 25 mm focal length lens, as used to collect this imagery. In this condition, a 3-meter sized object (approximately the size of this helicopter at various aspect angles) will become spatially unresolved at a range of approximately 3.8 km.
The temporally modulated image of the main and tail rotor blades in this sequence was measured and the infrared signal as a function of time is shown below in Figure 14. The temporal modulation was produced by the reflection of the engine’s heat signature onto rotating surfaces of the respective rotors and two reflective pulses were produced per revolution because there exist two blades per rotor. By measuring the period of time between successive flashes, one may easily determine the rotational speed of these rotors; the main rotor was measured to be spinning at 340 rpm and the tail rotor was measured at 2830 rpm; these measurements are within 4% to 8%, respectively, of the OH-58Ranger helicopter’s quoted typical angular speeds.
Figure 14 The main and tail rotor angular rates from the image sequence of Figure 13 were extracted
from the foveal image region.
Power spectral density data may be extracted from the high frame rate data as shown in Figure 15. Application of such techniques could be useful in performing remote diagnostics for engine condition.
Figure 15 Power Spectral Densities pulled out of high-rate temporal modulation imagery of OH-58
Ranger helicopter.
Border Surveillance and Law Enforcement Surveillance systems will find many uses for the wide-field coverage and simultaneous high spatial resolution that VASI technology offers. The next figures demonstrate some of these techniques. Figure 16 and Figure 17 demonstrate the “Resolution-On-Demand™” feature of these camera systems that permits a user or controlling processor to “fly” numerous high-resolution foveae around the real-time scene so as to capture salient image data while still covering the full image field. While the full-field images in the left panels of these figures appear normal, they have actually been superpixelated to reduce image storage/transmission requirements in all regions except where the people are located. In Figure 16, this has produced a compression of 92.58% of the original 1K x 1K image data; in Figure 17, this has produced a 90.52% image compression performance. If a 92.58% reduction of the required number of pixels were applied to 8-bit transmitted 1K x 1K effective image data, 15 Hz image data could be transmitted over a bandwidth of approximately 9.3 Mbits/second. Near-broadcast quality resolution and wide-field coverage is possible using these systems using wireless 802.11 technology. Such programmable resolution will find applications in wireless image transmission and other low-bandwidth applications.
Figure 16 Resolution-On-Demand™ can be used for border surveillance application because it permits
significant image compression while retaining essential scene information.
Figure 17 Law enforcement surveillance applications will require substantial image compression to be
able to cover wide angles and still retain high spatial resolution where it's needed.
Search and Rescue, Unmanned Air Vehicle Applications As infrared variable acuity imaging becomes integrated into a variety of real-time applications, we believe that small, man-portable systems will be produced for applications such as search and rescue and unmanned air vehicles (UAVs). Nova Sensors is currently working on a variety of miniature systems for such applications. Hypertemporal Applications The high frame rates afforded by VASI sensing permits a variety of techniques to be applied to the image data that exploit the presence of temporal signatures in temporally-modulated infrared data. Initial work has been
performed by Kristl regarding long-range detection of ignition plumes5,6 and these applications will continue to be exploited. Nova Sensors has worked directly with Dr. Kristl of the Space Dynamics Laboratory/USURF and welcomes the opportunity to apply VASI cameras and these techniques to other applications.
5. SUMMARY AND FUTURE WORK Variable acuity superpixel imaging applications abound, mainly driven by system requirements for either (a) high frame rates with wide-angle coverage, or (b) low transmission bandwidth restrictions while requiring high spatial resolution and wide-angle coverage. These requirements are satisfied using VASI sensors because the spatial resolution in portions of the scene that are not as important is traded for either high frame speed or low transmission bandwidth. Nova Sensors is currently working on VASI systems that incorporate uncooled infrared sensors (microbolometers and InGaAs detector arrays) and their respective small packages that are amenable to man-portable and UAV applications. In addition, the low bandwidth characteristic of these systems makes possible the development of wireless mesh-type networks that communicate real-time variable acuity image data between nodes. These systems permit individual node elements with their own image-compressing cameras to communicate between themselves and other host nodes so as to easily create wireless surveillance systems that offer very high resolution performance. Visible, cooled and uncooled infrared system developments as discussed are in-process at Nova Sensors and we hope to incorporate this new technology into existing systems that can take advantage of the speed/resolution/bandwidth trades offered by variable acuity imaging technology.
6. ACKNOWLEDGEMENTS This work was sponsored by the Munitions Directorate of the Air Force Research Laboratory, AFRL/MNG. The continued support and inspiration provided is much appreciated. The authors would like to thank Mr. Ric Wehling, Mr. Paul McCarley and Mr. Nick Rummelt of the AFRL/MNG for their dedication and continued support of biomimetic technology developments and demonstrations.
5 Kristl, Joseph, M. A. Massie, et al, “Improvements in Hypertemporal Detection of Boost Phase Missiles Using VASI Sensors”, MSS MD-SEA 2003” 6 J.A. Kristl, F.O. Clark, V. Falcone, "Application of Temporal Plume Intensity Modulation to Boost Phase Intercept", MD-SEA meeting on 04 February 2002.