Home > Systems and Analyses Center > IDA Research Divisions > SED > SED Research Areas > Analytical Support to the System Test and Evaluation Community

SED Research Area:
Analytical Support to the System Test and Evaluation Community

Previous Research Area

Next Research Area

SED provides analytical support to the Department of Defense’s system test and evaluation (T&E) community, particularly in the area of developmental test and evaluation. Our principal analytical support activities include reviewing test plans, observing tests, and analyzing test data to help ensure that test activities are revealing the systems’ progress toward meeting their requirements. Analytical support activities include developing assessment methodologies and simulation tools to aid test planning and the evaluation of test data. Some selected recent SED projects are briefly described below along with the publication year of the associated report.

  • FY15 NDAA Assessment of the Ground-based Midcourse Defense (GMD) Test Program - IDA conducted a congressionally mandated study of the GMD test program to (1) assess whether the test program to date has established that the system has a high probability of performing reliably and effectively, (2) assess whether the currently planned testing program is sufficient to establish reasonable confidence that the system will have a high probability of performing reliably and effectively, and (3) recommend improvements that could be made to the testing program.  (2016)

  • F-35 Joint Strike Fighter Developmental Testing In support of the Deputy Assistant Secretary of Defense (Developmental Test and Evaluation) - IDA assessed progress in the developmental testing of the F-35 Joint Strike Fighter to inform key DoD decision milestones.  For example, we provided objective technical assessments of issues related to the durability of structural materials and of a unique surface treatment process called Laser Shock Peening.  (2015, 2016)

  • Air Warfare Combat Assessment Methodology Development - IDA assisted the Director, Operational Test & Evaluation, the Joint Strike Fighter Operational Test Team, and the Air Force Operational Test and Evaluation Center with our assessments of the F-35 and F-22 aircraft in their intended combat environment.  We developed a quantitative methodology and sets of data for simulating the outcome of terminal missile engagements with these aircraft to be used during testing, training, and tactics development.  Our method is being implemented within man-in-the-loop simulators as well as on open-air ranges.  (2016)

  • In support of Director, Operational Test and Evaluation, our tactical communications test and evaluation project monitors and assesses systems in development.  The Soldier Radio Waveform (SRW) is the U.S. Army’s developmental networking software that aims to provide voice, data, and video capabilities to small combat units and unmanned systems.  Based on our observations from the Army’s Network Integration Exercises, we constructed representative scenarios for modeling and simulation purposes and configured the SRW nodes according to the Army’s implementation.   The locations of selected soldiers were determined from GPS data and accurate terrain and elevation information from Digital Terrain Elevation Data (DTED) data.  As an example, we investigated how terrain and choice of operating frequency affected communication between the Company Commander and 3rd Platoon Leader traveling in a convoy.   The bottom figure shows a cross section of the terrain elevation. The blue line indicates line of sight, and the yellow ellipse represents the first Fresnel Zone at a radio operating frequency of 2,000 MHz.  The top figure shows the signal attenuation plotted as a function of distance for two different frequencies (30 MHz – red; 2,000 MHz – yellow).  Thus, by transmitting with SRW over the higher frequencies rather than the legacy voice network offered by the Single Channel Ground-to-Air Radio System (SINCGARS) over the lower frequencies, the operator incurs an additional 4 dB of attenuation over these distances. (2015)
  • The estimating test requirements from historical data project for the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) used data from previous testing of large aircraft systems, air-launched weapons, helicopters, and ground vehicles to summarize the basic test parameters and identify 10 distinct types of test that were generally applicable across the four types of systems. (2013)
  • The tactical communications mobile ad hoc networking (MANET) project for the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) evaluated the MANET performance of military communications systems and characterized their physical constraints and operating environments to develop a modeling and simulation methodology that can be used to inform test planning, metrics collection, and the analysis of test results for current developmental systems. (2013)
  • The circular error probable (CEP) calculations for precision-guided munitions project for the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) investigated how the use of traditional statistical methods for estimating the accuracy of unguided (ballistic) munitions can lead to an overestimation of CEP when applied to precision-guided munitions. (2013)
  • The Ground Combat Vehicle (GCV) analysis tools project for the Director, Operational Test and Evaluation (DOT&E) developed and demonstrated modeling tools for assessing the operational implications of ground vehicle performance under a variety of operational conditions in the areas of operational availability, survivability, and mobility. (2012)
  • The development and application of land warfare modeling and simulation tools project for the Director, Operational Test and Evaluation (DOT&E) was a follow up to the GCV analysis tools project to validate and demonstrate the utility of the GCV operational availability modeling tool using data from the Joint Light Tactical Vehicle (JLTV) limited user test. (2014)
  • In support of Ballistic Missile Defense Command, Control, Battle Management and Communication test and evaluation SED studies error estimation.  When a sensor tracks an airborne object, estimated track states (position and velocity) can be accompanied by a covariance matrix that quantifies the estimation errors.  However, network bandwidth limitations may require that the full covariance data be transmitted only at some infrequent rate.  For track states not accompanied by error data, algorithms can estimate covariance by using, among other information, knowledge of the reporting sensor’s location.  IDA developed an approach for estimating, given certain assumptions, the sensor’s location by using data from two previously received tracks with full covariance to triangulate the sensor’s location.  To demonstrate the approach, we ran simulations of a two-dimensional crossing track scenario in which a single sensor tracks a target traveling east at a 250 m/s.  The top figure shows how the error in sensor location (color coded) varies as a function of separation angle between covariance triangulation points as well as time in track.  Larger errors (denoted by “A”) occur when the track is young, but improve once the track has settled.  The lower, zoomed-in, figure shows that, when the separation angle is greater than 20% and track age is greater than 200 seconds, sensor location errors range between 2,000-3,000 meters. (2014)

  • The evaluation of intelligence, surveillance, and reconnaissance (ISR) system performance project for the Director, Operational Test and Evaluation (DOT&E) used the IDA Sensing Effectiveness Evaluator (ISEE) model to simulate the unmanned aerial vehicle (UAV)-based Multi-spectral Targeting System (MTS-B) in different operational environments and extract factors relevant to the planning of operational test events. The project also used new ISEE model capabilities to compare the predicted performance of the Multi-Function Active Sensor (MFAS) radar in Doppler-beam-sharpening wide-area search (DBS WAS) mode to the system requirements. (2014)
  • Analytical Support to SystemsThe planning of developmental and operational tests of new unmanned aerial vehicle (UAV) reconnaissance systems can be aided by modeling the performance of the systems’ electro-optical and infrared imaging sensors in different operational environments.  The U.S. strategic intelligence community assesses image quality according to the National Imaging Interpretability Rating Scale (NIIRS).  NIIRS values range from 1 (lowest quality) to 9 (highest quality) and are subjectively assigned based on expert judgment by imagery analysts.

    The general image quality equation (GIQE) shown in this figure can be used to estimate the NIIRS value of an image collected by a particular sensor system based on the characteristics of the sensor system and the signal-to-noise ratio of the unprocessed image, which is determined by environmental factors and the position of the sensor relative to the target.  In this equation, GSD is the sensor system’s ground sample distance, RER is the system’s post-processing relative edge response, G is the system post-processing noise gain, SNR is the signal-to-noise ratio of the unprocessed image, H is the system post-processing edge overshoot factor, and a and b are coefficients that depend on the relative edge response.  The bottom of the figure shows some examples of images with different NIIRS values along with the associated approximate ground-resolved distance (GRD), which is related to but not equal to the sensor’s ground sample distance due to the other factors that contribute to the general image quality equation.

    Analytical Support to the System Test and Evaluation CommunityMilitary electro-optical and infrared imaging systems employ ultraviolet, visible, and infrared light to create images of targets.  The performance of these systems depends on the sensor optics, detector, and display, as well as on light propagation effects.  The figures provide four examples of image clarity issues that can make an image more difficult to interpret.  The figure at the top left has low contrast, which occurs when the intensity difference between the lightest and darkest portions of the image is small.  The figure at the top right has high blur, which can be caused by an imperfect optical focus or the diffraction of light from the target within the sensor system.  The figure at the bottom left has high spatial aliasing, which occurs when the target is discretely sampled at low spatial resolution.  The figure at the bottom right has high noise, which occurs when the detected light from a low-intensity target is overwhelmed by random light from sources other than the target. (2013)