Test and Evaluation

IDA continues to participate in a wide variety of test and evaluation activities in support primarily to the Director, Operational Test and Evaluation (DOT&E), the principal advisor to the Secretary of Defense on operational test matters. With IDA’s continuing support, DOT&E ensures that major weapons systems undergo operational and live fire tests and evaluations that are sufficiently realistic to determine their operational effectiveness, suitability, lethality, and survivability. In addition, IDA helps DOT&E ensure that the nation’s test infrastructure is capable of supporting future weapon development.

Below is a small sample of the numerous studies IDA's Studies and Analyses Center conducts in the area of test and evaluation.

Land Warfare Systems

The Stryker family of vehicles consists of two basic variants: the Infantry Carrier Vehicle and the Mobile Gun System. While the Infantry Carrier Vehicle has been used in Iraq, the Mobile Gun System required more developmental effort.  This past year, IDA monitored Mobile Gun System technical testing and assessed the results, which showed that although the Mobile Gun System’s reliability has improved, it was still well short of program requirements.

In parallel, IDA supported DOT&E and the Army in developing a novel test strategy in which the operational test for the Mobile Gun System would take place in conjunction with a unit’s training rotation at the National Training Center. With this approach, testing and evaluation needs were met, while supporting the Army’s planned deployments of the Mobile Gun System to Iraq. IDA also helped develop the live fire test and evaluation of the system’s vulnerability to threat attack, highlighting the need for testing against a specific threat that has emerged from combat experience in Iraq.

The Guided Multiple Launch Rocket System (GMLRS) is the Army’s precision indirect fire guided rocket. An early version of the unitary warhead variant of the GMLRS has been deployed to Iraq. Although the testing prior to the deployment was mostly technical, IDA worked with the Army to add important live fire and operational conditions to the test events. At the suggestion of IDA researchers, large wooden and metal plates were placed at various distances around the impact area to unobtrusively gather realistic fragment data. This allowed testers to determine the shape and extent of the fragment pattern. Also, this data allowed the Army to verify computer models used to estimate GMLRS lethality.

Air Warfare

To reduce losses of helicopters to man-portable air defense systems, the Army’s Special Operations Command intends to upgrade the missile warning systems on its helicopters. Because IDA had participated in many previous tests of missile warning systems, the Command asked us to compare the performance of existing missile warning systems with new systems likely to be available in the near future. We examined relevant tests for six systems, assessing results in terms of probability of detection, time to intercept, and false alarm rate.

The EA-18G Growler will replace the EA-6B Prowler Electronic Attack radar and communications jamming aircraft as the Prowlers are retired in 2012. Significant tests and an acquisition decision on the Growler are anticipated in 2007. In preparation for these events, IDA outlined concepts for testing and evaluating the system. The concepts linked mission areas, critical operational issues, measures of effectiveness, suitability, survivability, and, ultimately, testing requirements. Our study concepts influenced the Growler’s test master plan approved by DOT&E.

Strategic Warfare and Net-Centric Systems

DoD weapons, command and control, and surveillance systems are being networked to operate together and share information in real-time. While increasing U.S. capabilities, networks are subject to attack or exploitation from a variety of sources ranging from casual hackers to state-sponsored organizations. For the past several years, IDA has supported DOT&E’s assessment of the Combatant Commands’ ability to protect and defend operational networks and maintain the availability, confidentiality, and integrity of the information.

Depending on the theater of operations, a particular Service operational test agency leads the assessment effort for DOT&E during an exercise, with support from Service and National Security Agency Red Teams as well as Joint Task Force - Global Network Operations, as needed. The Red Teams try to penetrate the DoD networks using tools available in open sources. IDA provides DOT&E with an independent, high-level assessment that focuses on consistency in processes, methodologies, metrics, and overarching trends. IDA also is exploring measures to characterize a command’s ability to protect its networks. Results of the overarching assessment are reported annually to Congress.

Our researchers participated in the planning, observation, and analysis of the operational testing of the Deployable Joint Command and Control (DJC2) system that integrates shelters, communications systems, and information technology into a deployable operations center to support joint task forces. We observed Combatant Command exercises and participated in the Multi-Service Operational Test and Evaluation. In this test, the U.S. Southern Command’s Standing Joint Force Headquarters completing scenario tasks as a forward deployed element of a joint task force conducting disaster relief and counter-terrorist operations. IDA integrated data from developmental tests, operational assessments, and operational tests. Our analysis confirmed that the support crew could marshal, deploy, assemble, and support the DJC2 and that the staff could successfully conduct operations within the DJC2.

In 2006, IDA worked with DOT&E to establish criteria for characterizing the operational capability of the Ballistic Missile Defense System (BMDS). Because this system and its constituent elements are developmental programs, many of the tests conducted to date have demonstrated capabilities that satisfied developmental, not operational, objectives. In support of future testing, IDA helped DOT&E develop a test and evaluation concept that identifies operational issues and performance measures for the BMDS. Our researchers used criteria from this test and evaluation concept to assess the operational realism of flight tests conducted in 2006. These ongoing assessments highlight test aspects where operational realism is lacking and provide specific ways to improve the operational realism of future flight tests of BMDS.

Naval Warfare

For the last eight years, DoD has been replacing sonar system processing software and computers in nuclear submarines with the Acoustic Rapid Commercial-Off-the-Shelf Insertion (ARCI) system. In 2006, the Navy conducted an operational test pitting an ARCI-equipped U.S. submarine against the Swedish diesel-electric submarine, HSWMS Gotland. IDA participated in planning the exercise and analyzed the results. This operational test also provided DOT&E with an initial evaluation of U.S. submarine capability against the diesel-electric submarine threat. IDA remains heavily involved in the test planning of each upgrade of ARCI, the most significant of which will be aboard USS Ohio, the lead ship in the newly converted and modernized SSGN class of submarines.

Over the next decade, the Navy plans to spend approximately $1.1 billion testing ship self-defense systems against anti-ship missiles. IDA worked with DOT&E and the Navy to develop the Capstone Enterprise Air Warfare Ship Self Defense Test and Evaluation Master Plan, which consolidates previously separate test events into fewer, multi-purpose events. In addition to saving about $240 million, this approach provides more robust testing by examining end-to-end performance of the combat system rather than focusing on a slice of combat capability – e.g., the radar or the air-defense missile by themselves.

Live Fire Test and Evaluation

The Small Diameter Bomb (SDB) is an all-weather precision attack glide bomb that relies on the Global Positioning System for guidance. In 2006, SDB completed operational testing, went into full-rate production, and has been used successfully in combat. The SDB program was noteworthy for the degree to which the activities supporting both operational and live fire testing and evaluation were integrated.

During all phases of the SDB testing, IDA advised DOT&E on test planning and conduct. First, IDA proposed a more robust flight test program using live weapons to attack a variety of realistic targets.  A total of 17 weapons were delivered against seven types of targets. Also, our researchers questioned the premature removal of a particular fuzing option without more test-based justification. Added test events showed that the observed fuzing problems were unique to the testing environment and would not be duplicated in operational use.

Resource Analyses for Test and Evaluation

IDA helps ensure DoD has the capability to test weapon systems by assessing various aspects of the test and evaluation (T&E) infrastructure, resources, and policy. This year IDA continued its support of the Test Resource Management Center (TRMC), helping it develop the “Strategic Plan for Defense T&E Resources” to guide the Services and defense agencies in their investments in T&E capabilities.

In a separate project for TRMC, IDA examined NASA’s aeronautical research facilities to determine which ones were important to DoD missions and suggested policy options for a new interagency agreement with NASA. The agreement, which established the National Partnership for Aeronautical Testing, provides the underpinning for a continuing partnership between the two agencies for managing their respective components of the federal aeronautics test infrastructure.
Also, IDA assessed the test resources available to support operational testing, and we studied organizational and policy alternatives for creating a combined test force for testing the Ballistic Missile Defense System. Our study identified ways to maintain the operational test representatives’ independent perspective and analytical credibility while economizing resources, eliminating duplication of effort, and supporting the Missile Defense Agency’s mission and objectives. MDA is now implementing most of this plan.

Additionally, IDA examined the demographics of the T&E workforce, and the findings are expected to support DoD decisions on workforce sizing and shaping initiatives.