Independent Model Evaluations


SED is sometimes called upon to apply its expertise in modeling and simulation to provide independent evaluations of models developed for the government. These evaluations often involve verification and validation (V&V) activities in an operational or user-oriented context, or head-to-head comparisons of competing models. Some selected recent SED projects are briefly described below along with the publication year of the associated report.

  • Collaborative Nuclear Targeting — IDA’s collaborative nuclear targeting analyses project for the Defense Threat Reduction Agency develops and evaluates, in collaboration with the Department of Energy, prototype modeling and simulation capabilities to predict the effects and consequences of nuclear weapon strikes, with an initial focus on predicting the ignition and subsequent spread of fires caused by nuclear weapon detonations. (2016)
  • SED’s Biosurveillance Ecosystem verification and validation project developed a framework, algorithms, and basic capabilities for searching Twitter messages for those relevant to disease outbreaks.  The figure below depicts our methodology and example results. Our rule-based filtering process included two steps: (1) filtering using (approximately 15,000) keywords related to disease outbreaks and (2) filtering using Twitter’s metadata embedded in each tweet to remove those that are non-English, marked as retweets, or contain more than two hashtags or two mentions (which we observed are strong indicators of spam). Supervised learning methods further enhanced the relevancy of the discovered messages. These included the use of a list of linguistic features identified in a sample set of tweets and a set of manually labeled tweets to train the algorithms. (2015)
    Biosurveillance model
  • The verification and validation nuclear and radiological weapon effects models project for the Defense Threat Reduction Agency (DTRA) examined the modeling tools used by DTRA to predict nuclear and radiological weapons effects for the U.S. Strategic Command (USSTRATCOM) (2014)
  • The verification and validation of the Bid Evaluation Model (BEM) for bulk fuels project for the Defense Logistics Agency (DLA) examined DLA’s model for procuring bulk fuel at the lowest laid-down cost subject to meeting the demand for fuels and other constraints. (2011)
  • Due to the random effects of atmospheric turbulence it is impossible to predict exactly how a hazardous atmospheric plume will evolve. Most of the desktop computer models presently used for predicting the consequences of an airborne release of a toxic agent, such as a nerve gas, produce only a smoothed-out “mean” plume that approximates the average over the ensemble of many possible turbulence-induced plume realizations that could represent a real plume dispersion event. Since there is presently no straightforward mathematical relationship between ensemble-average model outputs (such as the ensemble-average atmospheric concentration) and consequence measures (such as the ensemble-average hazard area or number of casualties), there is a potential disconnect between the output of common modeling systems and the consequence measures that are of interest to decision-makers.

    The figure below compares one possible realization of a turbulent hazardous atmospheric plume generated by the National Center for Atmospheric Research’s high-performance computing VTHREAT model (left, overhead plot of the logarithm of the atmospheric concentration of material) with the average of 20 such realizations (right). The former plot represents a “realistic” turbulent plume while the latter plot approximates the type of output generated by most desktop-based ensemble-average consequence prediction models. (2012)graphics for atmospheric turbulence
  • The atmospheric transport and dispersion modeling supplement to Allied Tactical Publication-45 hazard area prediction templates project for the Defense Threat Reduction Agency (DTRA) investigated the conditions under which modern atmospheric transport and dispersion models could be used to improve on the standard NATO/U.S. doctrine for responding to nuclear, biological, or chemical attacks. (2012)
  • The comparative investigation of source term estimation algorithms for hazardous material atmospheric transport and dispersion prediction tools project for the Defense Threat Reduction Agency (DTRA) used data from an atmospheric dispersion field experiment to assess and compare state-of-the-art tools for predicting the location and size of an atmospheric release of hazardous materials (such as a chemical weapons attack) using data from a limited number of sensors. (2012)
  • The assessment of waterborne hazardous material transport and prediction tools for defense applications project for the Defense Threat Reduction Agency (DTRA) examined two commercially-developed models for predicting the dispersion of hazardous materials (such as toxic chemicals or radiological materials) in riverine or littoral waters and assessed their potential for inclusion in a Department of Defense hazard prediction modeling system. (2013)