34

Investing inSimulationCredibility William L. Oberkampf, WLO ConsultingMartin Pilch, MPilchConsulting

How can managers with little training in science or engineering understandwhy verification, validation, and uncertainty quantification add value tosimulation credibility? How can they understand the technical concepts?

When managers and decision makers use simulation results, they shouldunderstand the confidence level of the results, as well as the potential risksassociated with any weaknesses or limitations. Later this year, NAFEMS will publish“Simulation Verification and Validation for Managers”, which will aim to address thisvery point. This article, based on an extract from the book, introduces the risks andtrade-offs of using simulation in decision making and discusses how perceptions ofrisk can differ between business organizations and regulatory agencies.

Verification, validation, and uncertainty quantification (VVUQ) should be viewed as atrade-off between increased confidence in simulation results and increased riskwhen using simulation results with unknown or poorly understood reliability. Whyshould managers invest further in simulation credibility even if their organization isalready reaping the benefits of simulation?

35

Perspectives of Risk and Decision ErrorsFormally, risk is defined as the product of probability andthe consequence of a harmful outcome. Although theprobability of a harmful outcome can be difficult toquantify from a simulation perspective, it is much moredifficult to quantify the short-term and long-termconsequences of a decision. While governmentorganizations mainly seek to minimize risk, businessesrecognize that decisions are also made to generate acompetitive advantage and new opportunities. Managersmust weigh the risks of making a wrong decision relativeto the opportunity or potential advantage whenconsidering simulation as part of the decision-makingprocess.

Organizations and individuals can view risk from differentand sometimes conflicting perspectives. Businessorganizations are interested in optimizing productperformance relative to profitability in a competitivebusiness environment. Consequently, product innovation,design procedures, manufacturing processes, andcertification activities are proprietary. In the use ofcomputer simulations, most businesses appear to be risk-tolerant when balancing the potential opportunitiesagainst the short-term and long-term risks. Businessesface product risk in diverse forms: legal liability, lostmarket share, lost customer confidence, environmentalimpact, political impact, and lost credibility with regulatoryagencies and customers. Some businesses are part of aregulated industry where certification risks are sharedwith regulatory agencies. Consequently, businessesshould not focus on just producing reliable simulationresults, but should focus on accumulating evidence of thecredibility of simulation results as an explicit element oftheir risk management planning for important programsthat rely heavily on simulation for success.

Government policy or regulatory agencies have anincreasing role in the credibility of simulation results inproduct development or design activities becausebusinesses are increasing their reliance on simulation inproduct performance and safety qualification. Regulatorydecisions have indirect value to the public, e.g.,certification of new medical device that could potentiallysave lives, but there is little direct value to the governmentor regulatory agency. Government and regulatory agenciesmust focus on public safety risks, environmental risks,and risks associated with the loss of political support orthe loss of public confidence in the agency. As a result,regulatory agencies tend to be more risk-averse in theiruse of simulation and, appropriately, tend to demand moretransparency, rigor, and documentation in the decision-making process.

Incorrect decisions derived from simulation informationare commonly blamed on simulations that are misleadingor even wrong. Examples include a simulation thatinadequately reduces the numerical error due to meshresolution or a simulation that inadequately models acoupled physics phenomena. This type of simulation erroris commonly referred to as model builder’s risk. In somecases, management’s expectations for simulation, oftenbased on polished marketing campaigns by software

companies, are inconsistent with the state-of-the-arttechnology in a particular discipline or class of physicalprocesses. Incorrect decisions can also be made by usingsimulations that are correct, but the results are misusedin various ways. This type of decision error is referred to asmodel user’s risk. For example, the simulation results aretechnically correct, but the simulation addresses thewrong problem. This is a common criticism of modelingand risk assessment when catastrophes occur and aredescribed in news media as a “complete surprise” or an“act of God”. A common root cause is that the simulationdid not address the relevant system environments andscenarios or that important failure mechanisms (bothtechnical and human) were not considered or wereincorrectly dismissed. Decision errors can be stronglyinfluenced by well-recognized cognitive and personalbiases in decision making, such as self-serving bias,cultural bias, or confirmation bias.

Many organizations equate the credibility of simulationresults with the experience and judgment of their senioranalysts. Without diminishing the value of senior analysts,an evidence-based approach to simulation credibility willreduce both the technical risk of producing incorrectsimulation results and management’s risk in the use ofsimulation results in a decision-making context.

Simulation, Credibility and Testing Trade-OffsThe risks and opportunities of simulation cannot beconsidered in a vacuum, but should be considered relativeto other information sources supporting decision making,such as physical testing and previous experience. Of thefour types of information sources (simulation, physicaltesting, simulation credibility, and experience), only thefirst three are considered as adjustable in the time frameduring which decisions are typically made. Decisionmakers are expected to do the best they can with what iscurrently available in terms of resources and schedule,thus necessitating trades and explicit compromises.Sometimes these trades and compromises occur at thecost of increased decision risks between simulation,testing, and credibility.

Testing can be viewed as full reality, partially revealed. Thetechnical and practical limitations of physical testing arethe inability to observe or measure all of the quantities ofinterest or to fully explore all of the environments andscenarios of interest. Testing, especially system testing,plays a unique role because it allows nature anopportunity to reveal surprises. However, testing has itslimitations and drawbacks, such as: (a) expense andextended time scales, (b) relevant system environmentsand final system geometries being impractical toreproduce and construct, and (c) performance marginsused to evaluate robustness of designs over a wide rangeof conditions being typically impossible to quantify.

Contrastingly, simulation can be viewed as approximatereality, fully revealed. While all models are approximationsto reality, many can be useful for the intended purpose ofdecision making. Simulation’s strength is anchored in the

36

fact that a large number of simulations can be efficientlycomputed to support design, the quantification ofperformance margins, and discovery over a very broadrange of environments and scenarios. Every minutiagenerated in a simulation can be examined to betterunderstand why the system is predicted to perform theway it does.

The spectrum of computer simulation uses) lends itselfto a hierarchy in terms of the requirements for (a)simulation accuracy and extensiveness of performanceand safety exploration and (b) detail andcomprehensiveness of the evidence of simulationcredibility (see Table 1). At the lower levels of thishierarchy (top row of the table), simulation can be used topredict trend behaviors and qualitative features forproduct innovation and preliminary design optimization.At this level, agility and rapid turnaround of simulationresults are the top priorities. At a more demanding level,simulation is relied upon to produce quantitative results,such as product design and optimization. At thisdemanding level, there is a balance between agility andsimulation credibility. At the highest level, simulation isrelied upon to provide performance and safetyguarantees, such as those required in extremelycompetitive environments and by government regulatoryagencies. Evidence-based credibility is required for boththe details of the simulation and the physical tests, aswell as the system scenarios and uncertaintiesinvestigated at the highest level. Typically, performance,safety, and reliability margins need to be quantified over awide range of normal and off-normal operatingconditions. At this top level of simulation accuracy andcredibility, managers may be risking the profitability oreven the survival of their corporation, while regulatorsmay be risking public safety or severe environmentalimpact.

The availability of relevant physical tests always reducesthe risk in using simulation. The greatest demands onsimulation credibility occur when high consequencedecisions need to be made without the benefit of relevanttests.

Higher fidelity simulations and credibility evidence arealways desired in order to minimize the shortcomingsassociated with testing. Developing that credibility,however, consumes time and resources above andbeyond that which is necessary to generate qualitativesimulation results. Validation is crucial when establishingthe credibility level of the simulation relative toexperimental observations of nature. In combination withvalidation, model or parameter calibration is commonlyrequired. Both validation and model calibration rely ontest results that are relevant to the system conditions ofinterest.

Simulation and testing are naturally synergistic in theirstrengths and weaknesses and, as a result, optimaldecisions come from an integration of simulation andtesting. There are always opportunities to traderesources and schedule between the elements ofsimulation, testing, and credibility. When decision risk islow, testing and establishment of credibility can be de-emphasized in favor of prior experience using similarmodels for similar applications. However, when decisionrisk is high, establishing decision-specific simulationcredibility is much more important. For example,managers would make different trades betweencost/schedule, computing capabilities, and modelinglimitations when faced with low-risk versus high-riskdecisions. Management must clearly evaluate,understand, and communicate the risk in trade-offsbetween simulation, testing, and credibility whendecision-risk and decision-opportunity are high.

The NAFEMS publication, "Simulation Verification and Validation for Managers", by the same authors,expands on the topics mentioned in this article and also covers simulation-informed decision making, aswell as the responsibilities and costs of simulation credibility. �

Simulation Risk

• Low-risk product innovationand preliminary designactivities

• Medium-risk decisionsupport activities

• High-risk simulation-informed or simulation-based certification

Simulation Uses

• Predict trend behaviors andqualitative features

• Quantify product design andoptimization

• Assess if performance,safety, and reliability goalsare met

• Quantify margins relative toperformance, safety, andreliability goals

Simulation Credibility

• Credibility primarily derivedfrom judgment andexperience

• Credibility derived fromsimulation managementprocesses and a balancebetween judgment and VVUQevidence

• Evidence derived fromsimulation managementprocesses

• Evidence-based (VVUQ)credibility is top priority

• Credibility evidence andsimulation details areformally documented

Table 1: Hierarchy of simulation credibility determined by intended use of simulations and simulation risk

Dr. Oberkampf will be presenting on this topic in the Simulation Governance sessionat the NAFEMS World Congress. (Session 5B - Tuesday 13th of June - 11:00)