validation of rismc models...2017/07/07 · – hazard analysis – plant-system and structure...
TRANSCRIPT
Validation of RISMC Models
Nam Dinh, Linyu Lin, Paridhi Athe
Multiphysics Model Validation WorkshopRaleigh, NC -- June 27-29, 2017
7/24/2017 1
Acknowledgement
The Integrated Research Project (IRP)
“Development and Application of a Data-Driven Methodology for Validation of Risk-Informed Safety Margin Characterization Models”
is supported by the U.S. Department of Energy’s Nuclear Energy University Program and managed by the Light Water Reactor Sustainability Program’s RISMC R&D Pathway.
7/24/2017 2
IRP Team
• Steve Prescott (INL)• Cristian Rabiti (INL)• Robert Youngblood (INL)• Weiju Ren (ORNL)• Ram Sampath (CPL)• Jeffrey Lane (Zachry)• Robert Sewell (RSA)• Anh Bui (NCSU)
• Maria Avramova (NCSU)• John Baugh (NCSU)• Igor Bolotnov (NCSU)• Nam Dinh (NCSU)• Abhinav Gupta (NCSU)• Hany Abdel-Khalik (Purdue)• Philippe Bardet (GWU)• Xiaodong Sun (U-Mich)
* Curtis Smith (INL, LWRS)
Contents
• Background – CSAU/EMDAP– PCMM– RISMC– RISMC models
• IRP Approach – Technical components– PCMQ
T-H model validation: Model Characteristics
• Designed to predict a broad range of nuclear (reactor coolant) system T-H behaviors– Various subsystems, components– Steady state regime, transients and accident scenarios
• Multi-fluid (multi-field) averaged conservation treatment (equations)– (Physics-based) Reduced Order Modeling– Use “closure relations” (or constitutive laws) to substitute for
information/ interactions lost in “averaging” • Large number of correlations,
– Each describes a particular physics/phenomenon/effect/ interaction– Each obtained under certain experimental conditions
• Low-dimensionality formulation (1D; 1.5D)– RELAP5, TRACE, CTF
• Exception GOTHIC (0D, 1D, 2D, 3D for CTH) and future STH code
System T-H model validation
• Legacy code development and assessment over decades– 1960-1980: RELAP series; TRAC– Since mid 90s: RELAP5.3D; TRACE– Large-scale test programs (e.g., LOFT, Semiscale, APEX)
• US NRC CSAU Code Scaling, Applicability and Uncertainty– PIRT Phenomena Identification and Ranking Table
• US NRC Reg.Guide 1.203 “Transient and Accident Analysis”– EMDAP: Evaluation Model Development and Assessment Process
• PIRT guided Assessment base (SET, IET)• Sensitivity and uncertainty analysis
• BEPU: Best Estimate Plus Uncertainty
System T-H model validation: Existing Approaches
• US NRC CSAU and EMDAP– High level, methodological (heuristic)– Acceptance criteria not defined– Limited implementation in practice– Little attention on code verification and solution verification
• Focused on validation– To a large extent, MET and IET calibration and benchmarks– Sequential calibration– “Viewgraph norm” comparison
• “User effects”– Nodalization effect
• “Discretization” error? “Averaging” error?
• Uncertainty Quantification– Lack of UQ-grade data – Deficient techniques on Data assimilation/ UQ for T-H complex models
Code Scaling, Applicability, and Uncertainty (CSAU)
7/24/2017CSAU, “Code Scaling, Applicability, and Uncertainty”, US NRC, NUREG/CR-5249 (Boyack et al.), 1989. US
NRC Regulatory Guide 1.157 (“Best-Estimate Calculations of ECCS Performance”, 1989). 8
• Specify scenario, System conditionIdentify and rank phenomena (PIRT)
• Select frozen code, provide necessary documentation
Element 1: Requirement and Code Capability
• Establish assessment matrix for validation (set of integral and separate effect test)• Define Nodalization• Determine accuracy, scale effects, bias & uncertainty
Element 2: Assessment and ranging of parameters
• Determine impact of input parameter uncertaintyCombine bias and uncertainty + additional margin
Element 3: Sensitivity and uncertainty analysis
Systematic process for Complexity resolution and
uncertainty quantification
Westinghouse Non-Proprietary Class 3 © 2014 Westinghouse Electric Company LLC. All Rights Reserved.
Evaluation Model Development and Assessment Process- EMDAP
Six Basic Principles (Elements) ofEvaluation Model Development andAssessment Process (EMDAP) arepresented in Transient and Accident Analysis Methods (USNRC, Regulatory Guide 1.203, 2005).
Scaling Analysis is a part of EMDAP and Transient and Accident Analysis Methods.
9
Regulatory Guide 1.203, “Transient and AccidentAnalysis Methods,” USNRC,December 2005
Evaluation Model Development and Assessment Process (EMDAP)
7/24/2017EMDAP, " Regulatory Guide 1.203: Transient and Accident Analysis Methods," US Nuclear Regulatory
Commission, Office of Nuclear Regulatory Research, 2005. 10
• Specify scenario, System condition, figure of merit• Identify, system, components, geometries, fields and process to be modeled
Identify and rank phenomena and process (PIRT)
Element 1: Establish Requirement for Evaluation Model Capability
• Identify Separate Effect Tests (SETs) and Integral Effect Tests (IET) • Evaluate IET distortions and SET scale up capability• Determine experiment uncertainty
Element 2: Develop Assessment Base
• Establish EM structure• Develop and incorporate closure models
Element 3: Develop Evaluation model (EM)
• Determine model pedigree and applicability • Assess model fidelity and accuracy• Assess scalability of models and integral calculations• Determine EM biases and uncertainty
Element 4: Assess Evaluation Model Adequacy
Emphasis on Hierarchical
system analysisAnd
Scaling analysis
• Probabilistic Risk Analysis– For each component, the failure criterion is firstly characterized by comparing the load and capacity,
where the load is calculated deterministically by system code. The safety margin is then defined by the ratio (or the difference) of an applied capacity to the load. i.e. Load/Capacity (or Load-Capacity)
• Risk Analysis for External Hazards– Hazard analysis– Plant-system and structure response analysis– Evaluation of fragility and vulnerability of components– Plant system and sequence analysis– Consequence analysis
11
Loss of Preferred Power Event Tree
Fragility curvesHazard intensity
curve
• Risk-Informed Safety Margins Characterization– Risk-Informed Safety Margins Characterization
(RISMC) Pathway is to “conduct research and development for advanced methods and tools to support NPP safety assessments and management”
– Load and capacity distributions are determined by advanced simulation tools
• RISMC Analysis for External Hazards of Flooding
12
Load Capacity
INL, 2016
Science and Practical Challenges
Risk = ∑ Probability * Consequence = ∑ P * C i
[Small] [Large]
i
Uncertainty
Sequences
Uncertainty in Characterization of Rare Events, High Hazards
Experience, Culture
Knowledge, Capability (R&D)
Decision Making, Regulation
$$$ (ALARA)
Risk, Safety and Licensing: From Perception to Practice and Public Acceptance
What ArePossible UpsetConditions?
How Do They Occur?
How Often DoThey Occur?
What AreConsequences?
Imagination, Precaution,
Operating Experience
Comprehensive Safety Analysis vs. Surrogate Licensing Criteria
PRA: Probabilistic Risk Assessment
DSA: Deterministic Safety Analysis (Mechanistic)
Experiments, Simulations
How plant aging affects these four questions?
How confident are we in the answers to these questions (UQ)?
16
Safety Figure of Merit
a number
a number
LoadCapacity
“A decision made without taking uncertainty into account is barely worth calling a decision”,
R.E. Wilson (1985)
CapacityLoad
?
AgingPower uprate, Increasing Fuel burnups
Higher frequencies of IEsNew IEs and sequencesCascading failures
Potential Developments
Risk-Informed Safety Margin Characterization (RISMC)
Safety Figure of Merit
The REMDAP of SPH (Flood Simulation) and STH in flooding-induced extended SBO RISMC applications
System T-H model validation: RISMC Requirements
• RISMC requires a large number of simulations– Computationally efficient– (Yet) Sufficiently accurate
• Realistic scenarios– Sufficiently accurate representation of safety-significant SSCs– Realistic boundary conditions for sub-domain analysis
• Realistic simulations– Bounded numerical errors– Bounded model parameter errors– No significant model form biases
• Predictive capability maturity evaluated to be Level 1, preferred Level 2– See PCMM definition of levels
• Uncertainty quantification– Uncertainty sources identified, evaluated, and propagated to FOM/ QOI
System T-H model validation: Data-Driven
• Data requirements– Use high-resolution validation-grade data
• From relevant computational and physical experiments– Covering key phenomena identified in PIRT
• Sufficient coverage of parameter range to ensure prototypic flow patterns
• Data status– Limited amount of data, primarily from METs and
IETs• Scaling (applicability) is subjective
– Potential use of high-resolution CFD/ CMFD • Including DNS, LES, and RANS
RISMC Models
V&V-UQ Methods & Tools
RISMC Applications Validation DataIntegrated
Research
Key attributes of the IRP approach
• Risk-informed approaches for collection and characterization of evidence,
• Decision-oriented framework for quantification of “validity” (or “maturity”), with a probabilistic criterion for adequate level of validation.
• Techniques for physics-based simulation-aided scaling, uncertainty and sensitivity analysis (to compute and optimize the relevance and value of experiments),
• Methods for data-driven model development and assessment process that effectively utilizes data from physical and numerical experiments;
Key Ideas on Validation
• Validation of RISMC Tool = – RISMC Tool is to support a practical decision
• Decision is always about managing a risk– Risk is rooted in uncertainty
– Decision-Oriented Risk-Informed, S/UA-aided Validation• (S/UA = sensitivity/uncertainty analysis)
• Validation = Predictive Capability Maturity– Maturity for a Decision
• Reliability / Robustness of Decision
• Validation– Accuracy assessment– Adequacy assessment
Key Ideas on Validation Data
• “Data Realism”– Data Quality/Relevancy/ Applicability
• Always deficient;• New data (massive)• Multiscale heterogeneous data
• Data generation– Advanced diagnostics experiments– High-fidelity numerical simulation
• Data-driven – Data assimilation
William L. Oberkampf, Martin Pilch, and Timothy G. Trucano, SAND2007-5948
Predictive Capability Maturity Model for Computational Modeling and Simulation
Stringency increases with potential impact of decision (consequences) to be made with
using findings from simulation
Risk-informed Validation
Decision Model
Elements of the Framework forPredictive Capability Maturity Quantification
(PCMQ)
7/24/2017 27
Model CoverageCMP = PM/PP
PCMQ: How good is our validation?
PP
PIRT-basedPhenomena
Pyramid
PM
Code System Model
Pyramid
PE
ValidationExperiment
Pyramid
PCMQ = CMP *CME*CEP
7/24/2017
Elements of the framework
Preprocessing for the framework development Specification of issue, simulation
tool, decision objective Specification of scenario, FOM Phenomena resolution PIRT
Structural Knowledge representation Hierarchical pyramids and Goal
Structuring notation (GSN)
Classification and Characterization of Evidence Direct evidence Indirect evidence (Process quality
assurance) Data characterization
• Scaling analysis
Formulation of the decision model Structural decision model Maturity quantification
• Bayesian Network• Fuzzy logic & fuzzy
inference system• Evidential reasoning
Phenomena knowledge base, model base and data base for validation
GSN modules
II. STRUCTURAL KNOWLEDGE REPRESENTATION
VII. REFINEMENT
GSN Modules
III. CLASSIFICATION AND CHARACTERIZATION OF EVIDENCES
V. EVALUATION
VI. APPRAISAL
ACT
IV. DECISION MODEL
I. PREPROCESSING REQUIREMENT FOR THE FRAMEWORK DEVELOPMENT
Computable Network
Structural decision model
GSN module
Structural Knowledge Representation- Implementation Requirements
Clarity and Traceabilityo Detailed structured information representationo Multi-level information abstraction
Indicators for knowledge, importance, status of work Ability to manage and organize large chunks of information
Tool/technique employed Goal Structuring Notation (Adelard’s Assurance and safety case environment, ASCE 4.2)o Goal Structuring Notation (GSN) is an argument modeling technique that was
primarily developed to represent assurance arguments for safety caseso “How does argumentation helps in structuring knowledge?”
7/24/2017 30
Overview of Goal structure Notation
7/24/2017 31
Maturity Quantification
Step 1: Formulate the decision model using GSNo Provide criteria, sub-criteria, evidence incorporation, and dependency relations
Step 2: Transforming the GSN based decision model into a computable network
Step 2: Quantitative Confidence assessmento Decision model consists of two sets of information / data: (i) Subjective data
based on expert opinion, and (ii) objective data based on evidenceo Assimilate these two sets of information to provide quantitative maturity
assessment
7/24/2017 32
Case study : Maturity Quantification ( Fuzzy Logic & Fuzzy Inference System)
7/24/2017 33
Assumptions Measurement uncertainty is negligibleValidation results(VR) are evaluated on the basis of “Bias”VR=1-BiasData applicability (DA) is determined by scaling analysisDA=1-SD, SD scale distortion
Maturity quantification for this example would imply a quantitative evaluation of the claim G1, i.e. “Code x is suitable for predicting the application of interest.”
34
Fuzzy Logic
If (DA is MF1) and (VR is MF2) then (CA is MF3)
Deg
ree
of m
embe
rshi
pD
egre
e of
mem
bers
hip
Deg
ree
of m
embe
rshi
p
Validation Results (VR)
Code adequacy (CA)
Data Applicability (DA)
MF1
MF3
MF2
CA
DA VR
CA
VR DA
Indirect evidence (Process quality assurance factors)
PQA of Phenomenology pyramid based on Process factors:o Past Use What is confidence in the past use of the techniques used for phenomena resolution (i.e.
PIRT process : Knowledge and importance grading)?o Peer review What is the confidence that adequate peer review of the PIRT has been carried out? o Definition What is the confidence that phenomenon have been clearly defined/described in the context
of the Application of interest?o Standard Procedure What is the confidence that the standard procedure as specified in the EMDAP
process has been followed?
All factors can be evaluated based on the confidence expressed as:No opinion / Low / Medium / High
7/24/2017 35
Process
• Past Use• Peer review• Definitions• Standard procedure
Personnel
• Past Knowledge• Domain Expirience• Independence• Competency
Tool/Techniques
• Bound qualification• Standard qualification
Case Study: Representation of Phenomenology Pyramid (PIRT) using GSN
Process Quality Assurance of Phenomenology pyramid Development of structured Phenomenology Pyramid Example: CASL Challenge Problem CRUD Induced Power Shift (CIPS)
7/24/2017 36
Phenomenology pyramid for CIPS (showing only Top goal and sub-goals)
7/24/2017 37
All nodes corresponding to each Sub tree (G 1.1, G1.2, G 1.3 and G 1.4 ) has beencollapsed for clarity and visibility.“[+]” sign indicated that the node has further expansion.ASCE 4.2 facilitates expansion/collapse of any part of the network in the GSN tree.
Coupling is not considered in this decomposition but can be incorporated as a separate goal at a later stage (see assumption A1a)
Multi-level information abstraction
7/24/2017 38
Multi-level information abstraction
7/24/2017 39
Solution node contains evidences in the form ofexcerpt or link from research paper, CASLreports or other documentation, e.g., Sn1.1.2.2 contains the evidence that explains howCRUD removal due to mechanical contractionin Clad can impact the FOM 3 (Axial offset).In this way, supporting evidence for a completebranch ( G 1, G 1.1, G 1.1.2, G 1.1.2.2) canbe incorporated in the GSN tree
Furtherdecomposed in
Decomposition of Goal G 1.1, Governing phenomena in Fuel Performance
Concluding Remarks (for PCMM)
PCMQ supports implementation of PCMM by providing a structured approach for knowledge representation, information abstraction, integration and evidence incorporation
It provide support for structural organization and classification of evidence for maturity assessment
Subjective information based on expert knowledge can be incorporated in the quantification process using Bayesian inference / fuzzy logic.
PCMQ can provide a formal approach for code review that may help in expediting the code licensing process
7/24/2017 40