Computational Framework for Computational Framework for Subsurface Energy and Subsurface Energy and

Environmental Modeling and Environmental Modeling and SimulationSimulation

Mary Fanett Wheeler, Sunil ThomasMary Fanett Wheeler, Sunil Thomas

Center for Subsurface ModelingCenter for Subsurface ModelingInstitute for Computation Engineering and SciencesInstitute for Computation Engineering and Sciences

The University of Texas at AustinThe University of Texas at Austin

Acknowledge Collaborators:

• Algorithms: UT-Austin (T. Arbogast, M. Balhoff, M. Delshad, E. Gildin, G. Pencheva, S. Thomas, T. Wildey): Pitt (I. Yotov); ConocoPhillips (H. Klie)

• Parallel Computation: IBM (K. Jordan, A. Zekulin, J. Sexton); Rutgers (M. Parashar)

• Closed Loop Optimization: NI (Igor Alvarado, Darren Schmidt)

Support of Projects: NSF, DOE, and Industrial Affiliates (Aramco, BP, Chevron, ConocoPhillips, ExxonMobil, IBM, KAUST)

Outline Introduction General Parallel Framework for Modeling Flow,

Chemistry, and Mechanics (IPARS)• Solvers• Discretizations• Multiscale and Uncertainty Quantification• Closed Loop Optimization

Formulations (IPARS-C02)

• Compositional and Thermal Computational Results

• Validation and Benchmark Problems Current and Future Work

Resources Recovery • Petroleum and natural gas recovery from

conventional/unconventional reservoirs • In situ mining • Hot dry rock/enhanced geothermal systems • Potable water supply • Mining hydrology

Societal Needs in Relation to Geological Systems

Site Restoration • Aquifer remediation • Acid-rock drainage

Waste Containment/Disposal • Deep waste injection • Nuclear waste disposal • CO2 sequestration • Cryogenic storage/petroleum/gas

Underground Construction • Civil infrastructure • Underground space • Secure structures

Petroleum Engineering

Highly Integrated Highly Integrated Multidisciplinary, Multiscale, MultiprocessMultidisciplinary, Multiscale, Multiprocess

GeologicalEng.

Geology

PhysicsMathematics

Mining Engineering

MechanicalEngineering

Computer Sciences

Geochemistry

Geophysics

Geomechanics

GeoHydrology

Civil Engineering

ExplorationCharacterization

Fluid Flow Waste

isolation

Earth StressesMechanicalRock/Soil Behavior

Mining Design/Stab, Waste,

Land Reclamation

ExplorationCharacterizationDiagnostics

HydrocarbonRecoverySimulationConstruction

Soil Mech.Rock Mech.Struct. Anal.

Drilling & Excav.,Support, Instruments

Code Development, Software Engineering

Waste isolation

Geomechanics

Complex Geosystem

ManagementOptimization and Control

Multiscale Simulation

Characterization & Imaging

Geophysical Interpretation

Uncertainty Assessment

Sensor Placement

Sensor Data Management

Petrophysical Interpretation

Multiphysics Simulation

A Powerful Problem Solving Environment

3D Visualization & Interpretation

Data Management

Long Range Vision: Characterization Long Range Vision: Characterization And Management of And Management of DiverseDiverse Geosystems Geosystems

Framework Components High fidelity algorithms for treating relevant physics:

• Locally Conservative discretizations (e.g. mixed finite element and DG)

• Multiscale (spatial & temporal multiple scales)• Multiphysics (Darcy flow, biogeochemistry, geomechanics)• Complex Nonlinear Systems (coupled near hyperbolic &

parabolic/ elliptic systems with possible discrete models)• Robust Efficient Physics-based Solvers (ESSENTIAL)• A Posteriori Error Estimators

Closed loop optimization and parameter estimation• Parameter Estimoation (history matching) and uncertainty

quantification

Computationally intense: • Distributed computing• Dynamic steering

The Instrumented Oil Field The Instrumented Oil Field

Detect and track changes in data during production.Invert data for reservoir properties.Detect and track reservoir changes.

Assimilate data & reservoir properties into the evolving reservoir model.

Use simulation and optimization to guide future production.

IPARS: Integrated Parallel and IPARS: Integrated Parallel and Accurate Reservoir SimulatorAccurate Reservoir Simulator

PHYSICS BASED SOLVERSPHYSICS BASED SOLVERS

PhysicsPhysics

Heterogeneity

Multiple Physics

K Tensor

Flow Regimes

Fractures

Well Operations

DiscretizationDiscretization

MFE

CVM

DG

MPFA

Mortar

FDM

Numerical representati

on

HPCHPC

Insights

SolversSolvers

AMG

AML

DD

Krylov

LU/ILU

Numerical Solution

RandomizedRandomizedAlgorithms

MultiresolutionMultiresolutionAnalysis

Random GraphRandom GraphTheoryTheory

ReinforcedReinforcedLearning

PhysicsPhysics-based-basedSolversSolvers

Why Multiscale? Subsurface properties vary on the

scale of millimeters Computational grids can be refined

to the scale of meters or kilometers Multiscale methods are designed to

allow fine scale features to impact a coarse scale solution• Variational multiscale finite

elements Hughes et al 1998 Hou, Wu 1997 Efendiev, Hou, Ginting et al

2004• Mixed multiscale finite elements

Arbogast 2002 Aarnes 2004

• Mortar multiscale finite elements Arbogast, Pencheva, Wheeler,

Yotov 2004 Yotov, Ganis 2008

Upscale

Basic Idea of the Multiscale Mixed Mortar Method

Multiscale Mortar Mixed Finite Element Method

Domain Decomposition and MultiscaleDomain Decomposition and Multiscale

Domain Decomposition

For each stochastic realization,time step and linearization

Compute data forinterface problemCompute data forinterface problem

Subdomainsolves

Preconditiondata

Preconditiondata

Solve theinterface problem

Solve theinterface problem

Solve local problemsgiven interface valuesSolve local problems

given interface values

Applyprecond.

Multiplesubdomain

solves

Multipleprecond.

applications

Subdomainsolves

Multiscale Approach

For each stochastic realization,time step and linearization

Compute data forinterface problemCompute data forinterface problem

Subdomainsolves

Compute multiscalebasis for coarse scaleCompute multiscale

basis for coarse scale

Solve theinterface problem

Solve theinterface problem

Solve local problemsgiven interface valuesSolve local problems

given interface values

Multiplesubdomain

solves

Multiple linearcombinations

of basis

Subdomainsolves

Domain Decomposition and MultiscaleDomain Decomposition and Multiscale

For each stochastic realization,time step and linearization

Compute data forinterface problemCompute data forinterface problem

Subdomainsolves

Preconditiondata

Preconditiondata

Solve theinterface problem

Solve theinterface problem

Solve local problemsgiven interface valuesSolve local problems

given interface values

ApplyMultiscaleprecond.

Fixed numberof subdomain solves

Fixed number ofmultiscale precond.

applications

Subdomainsolves

Compute the multiscalebasis for a training operator

Compute the multiscalebasis for a training operator

Multiple subdomain solves

Example: Uncertainty Quantification

360x360 grid 25 subdomains of equal size 129,600 degrees of freedom Continuous quadratic

mortars Karhunen-Loéve expansion

of the permeability truncated at 9 terms

Second order stochastic collocation

512 realizations Training operator based on

mean permeability

Mean Permeability

Mean Pressure

Number of Interface Iterations

Interface Solver Time

Example: IMPES for Two Phase Flow

360x360 grid 25 subdomains of equal

size 129,600 degrees of

freedom Continuous quadratic

mortars 50 implicit pressure

solves 100 explicit saturation

time steps per pressure solve

Training operator based on initial saturation

Absolute Permeability

Initial Saturation

Number of Interface Iterations

Interface Solver Time

Finite Element Oxbow Problem

FD & FEM Couplings: 3 Blocks with Fault

Solution

Continuous Measurement and Data Analysis for Reservoir Model

Estimation

Source: E. Gildin, CSM, UT-Austin

Continuous Measurement and Data Analysis for Reservoir Model

Estimation

IPARS

Data Acquisition (Sensors + DAQ)

Online Analysis(Data Fusion,

Denoising, Resampling…)

Optimization & Supervisory

Control

Reservoir

Data Assimilation(EnKF)

Field Controller(s)

DynamicI/F

Source: I. Alvarado and D. Schmidt, NI

Parameter Estimation Using SPSA

Key Issues in C02 Storage What is the likelihood and magnitude of CO2 leakage and what are

the environmental impacts?

How effective are different CO2 trapping mechanisms?

What physical, geochemical, and geomechanical processes are important for the next few centuries and how these processes impact the storage efficacy and security?

What are the necessary models and modeling capabilities to assess the fate of injected CO2?

What are the computational needs and capabilities to address these issues?

How these tools can be made useful and accessible to regulators and industry?

groundwater flow

CO2 leakage

deep brine aquifer

drinking-water aquifer

Global Experience in COGlobal Experience in CO22 Injection Injection

From Peter Cook, CO2CRC

COCO22 Sequestration Modeling Approach Sequestration Modeling Approach

Numerical simulationNumerical simulation Characterization (fault, fractures)Characterization (fault, fractures)

Appropriate griddingAppropriate gridding

Compositional EOSCompositional EOS

Parallel computing capabilityParallel computing capability

Key processesKey processes COCO22/brine mass transfer/brine mass transfer

Multiphase flowMultiphase flow

During injection During injection (pressure driven)(pressure driven)

After injection After injection (gravity driven)(gravity driven)

Geochemical reactionsGeochemical reactions Geomechanical modelingGeomechanical modeling

ParallelParallel

IPARS-COMPIPARS-COMP

ThermalThermal2-P Flash2-P Flash

GeomechanicsGeomechanics

Numerics Numerics

GraphicsGraphics

EOSEOS CompComp.GeochemicGeochemic

alalReactionReaction

GriddingGridding

SolversSolvers

Physical Physical PropProp

IPARS-COMP Flow EquationsIPARS-COMP Flow Equations

Mass Balance EquationMass Balance Equation

Pressure EquationPressure Equation

Solution MethodSolution Method Iteratively coupled until a volume balance convergence Iteratively coupled until a volume balance convergence

criterion is met or a maximum number of iterations criterion is met or a maximum number of iterations exceeded.exceeded.

ii i i i

N. u S D q

t

Thermal & Chemistry EquationsThermal & Chemistry Equations

Energy BalanceSolved using a time-split

scheme (operator splitting)Higher-order Godunov for

advectionFully implicit/explicit in time and

Mixed FEM in space for thermal conduction

ChemistrySystem of (non-linear) ODEsSolved using a higher order

integration schemes such as Runge-Kutta methods

Tp H

T

T s vs v

M T. C u T T q

t

Internal energy : M

M 1 C C S

Coupled Flow-Thermal-Chemistry AlgorithmCoupled Flow-Thermal-Chemistry Algorithm

CO2 EOR Simulations

ValidationValidation

SPE5 -- A quarter of 5 spot benchmark WAG problemSPE5 -- A quarter of 5 spot benchmark WAG problem

3-phase, 6 components C1, C3, C6, C10, C15, C203-phase, 6 components C1, C3, C6, C10, C15, C20

IPARS-CO2 vs CMG-GEMIPARS-CO2 vs CMG-GEM

Cum. oil produced Cum. gasInj

Prod

ValidationValidation

COCO22 pattern flood injection pattern flood injection3-phase, 10 components 3-phase, 10 components CO2, N2, C1, C3, C4, C5, C6, C15, C20CO2, N2, C1, C3, C4, C5, C6, C15, C20

IPARS-CO2 vs CMG-GEMIPARS-CO2 vs CMG-GEM

CO2 conc.Cum. gas

Inj

Prod.

Parallel SimulationsParallel Simulations

Modified SPE5 WAG injectionModified SPE5 WAG injection Permeability from SPE10Permeability from SPE10 160x160x40 (1,024,000 cells)160x160x40 (1,024,000 cells) 32, 64, 128, 256, 512 processors32, 64, 128, 256, 512 processors

Oil pressure and water saturation@ 3 yrs

Gas saturation and propane conc. @ 3 yrs

HardwareHardwareLonestar: Linux Lonestar: Linux cluster systemcluster system

Blue GeneP: CNK Blue GeneP: CNK system, Linux I/Osystem, Linux I/O

1,3001,300 Nodes / Nodes / 5,200 cores5,200 cores

262,144 Nodes / 262,144 Nodes / 1,048,576 cores1,048,576 cores

Processor Arch: Processor Arch: 2.66GHz, Dual 2.66GHz, Dual core, Intel Xeon core, Intel Xeon 5100, Peak: 55 5100, Peak: 55

TFlops/s TFlops/s

Processor Arch: Processor Arch: 850MHz, 850MHz,

IBM CU-08, Peak: IBM CU-08, Peak: ~1 PFlop/s~1 PFlop/s

8 GB/node8 GB/node 2 GB/node2 GB/node

Network: Network: InfiniBand, 1GB/sInfiniBand, 1GB/s

Network: Network:

10Gb Eth,1.7GB/s10Gb Eth,1.7GB/s

SoftwareSoftwareGMRES, BCGS, LSOR, Multigrid.GMRES, BCGS, LSOR, Multigrid.

MPI: MVAPICH2 library for parallel MPI: MVAPICH2 library for parallel communicationcommunication

Texas Advanced Computing Center The University of Texas at Austin

Parallel ScalabilityParallel Scalability

Scalability On Ranger (TACC) & Blue Gene PScalability On Ranger (TACC) & Blue Gene P

Ranger (TACC)Ranger (TACC) Blue Gene PBlue Gene P

GMRES solver with Multigrid Preconditioner3500ft, 3500 ft, 100ft reservoir40x160x160=1,024,000 elementsCPUs: 32, 64, 128, 256, 512, 1024

CO2 Storage Benchmark Problems

A Benchmark-Study on Problems Related to CO2 Storage inGeological formations, Summary and Discussion of the Results H. Class, A. Ebigbo, R. Helming et al., 2008

Benchmark Problem 1.1Benchmark Problem 1.1COCO22 Plume Evolution and Leakage via Plume Evolution and Leakage via

Abandoned WellAbandoned Well

ObjectiveObjectiveQuantification of leakage rate Quantification of leakage rate in deep aquifer @2840-3000 min deep aquifer @2840-3000 m

OutputOutput1- Leakage rate = %CO1- Leakage rate = %CO22 mass mass

flux/injection rateflux/injection rate

2- Max. leakage value 2- Max. leakage value

3- Leakage value at 1000 d3- Leakage value at 1000 d

=0.15

K = 20 md

P = 3.08x104 KPa

Benchmark Problem 1.1Benchmark Problem 1.1Leakage Rate of COLeakage Rate of CO22

CO2 BT: 10 days

Peak Leakage value: 0.23%Final leakage value: 0.11%Agrees with semi-analytic solution (Nordbotten et al.)

Comparison with Published ResultsComparison with Published Resultsat 80 daysat 80 days

Gas Saturation

Pressure

Ebigbo Ebigbo et al.et al., 2007, 2007IPARS-COMP

Frio Brine COFrio Brine CO22 Injection Pilot Injection Pilot

Bureau of Economic GeologyJackson School Of GeosciencesThe University of Texas at AustinFunded by DOE NETL

Frio Brine Pilot SiteFrio Brine Pilot Site

Injection interval: 24-m-thick, Injection interval: 24-m-thick, mineralogically complex fluvial mineralogically complex fluvial sandstone, porosity 24%, sandstone, porosity 24%, Permeability 2.5 DPermeability 2.5 D

Unusually homogeneousUnusually homogeneous Steeply dipping 16 degreesSteeply dipping 16 degrees 7m perforated zone7m perforated zone Seals Seals numerous thick shales, numerous thick shales,

small fault blocksmall fault block Depth 1,500 mDepth 1,500 m Brine-rock, no hydrocarbonsBrine-rock, no hydrocarbons 150 bar, 53 C, supercritical CO150 bar, 53 C, supercritical CO22

Injection interval

Oil production

From Ian Duncan

Frio Modeling EffortFrio Modeling Effort Stair stepped approximation on a 50x100x100 grid (~70,000

active elements) has been generated from the given data. Figure shows porosity in the given and approximated data.

Solution profilesSolution profiles Pressure and close-up of top-view of gas (CO2) saturation at t=33

days. Simulations on bevo2 cluster at CSM, ICES on 24 processors.

CO2 Plume TransportCO2 Plume Transport CO2 saturation as seen below the shale barrier at t=2 and 33 days.

Breakthrough time is observed to be close to 2 days.

Current Research Activities at CSMCurrent Research Activities at CSM

Model COModel CO22 injection either in deep saline aquifers or injection either in deep saline aquifers or depleted oil and gas reservoirs using compositional depleted oil and gas reservoirs using compositional and parallel reservoir simulator (IPARS-COand parallel reservoir simulator (IPARS-CO22))

Large scale parallel computingLarge scale parallel computing Efficiency with different solversEfficiency with different solvers Couple IPARS-COCouple IPARS-CO22 with geochemistry with geochemistry Couple IPARS with geomechanicsCouple IPARS with geomechanics Enhance EOS model and physical property Enhance EOS model and physical property

models (effect of salt, hysteresis, etc)models (effect of salt, hysteresis, etc) Data sources, field sites, practical applicationsData sources, field sites, practical applications

(in collaboration with Duncan from BEG at UT)(in collaboration with Duncan from BEG at UT) Gridding and a posteriori error estimatorsGridding and a posteriori error estimators OptimizationOptimization Risk and uncertainty analysisRisk and uncertainty analysis