an approximate dynamic programming approach to food ... · seoul, south korea, may 26-30, 2019 an...

13th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP13Seoul, South Korea, May 26-30, 2019

An approximate dynamic programming approach to food security ofcommunities following hazards

Saeed NozhatiGraduate Student, Dept. of Civil and Environmental Engineering, Colorado StateUniversity, Fort Collins, USA

Yugandhar SarkaleGraduate Student, Dept. of Electrical and Computer Engineering, Colorado StateUniversity, Fort Collins, USA

Bruce R. EllingwoodProfessor, Dept. of Civil and Environmental Engineering, Colorado State University, FortCollins, USA

Edwin K.P. ChongProfessor, Dept. of Electrical and Computer Engineering, Colorado State University, FortCollins, USA

Hussam MahmoudAssociate Professor, Dept. of Civil and Environmental Engineering, Colorado StateUniversity, Fort Collins, US

ABSTRACT: Food security can be threatened by extreme natural hazard events for households of all socialclasses within a community. To address food security issues following a natural disaster, the recovery ofseveral elements of the built environment within a community, including its building portfolio, must beconsidered. Building portfolio restoration is one of the most challenging elements of recovery owing to thecomplexity and dimensionality of the problem. This study introduces a stochastic scheduling algorithm forthe identification of optimal building portfolio recovery strategies. The proposed approach provides acomputationally tractable formulation to manage multi-state, large-scale infrastructure systems. A testbedcommunity modeled after Gilroy, California, is used to illustrate how the proposed approach can beimplemented efficiently and accurately to find the near-optimal decisions related to building recoveryfollowing a severe earthquake.

One of the principal objectives of the United Nations(UN) Sustainable Development Goals is achievingfood security. The Food and Agriculture Organiza-tion (FAO) describes food security as: "a situationthat exists when all people, at all times, have phys-

ical, social and economic access to sufficient, safeand nutritious food that meets their dietary needs andfood preferences for an active and healthy life" (FAO(2001)). Securing an adequate food supply to allcommunity inhabitants requires a food distribution

1

arX

iv:1

812.

1054

0v1

[cs

.CY

] 1

6 D

ec 2

018


system that is resilient to natural and man-made haz-ards. The growth of population in hazard-prone re-gions and climate change pose numerous challengesto achieving a resilient food system around the world.The resiliency concept applied to food distributionsystems can be evaluated with respect to two differ-ent time-frames, namely in "normal" times (i.e., priorto disasters) and in the aftermath of hazards. Sev-eral studies have investigated different approaches toenhance the resilience of agri-food systems (Seekellet al. (2017)). These studies have focused on re-silience in terms of biophysical capacity to increasefood production, diversity of modern domestic foodproduction, and the role played by social status andincome in the impact of food deficits. To mitigatefood security issues, the United States Departmentof Agriculture (USDA) Food and Nutrition Service(FNS) supplies 15 domestic food and nutrition assis-tance programs. The three largest are the Supple-mental Nutrition Assistance Program (SNAP - for-merly the Food Stamp Program), the National SchoolLunch Program, and the Special Supplemental Nu-trition Program for Women, Infants, and Children(WIC) (Oliveira (2017)). However, household foodsecurity following extreme natural hazard events isalso contingent on interdependent critical infrastruc-ture systems, such as transportation, energy, water,household units, and retailer availability.

This study focuses on the connection between fail-ures in food distribution and food retail infrastruc-ture and disruption in civil infrastructure and struc-tures. Household food security issues are consider-ably worsened following natural disasters. For ex-ample, Hurricanes Rita, Wilma, and Katrina, whichoccurred in 2005, caused disaster-related food pro-grams to serve 2.4 million households and distributed$928 million in benefits to households (Research andCenter (2017)). Three dimensions of food security -accessibility, availability, and affordability - are par-ticularly relevant for the nexus between infrastructureand household food security. Affordability capturesthe ability of households to buy food from food re-tailers, and is a function of household income, as-

sets, credit, and perhaps even participation in foodassistance programs. Accessibility is concerned withthe households’ physical access to food retail out-lets. Because at least one functional route must beavailable between a household unit and a function-ing food retailer, transportation networks are a ma-jor factor in accessibility. Availability is concernedwith the functionality of the food distribution infras-tructure, beginning with wholesalers, extending to re-tailers, and ultimately ending with the household asthe primary consumer. The functionality of food re-tailers and household units depends not only on thefunctionality of their facilities but also the availabil-ity of electricity and water. Therefore, the electricalpower network (EPN), water network (WN), and thebuildings housing retailers and household units mustbe considered simultaneously to address availability.

As is evident from the preceding discussion, foodsecurity relies on a complex supply-chain system. Ifsuch a system is disrupted, community resilience andthe food security will be threatened (Paci-Green andBerardi (2015)). In this paper, we focus only onhousehold unit structures, which forms the largest en-tity in community restoration. In this paper, we focuson household unit buildings, which usually form thelargest element of the built environment in commu-nity restoration. A literature review (Lin and Wang(2017)) shows that the recovery of building portfolioshas been studied far less than the recovery of otherinfrastructure systems. Building portfolio restorationis an essential element of availability and plays amajor role towards addressing food security issues.Effective emergency logistics demand a compre-hensive decision-making framework that addressesand supports policymakers’ preferences by provid-ing efficient recovery plans. In this study, we em-ploy Markov decision processes (MDPs) along withan approximate dynamic programming (ADP) tech-nique to provide a practical framework for represen-tation and solution of stochastic large-scale decision-making problems. The scale and complexity of build-ing portfolio restoration is captured by the proposedsimulation-based representation and solution of the

2


MDP. The near-optimal solutions are illustrated forthe building portfolio of a testbed community mod-eled after Gilroy, California, United States.

1. TESTBED CASE STUDYAs an illustration, this study considers the buildingportfolio of Gilroy, California, USA. The City ofGilroy is a moderately sized growing city in south-ern Santa Clara County, California, with a populationof 48,821 at the time of the 2010 census. The studyarea is divided into 36 rectangular regions organizedas a grid to define the properties of the communitywith an area of 42 km2 and a population of 47,905.Household units are growing at a faster pace in Gilroythan in Santa Clara County and the State of Califor-nia (Harnish (2014)). The average number of peopleper household in Gilroy in 2010 was 3.4, greater thanthe state and county average. Approximately 95% ofGilroy’s housing units are occupied. A heat map ofhousehold units in the grid is shown in Figure 1. Agedistribution of Gilroy is tabulated in Table 1.

Figure 1: Housing units over the defined grids.

Table 1: Age distribution of Gilroy (Harnish (2014)).

Age Group PercentChildren (0-17 years) 30.60Adults (18-64 years) 61

Senior Citizen (65+ years) 8.40

2. SEISMIC HAZARD AND DAMAGE AS-SESSMENT

The seismic hazard is a dominant hazard of Cali-fornia. Hence, we consider a seismic event of mo-ment magnitude Mw = 6.9 that occurs at one of theclosest points on the San Andreas Fault to down-town Gilroy with an epicentral distance of approxi-mately 12 km. We used the Abrahamson et al. (2013)ground motion prediction equation (GMPE) to evalu-ate the Intensity Measures (IM) and associated uncer-tainties, including the intra-event (within event) andinter-event (event-to-event) uncertainties, at the sitesof each of the 14,702 buildings in Gilroy. We as-sessed the damage to household units and food re-tailers with the seismic fragility curves presented inHAZUS-MH (FEMA (2003)). We considered repairvehicles, crews, and tools as available resource units(RUs) to restore the buildings following the hazard.One RU is required to repair each damaged build-ing (Masoomi (2018)). We adopted the synthesizedrestoration time from HAZUS-MH.

3. MARKOV DECISION PROCESS FRAME-WORK

We provide a brief description of MDPs; addi-tional details of MDPs are available elsewhere (Put-erman (2014)). A MDP is defined by the five-tuple(X ,A,P,R,γ), where X denotes the state space, A de-notes the action space, P(y|x,a) is the probability oftransitioning from state x ∈ X to state y ∈Y when ac-tion a is taken, , R(x,a) is the reward obtained whenaction a is taken in state x ∈ X , and γ is the discountfactor. A policy π : X −→ A is a mapping from statesto actions, and Π be the set of policies (π). The ob-jective is then to find the optimal policy, denoted byπ∗, that maximizes the total reward (or minimizes thetotal cost) over the time horizon, i.e.,

π∗ := arg sup

π∈Π

V π(x), (1)

where

V π(x) := E

[∞

∑t=0

γtR(xt ,π(xt))|x0 = x

], (2)

3


V π(x) is called the value function for a fixed policyπ , and 0 < γ < 1 is the discount factor (Puterman(2014)). The optimal value function for a given statex ∈ X is connoted as V π∗(x) : X −→ R given by

V π∗(x) := supπ∈Π

V π(x). (3)

Bellman’s optimality principle (Bertsekas (2005))is useful for defining Q-value function. Q-value func-tion plays a pivotal role in the description of the roll-out algorithm. Bellman’s optimality principle statesthat V π∗(x) satisfies

V π∗(x) := supa∈A(x)

[R(x,a)+ γ ∑

y∈XP(y|x,a)V π∗(y)

],

(4)The Q-value function associated with the optimalpolicy π∗ is defined as

Qπ∗(x,a) := R(x,a)+ γ ∑y∈X

P(y|x,a)V π∗(y), (5)

which is the inner-term on the R.H.S. in Eq. (4).Theoretically, π∗ can be computed with linear pro-

gramming or dynamic programming (DP). However,exact methods are not feasible for real-world prob-lems that have large state and action spaces, likethe community-level optimization problem consid-ered herein, owing to the curse of dimensionality;thus, an approximation technique is essential to ob-tain the solution. In the realm of approximate dy-namic programming (ADP) techniques, a model-based, direct simulation approach for policy evalu-ation is used (Sarkale et al. (2018)). This approach iscalled “rollout.” Briefly, an estimate Q̂π(x,a) of theQ-value function is calculated by Monte Carlo sim-ulations (MSC) in the rollout algorithm as follows:we first simulate NMC number of trajectories, whereeach trajectory is generated using the policy π (calledthe base policy), has length K, and starts from thepair (x,a); then, Q̂π(x,a) is the average of the samplefunctions along these trajectories:

Q̂π(x,a) =1

NMC∑

NMCi0=1

[R(x,a,xi0,1)+∑

Kk=1 γkR(xi0,k,π(xi0,k,xxi0,k+1))

].

(6)

For each trajectory i0, we fix the first state-action pairto (x,a); the next state xi0,1 is calculated when thecurrent action a in state x is completed. Thereafter,we choose actions using the base policy. A morecomplete description of the rollout algorithm can befound in (Bertsekas (2005); Nozhati et al. (2019)).

4. BUILDING PORTFOLIO RECOVERYEach household unit and retailer building remains un-damaged or exhibits one of the damage states (i.e.,Minor, Moderate, Major, and Collapse) based on thelevel of intensity measure and the seismic fragilitycurves. There is a limited number of RUs (definedearlier) available to the decision maker for the repairof the buildings in the community. In this study, wealso limit the number of RUs for each urban grid sothat the number of available RUs for each grid RUgis 20 percent of the number of damaged buildings ineach region of the grid. Therefore, the number ofRUs varies over the community in proportion to thedensity of the damaged buildings.

Let xt be the state of the damaged structures of thebuilding portfolio at time t; xt is a vector, where eachelement represents the damage state of each build-ing in the portfolio based on the level of intensitymeasure and the seismic fragility curves. Let ag

t de-note the repair action to be carried out on the dam-aged structures in the gth region of the grid at timet; each element of ag

t is either zero or a one, wherezero means do not repair and one means carry out re-pair. Note that the sum of elements of ag

t is equalto RUg. The repair action for the entire commu-nity at time t, at , is the stack of the repair actionag

t . The assignment of RUs to damaged locations isnon− preemptive in the sense that the decision makercannot preempt the assigned RUs from completingtheir work and reassign them to different locationsat every decision epoch t. This type of schedulingis more suitable when the decision maker deals withnon-central stakeholders and private owners, which isthe case for a typical building portfolio. We wish toplan decisions optimally so that a maximum numberof inhabitants have safe household unit structures per

4


unit of time (day in our case). Therefore, the rewardfunction embeds two objectives as follows:

R(xt ,at ,xt+1) =r

trep, (7)

where r is the number of people benefited fromhousehold units after the completion of at , and trepis the total repair time to reach xt+1 from any initialstate x0. Note that the reward function is stochasticbecause the outcome of the repair action is stochas-tic. In this study, we set the discount factor to be 0.99,implying that the decision maker is “far-sighted” inthe consideration of the future rewards.

We simulated NMC number of trajectories to reacha low (0.1 in this study) dispersion in Eq. (6). AsEq. (6) shows, we addressed the mean-based op-timization that is suited to risk-neutral decision-makers. However, this approach can easily addressdifferent risk aversion behaviors. Figure 2 showsthe total number of people with inhabitable struc-tures (undamaged or repaired) over the community.We also computed the different numbers of children,adults, and senior citizens that have safe buildingsover the recovery. Different age groups have differ-ent levels of vulnerability to food insecurity; for ex-ample, children are a vulnerable group and must bepaid more attention during the recovery process.

Figure 2: Different numbers of people based on age withinhabitable structures.

Figure 3 depicts the spatio-temporal evolutionof the community for people with inhabitable

structurally-safe household units. This figure showsthat for urban grids with a high density of damagedstructures, complete recovery is prolonged despiteavailability of additional RUs. The spatio-temporalanalysis of the community is informative for policymakers whereby they can identify the vulnerable ar-eas of the community across time.

Figure 3: Number of people with inhabitable houses a)following the earthquake b) after 100 days c) after 600days.

5. CONCLUSION AND FUTURE WORKThe building portfolio restoration is one of the mostchallenging ingredients to address food security is-sues in the aftermath of disasters. Our stochastic dy-namic optimization approach, based on the methodof rollout, successfully plans a near-optimal buildingportfolio recovery following a hazard. Our approachshows how to overcome the curse of dimensionalityin optimizing large-scale building portfolio recoverypost-diaster. For future work, we consider severalaspects of a community from infrastructure systemsto social systems along with their interdependencies.We will also explore how to fuse meta-heuristics toour solution to supervise the stochastic search thatdetermines the most promising actions (Nozhati et al.(2018)).

6. REFERENCESAbrahamson, N. A., Silva, W. J., and Kamai, R. (2013).

“Update of the as08 ground-motion prediction equa-tions based on the nga-west2 data set.

Bertsekas, P. (2005). Dynamic programming and optimalcontrol, Vol. 1. Athena scientific Belmont, MA.

FAO (2001). “The state of food insecurity in the world(rome: Fao).

5


FEMA, H. (2003). “Multi-hazard loss estimation method-ology, earthquake model.” Washington, DC, USA: Fed-eral Emergency Management Agency.

Harnish, M. (2014). “2015-2023 housing element policydocument and background report.

Lin, P. and Wang, N. (2017). “Stochastic post-disasterfunctionality recovery of community building portfo-lios i: Modeling.” Structural Safety, 69, 96–105.

Masoomi, H. (2018). “A resilience-based decision frame-work to determine performance targets for the built en-vironment.” Ph.D. thesis, Colorado State University. Li-braries, Colorado State University. Libraries.

Nozhati, S., Sarkale, Y., Ellingwood, B., Chong, E.,and Mahmoud, H. (2018). “A modified approximatedynamic programming algorithm for community-levelfood security following disasters.” Proceedings of the9th International Congress on Environmental Mod-elling and Software (iEMSs 2018), Fort Collins, CO,June.

Nozhati, S., Sarkale, Y., Ellingwood, B., Chong, E. K., andMahmoud, H. (2019). “Near-optimal planning usingapproximate dynamic programming to enhance post-hazard community resilience management.” ReliabilityEngineering & System Safety, 181, 116–126.

Oliveira, V. (2017). “The food assistance landscape: Fy2016 annual report.

Paci-Green, R. and Berardi, G. (2015). “Do global foodsystems have an achilles heel? the potential for regionalfood systems to support resilience in regional disasters.”Journal of Environmental Studies and Sciences, 5(4),685–698.

Puterman, M. (2014). Markov decision processes: dis-crete stochastic dynamic programming. John Wiley andSons.

Research, F. and Center, A. (2017). “An advocate’s guideto the disaster supplemental nutrition assistance pro-gram.” Food Research and Action Center (FRAC).

Sarkale, Y., Nozhati, S., Chong, E., Ellingwood, B., andMahmoud, H. (2018). “Solving markov decision pro-cesses for network-level post-hazard recovery via sim-ulation optimization and rollout.” Proceedings of the14th IEEE International Conference on AutomationScience and Engineering (CASE 2018), Munich, Ger-many.

Seekell, D., Carr, J., Dell’Angelo, J., D’Odorico, P., Fader,M., Gephart, J., Kummu, M., Magliocca, N., Porkka,M., Puma, M., et al. (2017). “Resilience in the globalfood system.” Environmental Research Letters, 12(2),025010.

6

an approximate dynamic programming approach to food ... · seoul, south korea, may 26-30, 2019 an...

Documents