Advances in Water Resources 126 (2019) 79–95

Contents lists available at ScienceDirect

Advances in Water Resources

journal homepage: www.elsevier.com/locate/advwatres

PRIMo: Parallel raster inundation model

Brett F. Sanders a,b,∗, Jochen E. Schubert a
a
Department of Civil and Environmental Engineering, UC Irvine, Irvine, CA 92697, USA
b
Department of Urban Planning and Public Policy, UC Irvine, Irvine, CA 92697, USA

a r t i c l e i n f o a b s t r a c t

Keywords: Simulation of flood inundation at metric resolution is important for making hazard information useful to a wide
Flood inundation modeling range of end-users involved in flood risk management, and addressing the alarming increase in flood losses that
Godunov-type scheme have been observed over recent decades. However, high data volumes and computational demands make this
Finite volume method
challenging over large spatial extents comparable to the metropolitan areas of major cities where flood impacts
Parallel computing
are concentrated, especially for time-sensitive applications such as forecasting and repetitive simulation for uncer-
Digital elevation model
Upscaling tainty assessment. Additionally, several factors present difficulties for numerical solvers including combinations
Flood hazards of steep and flat topography that promote transcritical flows, the need to resolve flow in relatively narrow fea-
Flood risk tures such as drainage channels and roadways in urban areas which channel flood water during extreme events,
and the need to depict compound hazards resulting from the interaction of pluvial, fluvial and coastal flooding. A
new flood inundation model is presented here to address these challenges. The Parallel Raster Inundation Model
(PRIMo) solves the shallow-water equations on an upscaled grid that is far coarser than the underlying raster
digital topographic model (DTM), and uses a subgrid modeling approach so that the solution benefits from DTM-
scale topographic data. Additionally, an approximate Riemann solver is applied in an innovative way to integrate
fluxes between cells, as needed to update the solution by the finite volume method, which makes the method
applicable to subcritical, supercritical and transcritical flows. PRIMo is implemented using a two-dimensional
domain decomposition approach to Single Process Multiple Data (SPMD) parallel computing, and overlapping
communications and computations are implemented to yield ideal parallel scaling for well-balanced test cases.
With both a subgrid model and ideal parallel scaling, the model can scale to meet the demands of any applica-
tion. Several benchmarks are presented to demonstrate predictive skill and the potential for timely, whole-city,
metric-resolution flooding simulations. Limitations of the methods and opportunities for improvements are also
presented.

1. Introduction
Flooding has emerged as the most significant natural hazard facing
society (Jongman et al., 2012). Over the last decade, flooding accounted
for half of all weather-related disasters and affected 2.3 billion people
(CRED, 2015). The U.S. set a record in 2017 with $300 billion in disaster
losses from hurricanes and flooding (NOAA, 2018), and over the sum-
mer of that year, a humanitarian crisis emerged in Nepal, Bangladesh
and India where more than 1000 people died and at least 41 million
were affected by monsoon flooding and landslides (Gettleman, 2017).
Additionally, the human health risks of flooding are significant and un-
acceptable (CRED, 2015; Di Baldassarre et al., 2010; Few et al., 2004;
Watts et al., 2015). Several trends including rising sea levels, urbaniza-
tion, deforestation, and rural-to-urban population shifts are expected to
increase the impacts of flooding in the future (Hallegatte et al., 2013;
Jongman et al., 2012).
Flood inundation modeling is taking on an increasingly important
role in flood risk management and disaster risk reduction initiatives
(Djordjević et al., 2011; Hénonin et al., 2015; Sanders, 2017). While
the use of hydraulic or hydrodynamic flood models has traditionally
been limited to engineering design (e.g., sizing of culverts, placement
and height of levees), implementation of flood insurance programs, and
support of dam safety programs, many more applications are emerg-
ing as communities: (a) shift focus from controlling flooding hazards
to minimizing flooding consequences (Southgate et al., 2013), and (b)
embrace participatory methods for planning, preparedness and design
whereby experts and stakeholders co-develop interventions (Heimhuber
et al., 2015; Maskrey et al., 2016; Smith et al., 2017). Flood simulations
are powerful tools for anyone to immediately grasp the consequences of
flooding based on familiar reference points, and to identify steps that can

∗
Corresponding author at: Department of Civil and Environmental Engineering, UC Irvine, Irvine, CA 92697, USA.
E-mail addresses: bsanders@uci.edu (B.F. Sanders), jschuber@uci.edu (J.E. Schubert).
URL: http://sanders.eng.uci.edu (B.F. Sanders).

https://doi.org/10.1016/j.advwatres.2019.02.007
Received 12 October 2018; Received in revised form 16 February 2019; Accepted 17 February 2019
Available online 20 February 2019
0309-1708/© 2019 Elsevier Ltd. All rights reserved.
B.F. Sanders and J.E. Schubert
be taken to minimize impacts. Additionally, flood simulations can help
to orient diverse groups of stakeholders around flood risks and facilitate
dialogue about how to manage it (Djordjević et al., 2011; Lane et al.,
2011). Indeed, flood simulations offer support for all phases of the dis-
aster management cycle including mitigation, planning, preparedness,
early warning, emergency response and recovery (Mackay et al., 2015;
Sanders, 2017; Wilkinson et al., 2015). What is critical when using flood
simulations for flood risk management is the iterative engagement of
stakeholders. Early stakeholders involvement is important for building
trust and configuring simulations around priority issues facing commu-
nities (Evers et al., 2012; Luke et al., 2018). Repeated engagement is
important for ensuring that model output is tailored to local decision-
making needs (Djordjević et al., 2011; Evers et al., 2012; Luke et al.,
2018; Meyer et al., 2012; Sanders, 2017).
For simulations to depict flooding in an intuitive way that anyone can
understand, an accurate geometrical model of flood zones provided by a
digital elevation model is essential and high quality data are increasingly
available from laser scanning (lidar) and photogrammetry (Bates, 2012;
Sampson et al., 2012; Schumann et al., 2009). Moreover, there have
been recent calls to improve the availability of high quality data globally
based on its importance for flood management (Schumann et al., 2014).
However, using metric resolution data becomes increasingly challeng-
ing as the spatial extent of the modeling domain increases. The memory
of computing systems is strained by the vast number of points that need
to be resolved, and the workload of processors is intense based on the
number of points and flood model requirements for a short time steps
(seconds or less) for accuracy and stability. This has motivated several
lines of recent flood inundation modeling research including more ef-
ficient solution algorithms (Bates et al., 2010; Cea and Bladé, 2015),
parallel computing methodologies (Brodtkorb et al., 2012; Castro et al.,
2011; Neal et al., 2010; Sanders et al., 2010; Vacondio et al., 2014), and
subgrid models (sometimes called porosity models) for aggregating the
effects of fine scale topographic data at a relatively coarse resolution,
where flow calculations are made (Casulli and Stelling, 2011; Defina,
2000; Guinot et al., 2017; Hénonin et al., 2015; Neal et al., 2012a;
Özgen et al., 2016; Sanders et al., 2008; Soares-Frazão et al., 2008;
Stelling, 2012; Volp et al., 2013). Version 5.0 of the popular HEC-RAS
model (U.S. Army Corps of Engineers, Hydraulic Engineering Center,
Davis, CA) now supports two-dimensional flood inundation modeling
and uses an implicit finite volume method with a subgrid model for to-
pography, similar to the scheme reported by Casulli and Stelling (2011).
Models based on implicit finite volume or difference schemes are es-
pecially advantageous when there are slowly varying subcritical flows
because the time step is restricted based on the fluid velocity, not the
speed of gravity waves as with explicit models, and thus solutions can
be attained using fewer time steps (and fewer computations). However,
the advantage is diminished or reversed, compared to explicit meth-
ods, when flow is supercritical, changing quickly, or when there are
flood fronts moving across the land surface. Under these circumstances,
an implicit solver may need to use several iterations per time step to
converge and the allowable time step may become smaller. Moreover,
explicit schemes tend to support better parallel scaling because all cal-
culations are made locally compared to implicit schemes, which rely
on the solution of a (large) matrix problem to simultaneously update
the solution over the entire flow domain, and thus parallel efficiency
is closely tied to the parallel efficiency of the matrix solver used. The
LISFLOOD-FP model uses an explicit scheme for solving a simplified
form of the shallow-water equations and has received widespread use
for large scale flood inundation modeling with moderate resolution (10–
90 m) Digital Terrain Models (DTMs) (Bates et al., 2010; Neal et al.,
2012a; Wing et al., 2018), and it has been configured for parallel exe-
cution with shared memory hardware, distributed memory hardward,
and graphical processing units (Neal et al., 2010). Several variants of
LISFLOOD-FP have been reported with different features, including
schemes that use an approximate Riemann solvers to account for tran-
scritical flow (Neal et al., 2012b), and the version of LISFLOOD-FP
80
Advances in Water Resources 126 (2019) 79–95
used for large scale flood hazard model is designed to resolve subcritical
flows based on its simplified formulation (Neal et al., 2012a; Wing et al.,
2018). In summary, there are many different approaches for simulat-
ing flood inundation using different shallow-water equations, different
subgrid models, and different numerical solvers, but thus far there has
been limited success developing models that apply to all classes of flow
(subcritical, supercritical, and transcritical) and all sources of flooding
(pluvial, fluvial and coastal) and that efficiently scale on parallel com-
puting systems with hundreds or even thousands of processors. Models
that respond to this challenge could have tremendous impact as a flood
risk management tool, communicating flood hazards to a wide range
of end users in support of all phases of the disaster management cycle,
and especially serving the presently unmet needs for real-time decision-
support during major floods (Luke et al., 2018; Sanders, 2017).
This paper presents a flood simulation model that solves the shallow-
water equations by an innovative Godunov-type, finite volume al-
gorithm. The model is grounded in several conventional features of
Godunov-type finite volume schemes including use of the HLLC solver
to estimate fluxes at cell edges (Toro, 2001), fluxes and bed slope
source terms that are “well balanced” to prevent spurious acceleration
(Valiani and Begnudelli, 2006), and the implicit treatment of friction
terms for accuracy and stability (Xia et al., 2017). Hence, this paper
emphasizes aspects of the model that enable relatively fast execution
in large scale applications including: (a) a single process multiple data
(SPMD) parallel implementation based on domain decomposition which
limits both memory and workload on a per processor basis, (b) a sub-
grid model that supports a relatively coarse-grid solution update, and (c)
innovative optimizations of the subgrid model that further reduce com-
putations without overly sacrificing accuracy. Optimization involves the
sampling of topographic data in the subgrid model, which reduces the
required number of calls to the computationally demanding approxi-
mate Riemann solver and enables faster execution of look-up tables.
The model is designed to load elevation (and resistance) data from com-
monly used raster formatted digital topographic models (DTMs), mak-
ing it relatively easy to set up compared with unstructured grid models
that are constrained by important land features like levees and build-
ing walls, (e.g., Bilskie et al., 2015; Hagen et al., 2001; Schubert and
Sanders, 2012; Schubert et al., 2008). This inspires the model name:
Parallel Raster Inundation Model (PRIMo).
PRIMo is capable of resolving subcritical, supercritical and transcrit-
ical flows based on its use of an approximate Riemann solver to estimate
fluxes, and while it is formally first order accurate, it considers slopes in
the free surface elevation when reconstructing fluxes at cell edges which
improves accuracy and also enables more accurate downscaling for im-
proved visualization of results. Applications of the model are presented
to verify the model including its mass conservation and well-balanced
properties, document strengths and weaknesses, and report benchmark
performance standards useful for informing the readiness of the method
for large scale and/or real-time modeling applications where computa-
tional efficiency is paramount.
2. Model formulation
PRIMo solves the nonlinear shallow-water equations which predict
water level and horizontal velocities under the assumption of a hydro-
static pressure distribution and turbulent momentum dissipation scaled
by a Manning resistance coefficient. The integral form of the govern-
ing equations can be written for a two-dimensional (2D) domain Ω
with boundary Γ and unit outward normal vector 𝐧 = (𝑛𝑥 𝑛𝑦 )𝑇 as follows
(Valiani and Begnudelli, 2006),
𝜕𝑡 𝐔 dΩ + (𝐅 − 𝐙) ⋅ 𝐧 dΓ = 𝐑 dΩ + 𝐏 dΩ (1)
∫Ω ∫Γ ∫Ω ∫Ω
where 𝐔 = (𝑑 𝑢𝑑 𝑣𝑑 )𝑇 , d = depth, 𝑢 = 𝑥-velocity, 𝑣 = 𝑦-velocity, 𝐅 =
(𝐅𝐱 𝐅𝐲 ) where
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 1. PRIMo solves volume and momentum balance equations on a

relatively coarse “U grid” (red) compared with the “z grid” of elevation
40 data (black). In this example, the z grid has a cell size Δ𝑧 = 2 m and the
19 upscale factor is 𝛼 = 10. The indices (i, j) apply to the z grid, and the
17
indices (im , jm ) apply to the U grid. Note that edges of the U grid align
30 2 with edges of the z grid, and thus the cross-sectional shape along U grid
15
edges is ambiguous. In PRIMo, elevation data from either side of the
13 edge are combined (see methods). (For interpretation of the references
11 to colour in this figure legend, the reader is referred to the web version
20
9 of this article.)
7
10 1 5
3
1
0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

1 2 3
-10

-10 0 10 20 30 40 50 60 70

⎛ 𝑢𝑑 ⎞ ⎛ 𝑣𝑑 ⎞ downwards direction. Hence, careful attention to the ordering of data

⎜ 2 ⎟ ⎜ ⎟ points is needed when reading and writing gridded data. Based on this
𝐅𝐱 = ⎜𝑢 𝑑 + 1 𝑔𝑑 2 ⎟ and 𝐅𝐲 = ⎜ 𝑢𝑣𝑑 ⎟ (2)
2 number system, topographic data within model cell im , jm are indexed by
⎜ ⎟ ⎜ 2 1 ⎟
⎝ 𝑢𝑣𝑑 ⎠ ⎝𝑣 𝑑 + 2 𝑔𝑑 2 ⎠ 𝑖 = (𝑖𝑚 − 1)𝛼 + 1, … , 𝑖𝑚 𝛼 and 𝑗 = (𝑗𝑚 − 1)𝛼 + 1, … , 𝑗𝑚 𝛼. However, to sim-
and finally Z, R and P are related to traditional source terms of the plify the presentation in the material that follows, i and j will always be
shallow-water equations, bottom slope, resistance and the effective pre- presented as local indices to reference the 𝛼 2 z values located within an
cipitation (rainfall converted to runoff), respectively, and are given as individual U grid cell.
follows, Because edges of the flow model cells are not aligned with the center
of topographic data cells, there is ambiguity in the definition of topogra-
⎛ 0 0 ⎞ ⎛0⎞ ⎛1⎞ phy data used to compute fluxes, and different approaches can be taken.
( ) ⎜1 2 ⎟ ⎜ ⎟
𝐙 = 𝐙𝐱 𝐙𝐲 = ⎜ 2 𝑔𝑑𝜂𝑜 0 ⎟ 𝐑 = −𝑔𝑑 ⎜𝑆f𝑥 ⎟ 𝐏 = 𝑃 ⎜0⎟ (3) For example, Volp et al. (2013) used all topographic data from a ΔU × ΔU
⎜ ⎟ ⎜ 𝑦⎟ ⎜ ⎟ domain centered on the edge in the context of a semi-implicit finite vol-
⎝ 0
1
𝑔𝑑𝜂2 ⎠ ⎝ 𝑆f ⎠ ⎝0⎠
2 𝑜 ume scheme. For a Godunov-type scheme, it is important to define a
where g = gravity, 𝑆f𝑥 and 𝑆f𝑦 represent x- and y direction friction slopes, specific cross-sectional shape connecting two neighboring U grid cells,
respectively, 𝑑𝜂𝑜 = 𝜂𝑜 − 𝑧 and represents the depth based on a cell-wise across which an approximate Riemann solver can be applied. Therefore,
constant free surface elevation 𝜂 o , and P is the effective rainfall rate. a set of 𝛼 edge elevation points, 𝑧̂ 𝑘 , 𝑘 = 1, ⋯ , 𝛼 are computed as follows,
The form of the shallow-water equations appearing in Eq. (1) is cho-
sen because it simplifies balancing of fluxes and source terms when 𝑧̂ 𝑘 = max(𝑧̂ ∗𝑘 , minL (𝑧), minR (𝑧)) (4)
needed to preserve stationary solutions (Valiani and Begnudelli, 2006).
In essence, the computation of 𝐅 − 𝐙 along each edge of a flow model
where
cell represents a difference between the fluxes computed by the Rie-
mann solver and hydrostatic fluxes associated with the ground slope 1 L
𝑧̂ ∗𝑘 = (𝑧 + 𝑧R
𝑘) (5)
source term. In the limit of a stationary solution, these two terms are 2 𝑘
identical and thus 𝐅 − 𝐙 = 0 on an edge by edge basis. Additional detail
Here, 𝑧L𝑘 and 𝑧R𝑘
represent elevation in the z grid cell to the left and
is provided in Section 2.4.
right, respectively, of the kth segment of the U grid edge, and min L (z)
and min R (z) represent the minimum elevation among the 𝛼 2 elevation
2.1. Data structure data points in the U grid cells on the left and right, respectively, of the U
grid edge. Hence, z values along U grid edges are taken as a simple av-
Similar to previously reported models involving upscaling of topo- erage of neighboring z grid data points unless the average is lower than
graphic data (Hénonin et al., 2015; Volp et al., 2013), PRIMo is de- the minimum elevation of z data within neighboring U grid cells. This
signed to load a raster grid of elevation data, z, with cell size Δz in both condition prevents the reconstruction of positive depth values, d > 0, at
the x and y directions and to solve Eq. (1) on a relatively course grid with U grid edges when adjacent cells are dry (no storage), which will lead
a cell size Δ𝑈 = 𝛼Δ𝑧 , where 𝛼 is an integer termed the upscale factor. to spurious fluxes. As will be described next, the free surface elevation
To succinctly differentiate between grids in the narrative that follows, in U grid cells is set equal to the height of the lowest of its 𝛼 2 z values
we will refer to the fine resolution grid as the “z grid” and the coarse when the cell is dry.
resolution grid as the “U grid”. PRIMo also accommodates a raster grid The discrete solution represents a cell average over the U grid cell,
of Manning resistance parameter values, nm , with cell size Δz . Ω𝑖𝑚 ,𝑗𝑚 with area Δ2𝑈 and includes information about storage per unit
Fig. 1 shows the U grid (red) superimposed on the z grid (gray) based area, h, and cell-average discharge per unit width in the x-direction, p,
on 𝛼 = 10, as well as the indices for the z grid, i, j, and the indices for the and y-direction, q, as follows,
U grid, im , jm . The origin for the indices is the lower left hand corner of
the grid, whereas raster-formatted data commonly uses an origin in the ⎛ ℎ⎞
upper left hand corner of the grid, with row indices that increase in the 𝐔𝑖𝑚 ,𝑗𝑚 = ⎜ 𝑝 ⎟ (6)
⎜ ⎟
⎝ 𝑞 ⎠𝑖
𝑚 ,𝑗𝑚
81
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

where whereby fewer than 𝛼 calls to the Riemann solver are made to integrate
1 the flux along the U grid edge. As an aside, the number of Riemann
ℎ𝑖𝑚 ,𝑗𝑚 = 𝑑 dΩ (7)
Δ2𝑈 ∫Ω𝑖𝑚 ,𝑗𝑚 calls can also be reduced by using a cross-sectionally averaged Riemann
solver that is called only once per cell edge, but this requires the intro-
1 1 duction of look-up tables for interpolating the cross-sectional geometry
𝑝𝑖𝑚 ,𝑗𝑚 = 𝑢𝑑 dΩ, 𝑞𝑖𝑚 ,𝑗𝑚 = 𝑣𝑑 dΩ (8)
Δ2𝑈 ∫Ω𝑖𝑚 ,𝑗𝑚 Δ2𝑈 ∫Ω𝑖𝑚 ,𝑗𝑚 as a function of water level, which adds to the computational effort.
Further investigation is left for a future study.
The notation h is introduced to distinguish the total storage of water
The solution is updated by Δt from time level n to 𝑛 + 1 using a frac-
in each U grid cell (an integral property of the solution) from the point-
tional step method such that friction terms are treated implicitly and
wise depth d which varies spatially within the cell and is downscaled
therefore do restrict the time step beyond what is required for stability
to the z grid resolution. Similarly, the notation p and q is also used to
in the absence of friction. In the first step, the solution is advanced to
designate integral, cell-average, properties of the solution in contrast to
an intermediate step by based on the flux terms as follows,
the pointwise variables ud and vd, respectively.
To link water level 𝜂 and storage h in each U grid cell, storage curves [( 𝛼 ) ( 𝛼 )
Δ𝑡 ∑ ∑
𝑛
are computed by summing over z grid cells as follows, 𝐔𝑖 ,𝑗 = 𝐔𝑖 ,𝑗 −
∗
(Δ𝐅𝐱 L )𝑘 𝑖𝑚 +1∕2,𝑗𝑚 − (Δ𝐅𝐱 R )𝑘 𝑖𝑚 −1∕2,𝑗𝑚
𝑚 𝑚 𝑚 𝑚 𝛼Δ𝑧 𝑘=1 𝑘=1
𝛼 𝛼
1 ∑∑ ( 𝛼
∑
) ( 𝛼
∑
) ]
ℎ(𝜂) = max(0, 𝜂 − 𝑧𝑖,𝑗 ) (9)
𝛼 𝑖=1 𝑗=1
2 + L
(Δ𝐅𝐲 )𝑘 𝑖𝑚 ,𝑗𝑚 +1∕2 − (Δ𝐅𝐲 )𝑘 𝑖𝑚 ,𝑗𝑚 −1∕2 𝑛
R
(13)
𝑘=1 𝑘=1
Eq. (9) assumes a horizontal water level within each U grid cell,
which is a good approximation of most floods, but for improved accu- where
racy PRIMo considers slopes in the water level, 𝛿 x 𝜂 and 𝛿 y 𝜂 when up-
Δ𝐅𝐱 L = 𝐅𝐱 − 𝐙𝐱 L Δ𝐅𝐱 R = 𝐅𝐱 − 𝐙𝐱 R (14)
dating the solution in time as reported by Begnudelli (2016). Slopes are
computed using the minmod limiter (e.g., Sanders and Bradford, 2006).
For example, to compute slopes in the x direction, first the left and right Δ𝐅𝐲 L = 𝐅𝐲 − 𝐙𝐲 L Δ𝐅𝐲 R = 𝐅𝐲 − 𝐙𝐲 R (15)
slope are computed separately as follows,
and the superscripts L and R indicate that 𝐙 is evaluated on the left and
(𝛿𝑥 𝜂)L = 𝑎 = 𝜂𝑖𝑚 ,𝑗𝑚 − 𝜂𝑖𝑚 −1,𝑗𝑚 (𝛿𝑥 𝜂)R = 𝑏 = 𝜂𝑖𝑚 +1,𝑗𝑚 − 𝜂𝑖𝑚 ,𝑗𝑚 (10) right hand side, respectively, relative to an observer positioned with the
Cardinal directions (i.e., either x or y) positive to the right. Here, the
and in a second step, the minmod limiter is applied to give the slope,
summation notation reflects the integration of fluxes over each U grid
{
sgn(𝑎) min(|𝑎|, |𝑏|), if 𝑎𝑏 > 0 cell edge based its 𝛼 values of z.
(𝛿𝑥 𝜂)𝑖𝑚 ,𝑗𝑚 = (11)
0, otherwise In the second step, the solution is advanced to the next full time level
by integrating the friction terms implicitly through a two-point iteration
This procedure is repeated in the y direction to compute (𝛿𝑦 𝜂)𝑖𝑚 ,𝑗𝑚 . as follows (Begnudelli, 2016),
We note that PRIMo does not compute slopes in discharge values as is
common in second order accurate Godunov-type finite volume schemes. 𝐔∗∗
𝑖 ,𝑗 = 𝐔𝑖
∗
𝐌𝑛𝑖 (16)
𝑚 𝑚 𝑚 ,𝑗𝑚 𝑚 ,𝑗𝑚
We summarize the main data structures in PRIMo as follows: topo-
graphic data z and (optionally) Manning resistance parameters nm are 𝐔𝑛𝑖 +1
,𝑗 = 𝐔𝑖 𝐌∗∗
∗
(17)
𝑚 𝑚 𝑚 ,𝑗𝑚 𝑖 ,𝑗
𝑚 𝑚
raster fields with a cell size Δz , and topographic data along edges of the
U grid, 𝑧̂ are also saved with a cell size of Δz . The solution to the shallow where
water equations is given by h, p, q, 𝜂, 𝛿 x 𝜂 and 𝛿 y 𝜂 which is saved on a the ⎛1 0 0 ⎞
U grid with resolution Δ𝑈 = 𝛼Δ𝑧 , where 𝛼 is the integer upscale factor. 𝐌 = ⎜0 (1 + 𝑟)−1 0 ⎟ (18)
To downscale the solution to the z grid resolution at location xi , yj , the ⎜ ⎟
⎝0 0 (1 + 𝑟)−1 ⎠
free surface elevation, 𝜂𝑖,𝑗
∗ , is computed as follows,
Here, r is a dimensionless flow resistance parameter that is described
𝜂𝑖,𝑗
∗
= 𝜂 + 𝛿 𝑥 𝜂 ( 𝑥 𝑖 − 𝑥 𝑐 ) + 𝛿𝑦 𝜂 ( 𝑦 𝑗 − 𝑦 𝑐 ) (12) in the next section. Iteration becomes increasingly important when the
where xc , yc represents the cell-center coordinates of the U grid cell. The depth is very small. Use of two iterations was found to be adequate in
downscaled depth is subsequently computed as 𝑑𝑖,𝑗 = max(𝜂𝑖,𝑗 ∗ − 𝑧 , 0).
𝑖,𝑗
the previously reported upscale-type models (Begnudelli, 2016), while
others developing Godunov-type flood inundation models have recom-
2.2. Solution update scheme mended use of the Newton–Raphson method to ensure that iterations
are adequate (Xia et al., 2017).
The solution is updated using a Godunov-type finite volume scheme,
which involves the application of approximate Riemann solvers to com- 2.3. Resistance discretization
pute mass and momentum fluxes. However, a distinguishing feature of
PRIMo is that the approximate Riemann solver is applied 𝛼 times along The resistance parameter r follows from an implicit discretization of
each U grid edge, once for each z grid edge, and summed to give the the source term Eq. (3) that is unconditionally stable. When solving the
integral along the boundary as indicated by Eq. (1). Hence, PRIMo is classical shallow-water equations, the resistance parameter is commonly
designed to leverage the fine resolution topographic data. In Godunov- evaluated as follows,
type finite volume schemes, the computation of fluxes by approximate
𝑟𝑛 = Δ𝑡𝑔𝑛2𝑚 𝑉 𝑛 ∕(ℎ𝑛 )4∕3 (19)
Riemann solvers is one of the most computationally demanding steps. In
a traditional model with N × N cells (for simplicity), there are 2𝑁 2 + 2𝑁 where 𝑉 𝑛 = [(𝑢𝑛 )2 + (𝑣𝑛 )2 ]1∕2 . However, when there is a distribution of
edges where the Riemann solver is applied and N2 cells where the so- topographic heights within a U grid cell and a distribution of resis-
lution is updated. However, by limiting flux calculations to the U grid tance parameters, Eq. (19) can lead to significant errors in the estimate
edges, the required number of calls to the Riemann solver is reduced to of the friction slope (Stelling, 2012). Several researchers have formu-
2𝑁 2 ∕𝛼 + 2𝑁 and the number of solution updates is reduced to (N/𝛼)2 . lated expressions designed to aggregate the effects of a spatial distri-
Hence, calls to the Riemann solver and solution updates are reduced by bution of depth, velocity and empirical resistance parameters such as
a factor of 𝛼 and 𝛼 2 per time step. We will show later that the num- Manning/Strickler or Chézy (Defina, 2000; Stelling, 2012; Volp et al.,
ber of Riemann calls can be reduced further with a sampling technique 2013). Here, we assume that the friction slope (and water level) is

82
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

spatially uniform within each U grid cell, and that the Manning equa- 2.4.1. HLLC Solver
tion can be applied individually to each z grid cell based on di,j and The Harten, Lax and van Leer (HLL) solver is a well known approx-
(nm )i,j (Stelling, 2012; Volp et al., 2013). Hence, starting with the imate Riemann solver that has been applied extensively in Godunov-
Manning equation for the magnitude of the discharge per unit width, type shallow-water models, and the HLLC solver is an extension for
1∕2
(𝑝2 + 𝑞 2 )1∕2 = 𝑆f 𝑑 5∕3 ∕𝑛𝑚 , the average discharge with each U grid cell two-dimensional flows that contain a shear wave (Toro, 2001). Imple-
follows as, mentations of the HLLC solver differ depending on how wave speeds are
computed and treatment of dry cells, and the approach used here is what
(𝑝2 + 𝑞 2 )1∕2 = (𝑆𝑓 f )1∕2 Σf (20)
Toro (2001, p. 182–183) describes as “a simpler version,” as follows.
where When both the left and right side cell is dry, 𝐅HLL = 0. Otherwise,
𝛼 𝛼
1 ∑ ∑ 5∕3 fluxes in the x-direction, 𝐅𝐻 = (𝐹1𝐻 𝐹2𝐻 𝐹3𝐻 )𝑇 , are computed as fol-
Σf = 𝑑 ∕(𝑛𝑚 )𝑖,𝑗 (21)
𝛼 2 𝑖=1 𝑗=1 𝑖,𝑗 lows,
⎧𝐹1,2 𝑠L > 0
L
Expressions for the friction source terms in the x and y direction ⎪𝐹 R
𝐹1H,2 = ⎨ 1,2 𝑠R < 0 (30)
appearing in Eq. (1) follow as,
⎪ 𝑠R 𝑈1,2 −𝑠L 𝑈1,2 +𝑠L 𝑠R (𝑈1,2 −𝑈1,2 )
L R R L

𝑔𝑑𝑆f𝑥 = 𝑔 ℎ𝑝(𝑝 + 𝑞 )2 2 1∕2

∕Σ2f 𝑔 𝑑𝑆f𝑦 = 𝑔 ℎ𝑞 (𝑝 + 𝑞 )
2 2 1∕2
∕Σ2𝑓 (22) ⎩ 𝑠R −𝑠L
otherwise

where the overline reflects a cell-average and we note that h, p and and
{
q appearing on the right side of these equations represent U grid cell- 𝐹1H 𝑣L 𝐹1H > 0
𝐹3H = (31)
average quantities. Finally, the dimensionless flow resistance parameter 𝐹1H 𝑣R 𝐹1H ≤ 0
r required by Eq. (16) is given by, When dL > 𝛿 w and dR > 𝛿 w , both the left and right sides of the edge
𝑛 𝑛+1 𝑛 2 𝑛 2 1∕2
𝑟 = Δ𝑡𝑔 ℎ [(𝑝 ) + (𝑞 ) ] ∕(Σ𝑛f +1 )2 (23) are wet and the wave speeds are given by,

and the expression used for Eq. (17) is given by, 𝑠L = min(𝑢L − 𝑎L , 𝑢∗ − 𝑎∗ ), 𝑠R = max(𝑢R + 𝑎R , 𝑢∗ + 𝑎∗ ) (32)
where 𝑎𝐿 = (𝑔𝑑𝐿 )1∕2 , 𝑎R = (𝑔𝑑R )1∕2 , and
𝑟∗∗ = Δ𝑡𝑔 ℎ𝑛+1 [(𝑝∗∗ )2 + (𝑞 ∗∗ )2 ]1∕2 ∕(Σ𝑛f +1 )2 (24)
1 1 1
Note that checks are needed to avoid division by zero. 𝑢∗ = (𝑢L + 𝑢R ) + 𝑎L − 𝑎R , 𝑎∗ = (𝑎L + 𝑎R ) + (𝑢L − 𝑢R ) (33)
2 2 4
Application of Eq. (23) is computationally demanding because of In the case where the left side of the edge is dry (dL ≤ 𝛿 w ), the wave
the need to sweep over 𝛼 2 topographic data points (z grid cells) for each speeds are based on the exact solution to the dry-bed problem as follows
U grid cell. Run time computational costs are minimized by creating
look-up tables for Σf as a function of 𝜂 in a pre-processing step. In the 𝑠L = 𝑢R − 2𝑎R , 𝑠R = 𝑢R + 𝑎R (34)
construction of lookup tables, inclusion of basis points comparable to the
and in the case where the right side of the edge is dry (dR ≤ 𝛿 w ), the
finest resolved depths is important for accuracy and stability (Xia et al.,
exact solution is used again,
2017).
𝑠L = 𝑢L − 𝑎L , 𝑠R = 𝑢𝐿 + 2𝑎L (35)
2.4. Computation of fluxes For fluxes in the y-direction, the HLLC solver differs only slightly,
⎧𝐹1,3 𝑠L > 0
L
The first step of the solution update, Eq. (13), involves the summa-
⎪𝐹 R 𝑠R < 0
tion of 𝛼 separate fluxes for each U grid cell edge. Calculation of fluxes 𝐹1H,3 = ⎨ 1,3 (36)
⎪ 𝑠R 𝑈1,3 −𝑠L 𝑈1,3 +𝑠L 𝑠R (𝑈1,3 −𝑈1,3 )
L R R L
is nearly identical in the x- and y-directions, so the description here is
⎩ 𝑠R −𝑠L
otherwise
limited to the x direction as follows,
and
(Δ𝐅𝐱 L )𝑘 = 𝐅H {
𝑘 − (𝐙𝐱 )𝑘 (Δ𝐅𝐱 R )𝑘 = 𝐅H
𝑘 − (𝐙𝐱 )𝑘
L R
(25)
𝐹1H 𝑢L 𝐹1H > 0
𝐹2H = (37)
where FHdenotes application of the Harten, Lax and van Leer (HLLC) 𝐹1H 𝑢R 𝐹1H ≤ 0
Riemann solver, as described in the next section, and the superscripts and the wave speeds are computed the same as in the x-direction, only
L and R imply reconstruction of the solution from data contained in U v is used in place of u. We note that Roe’s approximate Riemann solver
grid cells to the left and right of the cell edge, respectively. To apply the for the shallow-water equations (e.g., Bradford and Sanders, 2002) was
approximate Riemann solver for each value of z along the U grid edge, also tested and found to perform similarly to the HLLC solver at slightly
the depth on the left and right side is reconstructed as follows, greater ( ∼ 10%) computational expense.
1 1
𝑑𝑘L = max(𝜂 L + 𝛿 𝜂 L − 𝑧̂ 𝑘 , 0) 𝑑𝑘R = max(𝜂 R − 𝛿𝑥 𝜂 R − 𝑧̂ 𝑘 , 0) (26)
2 𝑥 2 2.5. Stability
and the discharge per unit width is reconstructed based on a tolerance
for depth, 𝛿𝑤 = 1 × 10−6 m, as follows The time step Δt is constrained by a Courant, Friedrichs, Lewy (CFL)
{ L,R condition which on a 2D cartesian grid appears as follows,
𝑝 𝑑𝑘L,R > 𝛿𝑤
𝑝L𝑘 ,R = (27) 𝜆max 𝜆max
𝑦 Δ𝑡
0 otherwise 𝑥 Δ𝑡
+ ≤𝛽 (38)
Δ𝑈 Δ𝑈
{ L,R
𝑞 𝑑𝑘L,R > 𝛿𝑤 where 𝜆max
𝑥,𝑦 represent the maximum wave speeds in the x- and y-
𝑞𝑘L,R = (28) directions, respectively, and 𝛽 = 1∕2 when 𝜂 slopes (𝛿 x 𝜂 and 𝛿 y 𝜂) are
0 otherwise
computed to update the solution. If the 𝜂 slopes are set to zero, this cor-
On the other hand, to compute the bottom slope terms Zx L and Zx R responds to a purely first order accuracy mode and 𝛽 = 1. The stability
(cf. Eq. (3)), the depth is reconstructed assuming a horizontal free sur- condition can be simplified assuming 𝜆max ≈ 𝜆max as follows,
𝑥 𝑦
face,
𝜆max Δ𝑡 1
(𝑑𝜂𝑜 )L𝑘 = max(𝜂 L − 𝑧̂ 𝑘 , 0) ( 𝑑 𝜂𝑜 ) R ≤ 𝛽 (39)
𝑘 = max(𝜂 − 𝑧̂ 𝑘 , 0)
R
(29) Δ𝑈 2
An important consequence of the above discretization of fluxes and Note that the maximum allowable time step for stability is increased
bottom slope source terms is that ΔFx L , ΔFx R , ΔFy L and ΔFy R are identi- by a factor of 𝛼 compared to a model run at a resolution of Δz , and this
cally zero at every z grid edge for stationary solutions defined by 𝛿𝑥 𝜂 = 0, implies that fluxes and solution updates are reduced by a factor of 𝛼 2
𝛿𝑦 𝜂 = 0, 𝑝 = 0 and 𝑞 = 0. and 𝛼 3 overall, respectively.

83
B.F. Sanders and J.E. Schubert
26 26
24 24
22 22
20 20
0 5 10 0 5
26 26
24 24
22 22
20 20
0 5 10 0 5
2.6. Data sorting, sampling, and look-up tables
PRIMo sorts the 𝛼 2 values of z in each U grid cell, and the 𝛼 values
of 𝑧̂ on each U grid edge, from lowest to highest in a pre-processing step.
Using the sorted data, two look-up tables are computed for each U grid
cell: storage as a function of water level, h(𝜂), and a friction parameter
as a function of water level, Σ2𝑓 (𝜂). The sorted edge data also set the
sequence by which fluxes are integrated along cell edges as indicated
by Eq. (13), i.e., the fluxes for the deepest edge are computed first and
the fluxes for the shallowest edge are computed last.
Computational costs are further reduced in PRIMo by limiting the
number of entries in look-up tables, which reduces the basis points used
for interpolation, and by limiting the number of 𝑧̂ values where fluxes
are computed, which reduces the number of calls to the approximate
Riemann solver. If we consider the number of z data points used for
either interpolation or flux calculation to be Np , then 𝑁𝑝 = 𝛼 for U grid
edges and 𝑁𝑝 = 𝛼 2 for U grid areas. To implement a sampling strategy
involving a reduced number of points Ns < Np , the indices for sampling,
𝑠𝑚 , 𝑚 = 1, ⋯ , 𝑁𝑠 , are computed as follows,
𝑠𝑚 = 1 + f loor [(𝑚 − 1)(𝑁𝑝 − 1)∕(𝑁𝑠 − 1)] 𝑚 = 1, ⋯ , 𝑁𝑠 (40)
which ensures that the lowest and highest values of z are always used
as basis points, in addition to intermediate points. Furthermore, when
fluxes are computed using a reduced number of z data points, each flux is
weighted by 𝛼/Ns to give the total flux correctly along the U grid edge
as required by Eq. (13). The process of sorting, sampling, and evenly
weighting the chosen topographic data points is presented graphically
in Fig. 2. When using a sampling strategy for z, the total number of edges
required for flux calculations is reduced in proportion to fraction of cells
that are sampled. For example, if 𝛼 = 10 and 𝑁𝑠 = 5 as shown in Fig. 2,
then the number of calls to the Riemann solver is reduced 50%. The
notation used later to indicate sample size for flux calculations, interpo-
lation of volume versus water level, and interpolation of the resistance
term versus water level is given by 𝑁𝑠f lux , 𝑁𝑠vol , and 𝑁𝑠fric , respectively.
2.7. Parallel implementation
PRIMo is configured for parallel execution using Message Passing In-
terface (MPI) directives so it is applicable to either shared memory, dis-
tributed memory or hybrid high performance computing systems. SPMD
is adopted whereby domain decomposition is used to create separate in-
put data for each process. The spatial data defined by a Nx × Ny raster
84
Advances in Water Resources 126 (2019) 79–95
Fig. 2. The topographic profile along U grid edges (A) defined by 𝛼 values
of elevation, 𝑧̂ 𝑘 , 𝑘 = 1, ⋯ , 𝛼, are sorted from lowest to highest (B), and a
reduced number of topographic data points such as 𝑁𝑠f lux = 3 (C) or 𝑁𝑠f lux =
5 (D) can be used to integrate fluxes along U grid edges. Note that the
sampling procedure retains the lowest and highest elevation value from
the cross-section.
10
10
at resolution Δz is subdivided into a grid of Mx × My subdomains (or
tiles) each containing Nx /Mx × Ny /My raster cells at resolution Δz. Care
is taken in grid preparation so Nx /Mx and Ny /My are multiples of the
upscale factor 𝛼 to enable the subgrid modeling technique. In many ap-
plications, there will be one or more tiles within the Mx × My grid where
no flooding occurs and there is no need to perform flood computations.
Under these conditions, PRIMo can be configured so processes only run
on tiles of data where flooding is expected.
The solution variables needed to compute fluxes on edges that di-
vide subdomains are exchanged between processes ever time step. Along
each subdomain boundary edge, the variables passed between processes
are on u, v and 𝜂 from the first layer of boundary data and 𝜂 from the sec-
ond layer of boundary data. Since these variables are all defined on the U
grid, there are Nx /(𝛼Mx ) values of each variable exchanged in the along
horizontal edges, and Ny /(𝛼My ) values of each variable shared along
vertical edges. These variables are packed into a single buffer for each
process subdomain, so at most eight messages are exchanged between
each process every time step (two-way messaging with four neighbors).
Additional messages containing topographic data are exchanged in a
pre-processing step so each process has the necessary data to compute
the elevation data required for fluxes and source terms, 𝑧̂ . Two layers
of 𝜂 values are exchanged so 𝜂 slopes (𝛿 x 𝜂 and 𝛿 y 𝜂) can be computed
within the first layer of U grid cells inside and outside of the subdomain
boundary.
Computing and communications are overlapped to maximize parallel
efficiency. Specifically, flux and slope calculations that are not depen-
dent on exchanged boundary data (edges and cells within the interior
core of each subdomain) are performed in parallel with non-blocking
MPI directives that exchange data between processes. This leads to
fluxes for edges along subdomain boundaries, and slopes in cells along
boundary edges, that are computed once by two different processes, and
thus greater computational effort overall than for sequential execution.
Duplication of computations could be avoided by sharing a combina-
tion of solution data and flux data across between subdomains, but this
comes at the cost of greater complexity to the message passing strat-
egy (and greater latency), and the added cost appears to have negligible
impact on performance based on results presented in the next section.
Lastly, MPI_ALLREDUCE is called once per time step to compute a
global time step (for all processes) that satisfies the CFL condition on all
subdomains. The main time loop of the algorithm proceeds as follows,
• Time Loop Begins
B.F. Sanders and J.E. Schubert
• Boundary data packed into buffer for transfer: one layer of u and
v values and two layers of 𝜂 values.
• Non-blocking MPI_IRECV called to receive boundary data.
• Non-blocking MPI_ISEND called to send boundary data.
• Slopes 𝛿 𝜂 and 𝛿 𝜂 computed for the interior core.
x y
• Fluxes computed for the interior core.
• MPI_WAIT called to ensure boundary data transfer complete.
• Transferred buffers unpacked into solution variables.
• Slopes computed for layer of cells (i) inside and (ii) outside of
model subdomain boundary.
• Fluxes computed on (i) boundary edges and (ii) first interior
edges of flow model subdomain grid.
• MPI_ALLREDUCE called to set global time step, Δt.
• Solution advanced Δt from time level n to 𝑛 + 1.
• Time Loop Ends
The parallel implementation of PRIMo leads to balanced compute
loads and message sizes when all cells across the domain domain are
wetted (h > 𝛿 w ), and unbalanced loads when there are partially wetted
cells and/or cells that are masked and saved as nodata. When cells are
not wetted (dry), PRIMo will sweep over adjacent edges to compute
fluxes and update the solution with (presumably) zero-valued fluxes.
Hence, dry cells also demand computations but the number of com-
putations is reduced because the approximate Riemann solver includes
logical checks that return zero-valued fluxes when two neighboring cells
are dry. Additionally, we note that after p and q are computed following
Eq. (16), u and v are computed as p/h and q/h, respectively, in all cells
with h > 𝛿 h , where 𝛿 h is typically set to 0.001 m although smaller values
can be used. Hence, PRIMo will exchange fluid between cells whenever
h > 𝛿 w , but u and v are computed only when h > 𝛿 h . Otherwise, u and v
are set to zero.
2.8. Summary of model features
As described above, PRIMo is developed with several features to
balance accuracy and computational effort:
1. The shallow-water equations are solved on an upscaled grid (U grid)
with cell size Δ𝑈 = 𝛼Δ𝑧 , where 𝛼 represents the upscale factor.
2. Fluxes are integrated along each U grid cell edge by summing over
a sample of 𝑁𝑠f lux ≤ 𝛼 elevation data points.
3. Storage is tabulated versus water level based on the 𝛼 2 elevation data
points within each U grid cell, and interpolation is performed using
a limited number of basis points 𝑁𝑠vol ≤ 𝛼 2 .
4. A friction parameter Σf is tabulated versus water level based on the
𝛼 2 elevation data points within each U grid cell, and interpolation is
performed using a limited number of basis points 𝑁𝑠fric ≤ 𝛼 2 .
5. Slopes in the water surface 𝛿 x 𝜂 and 𝛿 y 𝜂 are computed for each U
grid cell for more accurate estimation of fluxes and more accurate
downscaling of depth to the z grid.
6. Parallel execution is enabled for either shared-memory or
distributed-memory compute clusters with MPI directives.
The remainder of the paper presents test cases to verify the model
with analytical solutions and an observed dam-break flood, test its paral-
lel scaling and benchmark its performance in a hypothetical whole-city
scale application. Based on these results, strengths and weaknesses of
the method are presented in the Discussion section along with promis-
ing directions for future research.
3. Applications
3.1. Dry-bed dam-break problem
We consider a 1 km long and 100 m wide channel that is horizon-
tal and frictionless, and create a z grid at 1 m resolution for input into
PRIMo. Initially water is a depth of 3 m for x < 500 m and motionless,
85
Advances in Water Resources 126 (2019) 79–95
and dry elsewhere. The solution is integrated for 40 s and saved every
10 s with a velocity calculation tolerance 𝛿ℎ = 1 × 10−6 m for improved
accuracy at the wetting front. Fig. 3 compares predictions (red) with the
exact solutions (gray) under four different model configurations: (A) an
upscale factor 𝛼 = 1 (no upscaling) without 𝜂 slopes (i.e., 𝛿𝑥 𝜂 = 0), (B)
an upscale factor 𝛼 = 1 with 𝜂 slopes (i.e., 𝛿 x 𝜂 ≠ 0), (C) an upscale factor
𝛼 = 5 without 𝜂 slopes, and (D) an upscale factor 𝛼 = 5 with 𝜂 slopes.
These results show that PRIMo performs similar to other first-order ac-
curate Godunov-type schemes when 𝛼 = 1 and 𝛿𝑥 𝜂 = 0 (Fig. 3A). That is,
the solution remains monotone, there is a small diffusive error in depth
at the leading edge of the rarefaction wave, there is a phase error in the
leading edge of the wetting front, and there is diffusion of the velocity
profile at the leading edge of the wetting front. When 𝛼 = 1 and 𝜂 slopes
are computed (Fig. 3B), there is less numerical diffusion as expected
and the phase error of the wetting front increases slightly. Fig. 3C and D
show that the effect of upscaling is similar to the effect of grid coarsen-
ing. In this case, the effective grid resolution is 5 m (instead of 1 m) and
there is a noticeable increase in numerical diffusion in the depth profile
at the leading edge of the rarefaction wave and in the velocity profile at
the leading edge of the wetting front. However, increasing the upscale
factor does not substantially change the phase error of the wetting front.
3.2. Parabolic floodplain with meandering channel
We now consider an idealized parabolic floodplain with a mean-
dering channel to test the model for compound channel cross-sections.
As shown in Fig. 4, the spatial domain is rectangular with a length of
𝐿𝑥 = 6 km and a width of 𝐿𝑤 = 1 km, and we assume Δ𝑧 = 1 m. The
meandering channel has a rectangular cross-section with a width of
𝑊 = 20 m measured perpendicular to the channel centerline. Addition-
ally, the meander takes on a sinusoidal form with a wavelength of 1 km
and an amplitude of 100 m. Based on a coordinate system with the ori-
gin placed in the lower left corner of the domain, DTM elevations (in
meters) are computed as follows,
𝑧f loodplain (𝑥, 𝑦) = 1 × 10−4 𝑥 + 1 × 10−5 (𝑦 − 3, 000)2 (41)
𝑧channel (𝑥, 𝑦) = 1 × 10−4 𝑥 − 2 (42)
except for the first and last 500 m of the domain where 𝑧 = −2 m as
shown in Fig. 4. Note that the channel is 2 m deep at 𝑦 = 3000 m and
slightly deeper at the top and bottom of the meanders where the flood-
plain elevation is slightly higher due to the parabolic cross-sectional
profile.
The channel and floodplain are dry at 𝑡 = 0, all boundaries are
treated as solid walls, and flooding is initiated by a constant inflow of
𝑄 = 1000 m3 /s specified as a point source at 𝑥 = 5800 m and 𝑦 = 500 m.
This leads to a flood front moving right to left over a dry bed, first into
the channel and then over the floodplain. The solution is integrated for
2 hr, and flow resistance is modeled with 𝑛𝑚 = 0.03 m−1∕3 s. In each case,
the sampling for fluxes, volume and friction was given by 𝑁𝑠f lux = 𝛼∕2,
𝑁𝑠vol = 2𝛼, and 𝑁𝑠friction = 2𝛼. Fig. 5 shows contours of flood depth at
𝑡 = 100 min without 𝜂 slopes (left) and with 𝜂 slopes (right) for 𝛼 = 200,
100, 50 and 10 (top to bottom). First, the model is shown to route flows
through channels for all values of 𝛼, including cases where ΔU much
larger than the channel width (i.e., 𝛼 = 200 corresponds to Δ𝑈 = 200 m).
Fig. 5 also reveals sensitivities in the propagation of the flood front to
upscaling, and the use of 𝜂 slopes, that are similar to dam-break test
case considered previously. That is, simulations without 𝜂 slopes (la-
beled 𝛿𝜂 = 0 in Fig. 5) exhibit slightly faster flood front movement than
with 𝜂 slopes (labeled 𝛿𝜂 ≠ 0 in Fig. 5). Differences in the position of the
flood front and the depth of flooding due to upscaling appear minimal
when 𝜂 slopes are computed (𝛿𝜂 ≠ 0). Fig. 5 shows that the leading edge
of the flood front extends close to 𝑥 = 1000 m along the flood plain for
the four different values of 𝛼, and the leading edge of the flood front in
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 3. PRIMo predictions of depth and velocity for the dry-bed dam-break problem (ℎ𝐿 = 3 m) at 10 s intervals: (A) 1 m resolution without 𝜂 slopes, (B) 1 m
resolution with 𝜂 slopes, (C) 5 m resolution (𝛼 = 5) without 𝜂 slopes, and (D) 5 m resolution (𝛼 = 5) with 𝜂 slopes. (For interpretation of the references to color in the
text, the reader is referred to the web version of this article.)

Fig. 4. Idealized floodplain topography with a meandering channel. Topography is resolved at 1 m resolution, the floodplain is parabolic in cross-section, the channel
is rectangular in cross-section and 20 m wide, and the x-direction slope of both the channel and floodplain is 1 × 10−4 .

the channel extends very close to the channel outlet or slightly past it.
On the other hand, when 𝜂 slopes are not computed (𝛿𝜂 = 0), there is
a substantial differences in the progression of the flood front between
𝛼 = 10 (flood front extending roughly to 𝑥 = 1000 m) and 𝛼 = 200 (flood
front extending past 𝑥 = 500 m). Hence, flood simulations using 𝜂 slopes
appear to reduce sensitivity of the flood wave progression to the upscale
factor.
A weakness of upscaling methods that has been reported in previ-
ous studies is the potential for granularity in downscaled flood depths
(Hénonin et al., 2015), and the appearance of granularity is prominent
in Fig. 5 when 𝜂 slopes are not computed and 𝛼 = 100 and 200. How-
ever, granularity does not appear in Fig. 5 when 𝜂 slopes are computed
and used to compute the downscaled flood depth. Hence, use of the 𝜂
slopes offers an advantage with respect to more accurate downscaling

86
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 5. PRIMo predictions of downscaled flood depth at 𝑡 = 100 min following input of 𝑄 = 1000 m3 /s at 𝑥 = 5800 m and 𝑦 = 500 m. Predictions are shown without
𝜂 slopes (𝛿𝜂 = 0, left side) and with 𝜂 slopes (𝛿𝜂 ≠ 0, right side) and using 𝛼 = 200, 100, 50 and 10 (top to bottom).

of the flood depth. In the remaining tests, all simulations involve use he Table 1
use of 𝜂 slopes for solution updates and downscaling. Baldwin hills test case wall clock times and
flood extent accuracy metrics.

3.3. Baldwin hills dam-break test case 𝛼 twall (min) FA % FO % FU %

2 1090.8 71.8 7.7 20.5

Model accuracy in a practical test case is now considered by applying
PRIMo to simulate urban flooding from the 1963 failure of the Baldwin
Hills dam. Previous studies have shown that flood extent can be pre-
dicted with over 70% accuracy in this test case (Gallegos et al., 2009;
Schubert and Sanders, 2012). Hence, PRIMo is applied here using sev-
eral sources of data previously reported by Gallegos et al. (2009): (1) a
1.5 m resolution DTM (8.5 cm vertical RMSE), (2) the location, height,
and length of curb inlets to storm drains, (3) a spatially distributed
resistance parameter distribution based on land cover including vege-
tated open space (𝑛𝑚 = 0.05), concrete surfaces (𝑛𝑚 = 0.013), reservoir
(𝑛𝑚 = 0.013), roads (𝑛𝑚 = 0.014), Ballona Creek (𝑛𝑚 = 0.016), and build-
ings (𝑛𝑚 = 0.3), (4) breach geometry data, (5) flood extent data (for val-
idation), (6) as-built reservoir geometry data, and (7) an estimate of the
water level at the time of failure (𝜂 = 141 m NAVD88) needed for the ini-
tial condition. The effect of structures on flooding dynamics is treated
with the building resistance method (Schubert and Sanders, 2012), and
breaching (which took place over 20 min) was assumed to occur instan-
taneously. Fig. 6 shows the site topography, flood extent data, and storm
drain locations.
Modeling of storm drain interception of overland flow by curb inlets
is based on either an orifice or a weir equation (Gallegos et al., 2009).
Given a curb inlet with a length L and height ho , and a local depth
prediction d, the interception rate is computed as,
√ likely within the accuracy of the measured flood extent data. Hence, the
𝑄 = 𝑐𝐷 𝐿 min(𝑑 , ℎ𝑜 ) 𝜅𝑔𝑑 (43)
where cD is a discharge coefficient, 𝜅 = 1 corresponds to a weir approx-
imation for d < ho , and 𝜅 = 2 corresponds to an orifice approximation
5 82.2 70.8 7.9 21.3
10 19.2 68.9 9.7 21.5
20 5.1 60.4 15.4 24.2
that is used otherwise. Here, 𝑐𝐷 = 0.5 was specified to match a previous
calibration by Gallegos et al. (2009).
PRIMo was implemented in parallel using 16 processors on the UCI
HPC cluster, which consists of 64-core compute nodes with 2.33 GHz
AMD processors and 512 GB of RAM. Fig. 7 shows the accuracy of flood
extent predictions using 𝛼 = 2, 5, 10 and 20 which corresponds to Δ𝑈 =
3, 7.5, 15 and 30 m. Here, correct predictions of flooding (green), areas
of overprediction (blue), and areas of under prediction (red) are shown.
Additionally, Table 1 presents wall clock execution times and metrics
of flood extent accuracy including an agreement metric FA , and over-
prediction metric FO and an underprediction metric FU (Schubert and
Sanders, 2012). Perfect accuracy corresponds to 𝐹𝐴 = 100%, 𝐹𝑂 = 0%
and 𝐹𝑈 = 0%.
These results show that PRIMo simulates flood extent with a high
level of accuracy (roughly 70% for 𝛼 = 2, 5 and 10) without model-
specific calibration. That is, parameter values and taken from previous
studies. These results also show that flood extent accuracy differs by
only 1% between 𝛼 = 2 and 5 and by 3% between 𝛼 = 2 and 10, which is
potential exists for significant upscaling without significant loss of accu-
racy. However, there are limits: 𝛼 = 20 leads to substantially increased
overprediction and a reduction of flood extent accuracy by about 11%

87
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 6. Hillshade map of Baldwin Hills DTM with observed flood extent an location of curb inlets to the storm drain system. Baldwin Hills reservoir appears as a
rectangular bowl near the southern boundary.

Fig. 7. Classification of PRIMo flood inundation predictions for the Baldwin Hills dam-break test case including correct predictions of inundation (green), over-
prediction (blue) and underprediction (red). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this
article.)

88
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 8. A 4 km2 square block of 1 m resolution topographic data from

the Platte River in central Nebraska, and the domain decomposition
used for nproc = 16.

Fig. 9. Speedup of PRIMo versus number of processes for: (A) a 4 km2 test problem on the UCI HPC cluster (used in previous test problem) and the Cheyenne
cluster at the NCAR-Wyoming Supercomputing Center, and (B) a 1,024 km2 test problem on the Cheyenne cluster. The UCI cluster exhibits less than ideal speedup
when run in exclusive mode because sequential execution is considerably faster, presumably due to internal hardware optimizations that leverage ideal processing
resources, and the Cheyenne cluster may exhibit better than ideal scaling as the number of processes increases. Speedup is normalized by run times for nproc =
16 because memory demands prevented sequential execution.

compared to the case with 𝛼 = 2. Finally, Table 1 shows that increasing
𝛼 significantly reduces wall clock execution times, as expected. In the re-
maining test cases, attention turns to the parallel scaling of PRIMo and
the potential for fast simulation of flooding over large spatial domains.
3.4. Load balanced parallel performance testing
Domain decomposition into tiles with equal size leads to perfect
load balancing across processes if the domain is fully wetted, and thus
the number of operations on each processor per time step is equal.
Fig. 8 shows a square domain of 1 m resolution topographic data span-
ning 4 km2 of the Platte River in central Nebraska, data previously used
for hydrodynamic river modeling (Schubert et al., 2015). Hence, the
raster is 2000 × 2000 cells which is amenable to subdivision several
times into grid sizes that are multiples of 10 and 20 and supportive of
upscaling with relatively large values of 𝛼. The test problem involves
an initial condition corresponding to a constant water level 𝜂 = 712 m
(NAVD 88), which inundates the highest topography in the domain by
only several cm, and a fluid velocity of zero. Wall boundary conditions
are specified on all four sides, and a point source 𝑄 = 300 m3 /s is spec-
ified in the main channel near the left boundary of the domain which
represents the upstream end of this river reach. Flow resistance is not
important but is modeled using a spatially uniform Manning coefficient,
𝑛𝑚 = 0.03 m−1∕3 s. The simulation period for this test problem is 6000 s.
To test the parallel performance of PRIMo, the domain was divided
twice, four times, and eight times in each Cardinal direction leading to
grids designed for nproc = 1, 4, 16 and 64 processes and upscaling us-
ing 𝛼 = 10. As an example, Fig. 8 shows the domain decomposition for
nproc = 16 and we note that the numbering of processes begins with
zero, consistent with the standard MPI convention. PRIMo sampling
parameters used in this test case were 𝑁𝑠f lux = 5, 𝑁𝑠vol = 11, 𝑁𝑠fric = 11.
Testing was performed on the UCI HPC cluster and the Cheyenne
cluster at the NCAR-Wyoming Supercomputing Center which consisted
of 36-core compute nodes with 2.3 GHz Intel Xeon processors and 64
GB of RAM. Both systems use InfiniBand high-speed interconnect for
communication between nodes. Fig. 9 shows that parallel performance
closely tracks the ideal speedup profile on the Cheyenne cluster and on

89
B.F. Sanders and J.E. Schubert
Fig. 10. Wall clock times for PRIMo to simulate 1 h of flooding at 1 m reso-
lution over 1,024 km2 using nproc = 1,024 scale as a power law function of
the upscale factor, i.e., 𝑡 = 𝑡𝑜 𝛼 −𝑘 .
the UCI cluster when PRIMo is run in a shared execution mode whereby
other jobs run concurrently on the node. However, PRIMo does not
demonstrate ideal scaling on the UCI cluster when jobs have exclusive
access to the node. This is attributed relatively faster execution of the
test involving nproc = 1 which suggests that the hardware is perform-
ing internal optimizations to take advantage of idle computing power.
Similar behavior has been observed in previous studies of parallel per-
formance (e.g. Sanders et al., 2010), and run times using nproc = 64
were equivalent using the UCI cluster in shared and exclusive modes.
We note that the case involving nproc = 64 required use of two nodes
on the Cheyenne cluster, and communication between nodes, yet the
parallel performance exhibits no adverse consequences. Both clusters
could support larger jobs requiring significantly more cores, but this
2000 × 2000 cell test problem doesn’t justify use of additional cores.
Using 64 cores, the 6000 s simulation period requires only about 10 s
of wall clock time on the Cheyenne cluster, which is significantly faster
than the UCI cluster due to a compiler that is optimized for the hard-
ware.
A more demanding test problem requiring numerous compute nodes
is created by randomly generating topographic heights at 1 m resolu-
tion over a spatial extent of 1,024 km2 , assuming a uniform distribu-
tion between 0 and 1 m. This corresponds to a topographic grid of
32,000 × 32,000=1.024 × 109 cells which conveniently supports do-
main decomposition for 16 cores with a 4 × 4 matrix of 8,000 × 8,000
cell grids, 64 cores with a 8 × 8 matrix of 4,000 × 4,000 cell grids, 256
cores with a 16 × 16 matrix of 2,000 × 2,000 cell grids, and 1024 cores
with a 32 × 32 matrix of 1,000 × 1,000 cell grids. The test problem
is defined by an initial water elevation of 3 m, wall boundary condi-
tions on all four sides, a rainfall rate of 1 cm/hr, a spatially uniform
𝑛𝑚 = 0.03 m−1∕3 s, and a simulation period of 1 hr. PRIMo is run on
the Cheyenne cluster using nproc = 16, 64, 256 and 1024 processes
and 𝛼 = 10. PRIMo sampling parameters used in this test case were
𝑁𝑠f lux = 5, 𝑁𝑠vol = 11, 𝑁𝑠fric = 11. Fig. 9 shows that speedup is better than
ideal when normalized by the execution time for 16 cores. The reason
for better than ideal performance is not immediately clear, but it could
be due to low memory demands per node or to more efficient communi-
cation realized by smaller-sized arrays being passed between processes
as the number of processes increase.
The 1 hr simulation completed in 98 s using 1024 cores. How does
the choice of the upscale factor affect run times? Fig. 10 shows that
90
Advances in Water Resources 126 (2019) 79–95
wall clock times using 𝛼 = 4, 10 and 20 closely follow a power law func-
tion of the upscale factor of the form 𝑡 = 𝑡𝑜 𝛼 −𝑘 (ideal performance).
Hence, these results demonstrate that PRIMo achieves ideal (power
law) run time reductions from two separate mechanisms: parallel pro-
cessing and upscaling, and the combination of these mechanisms allows
for an hourly simulation of flooding at 1 m resolution over 103 km2 in a
matter of seconds, a level of performance well suited to nowcasting and
forecasting applications.
3.5. Los Angeles region extreme flooding test
As a final test, we consider a simulation of flooding across Los An-
geles, the second largest city in the USA, to benchmark wall clock times
and test the computational feasibility of PRIMo for time-sensitive ap-
plications such as real-time forecasting and uncertainty quantification
with Monte Carlo simulation. To achieve a reasonable calibration of a
flood hazard model for an urban area, many factors must be carefully
considered including the representation of flow pathways in the DTM,
subsurface drainage through pipes, storage in reservoirs, flow resistance,
the variability in precipitation, and surface/subsurface interactions such
as infiltration (e.g., Gallegos et al., 2009; Gallien et al., 2011). Since the
objective here is only to benchmark wall clock times and validate the
shallow-water routing of pluvial flooding, and calibration requires ob-
servations of observed flooding of extreme events that are not easily
obtained (for many reasons), no attempt is made to calibrate the model.
The Los Angeles model domain spans the portions of Los Angeles
County that drain to the south and west from the San Gabriel Moun-
tains. The model was developed from a 10 ft resolution DTM for the
County of Los Angeles (USGS, 2006) which was prepared for public ac-
cess by downsampling a 5 ft resolution DTM with 0.91 ft vertical ac-
curacy at the 95% confidence level. The spatial domain was trimmed
along the north and east based on the limits of the Los Angeles River
watershed, spanning the areas where the vast majority of the County’s
10 million people reside (U.S. Census Bureau, 2010). The trimmed do-
main was subsequently partitioned into 608 square tiles consisting of
103 × 103 raster grids, as shown in Fig. 11, a number that allowed for
parallel execution with 32 processes per node across 19 compute nodes
on Cheyenne. Hence, a total of 608 million z grid points are included in
the model, and the domain spans an area of 2181 mi2 or 5,652 km2 .
The hazard scenario that forms the basis of this test corresponds to
the coincidence of a record rainfall with record high tide, and likely has
a return period of at least 1000 years (Goodridge, 1994). A constant wa-
ter level boundary of 𝜂𝑏 = 8 ft (NAVD 88) is assigned along the coastal
boundary representative of the record high tide level, which roughly
corresponds to the maximum recorded water level in Los Angeles har-
bor (NOAA Gage #9410660), and a constant effective rainfall rate of
𝑃 = 1 in/hr is specified uniformly for a period of 𝑇 = 12 h over which
flooding is simulated. This rainfall rate roughly corresponds to historical
maximum conditions within Los Angeles County, where 26 in. of rain-
fall was once recorded over a 24 h period at a gage in the San Gabriel
Mountains, and on several occasions, more than 10 in. of rainfall have
been recorded over a 12 h period (Goodridge, 1994). The assumption
of spatially uniform rainfall at this rate is not consistent with obser-
vations, as rainfall is magnified in the San Gabriel Mountains by oro-
graphic effects. Nevertheless, it is a useful simplification for measuring
the run-time attributes of PRIMo and creating a scenario that can be
easily communicated, i.e., “1 in/hr rainfall for 12 h”. Four simulations
were completed using 𝛼 = 5, 10, 20 and 50 and other model parameters
shown in Table 2. PRIMo was executed on Cheyenne and wall clock
times are also reported in Table 2. This shows that wall clock times are
reduced from about 6 h using 𝛼 = 5 to about 15 s using 𝛼 = 50.
Flood depth at 𝑡 = 12 hrs using 𝛼 = 5 is shown in Fig. 12A, and points
to extensive inundation across the region as flood flows move south
from the San Gabriel Mountains towards Long Beach (cf. Fig. 11). In-
deed, the region between Los Angeles and Long Beach is an amalgam
of alluvial fans where the simulation shows flood water spreading out
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 11. Los Angeles spatial domain shown here with 103 × 103 tiles
of 10 ft resolution topographic data. There are 608 tiles containing
data, and PRIMo was configured for parallel execution with 608
processes (one tile per process). Ocean level was specified along the
coastal boundary (light blue), and a free-overfall (or dry) boundary
condition was used on all land boundaries. Results shown for areas la-
beled (AA) and (A)–(D). (For interpretation of the references to colour
in this figure legend, the reader is referred to the web version of this
article.)

Fig. 12. PRIMo simulation of (A) flood depth at 12 hrs using 𝛼 = 5 for the region indicated as (AA) in Fig. 11, and the difference in flood depth, Δd, at 12 h
between simulations using (B) 𝛼 = 10 and 5, (C) 𝛼 = 20 and 5, and (D) 𝛼 = 50 and 5. As 𝛼 is increased, predicted drainage patterns remain similar but the depth
behind obstructions such as freeways and dams is reduced and the depth near the southern coastline of the study area is locally increased.

91
B.F. Sanders and J.E. Schubert Advances in Water Resources 126 (2019) 79–95

Fig. 13. Simulation of flood inundation with PRIMo systematically captures many complex flooding processes (including compound flooding) such as: (A) storage
behind Sepulveda Dam, overtopping of its spillway, and the combine effects of fluvial and pluvial flooding downstream, (B) flow over a series of 17 drop structures
along the very steep San Gabriel River and storage behind Santa Fe Dam, (C) spreading of Los Angeles River floodwater and pluvial flooding in downtown Los Angeles
which is built on alluvial fan, and (D) the combined effect of fluvial, pluvial and coastal flooding around Long Beach and the Ports of Los Angeles and Long Beach.

Table 2 based on run times of 66 and 4.8 min, respectively, and the sensitivity
Los Angeles test case parameters and wall clock times. to the upscale factor.
i 𝜂b T twall More detail is found in Fig. 13, where flood depth in the regions
# nproc (in/hr) (ft) (hrs) 𝛼 𝑁𝑠f lux 𝑁𝑠vol 𝑁𝑠f ric (min) marked (A)-(D) in Fig. 11 is shown based on 𝛼 = 10. Fig. 13A shows
a reach of the Los Angeles river that flows East through a flood de-
1 608 1.0 8.0 12 5 3 6 6 380
2 608 1.0 8.0 12 10 5 11 11 66 tention basin, the Sepulveda Basin, and then through the community
3 608 1.0 8.0 12 20 5 11 11 4.8 of Sherman Oaks. PRIMo simulates overtopping of the Sepuveda Basin
4 608 1.0 8.0 12 50 5 11 11 0.25 spillway, which is associated with transcritical flow regimes, as well as
compound fluvial/pluvial flooding in Sherman Oaks due the combined
effects of Los Angeles River flows from the West and pluvial runoff from
the North. Fig. 13B shows where the San Gabriel river descends south
out of the San Gabriel Mountains, over 17 drop structures that dissipate
and pooling behind a major East-West highway (CA Route 91). As 𝛼 is energy, and into the Santa Fe Dam recreational area which is a major
increased, predicted drainage patterns remain similar but the depth be- flood control facility in the area. Modeling of flow over the drop struc-
hind obstructions such as freeways and dams is reduced and the depth tures leads to transcritical flows with hydraulic jumps that are easily
near the southern coastline of the study area is locally increased. The resolved by the model. Fig. 13C shows the combined effect of fluvial
mean absolute value of Δd was computed to be 0.17 ft (5.2 cm), 0.28 ft flooding and pluvial flooding in downtown Los Angeles. The area west
(8.5 cm) and 0.47 ft (14 cm) for 𝛼 = 10, 20 and 50 (compared to the of the Los Angeles River is an alluvial fan and the simulation shows sev-
𝛼 = 5 case). Reduced pooling behind obstructions is attributed to flow eral preferential pathways for floodwater within this highly urbanized
cell edges that no longer capture important blockage features such as area. Finally, Fig. 13D shows the combined effects of pluvial, fluvial
elevated roadways or embankments. This weakness of upscaling meth- and coastal flooding in the vicinity of Long Beach, where the Los An-
ods has previously been reported by Hodges (2015). Nevertheless, the geles River (East side of the Harbor complex) and Dominguez Channel
cases involving 𝛼 = 10 and 20 show potential for forecasting applications (West side of Harbor Complex) empty into the Pacific Ocean through

92
B.F. Sanders and J.E. Schubert
the Ports of Long Beach and Los Angeles, the busiest port in the USA
based on container shipments.
4. Discussion
Upscaling in shallow-water modeling of flood inundation is relatively
new. Presented here is a technique for retaining the precision of fine
scale topography, and the skill of approximate Riemann solvers to esti-
mate fluxes, in a relatively course solution update with a Godunov-type
finite volume scheme. The goal is to improve computational efficiency,
and resolve a wide range of flows, without overly sacrificing accuracy.
How fluxes are evaluated, and in particular, how topographic data are
used to inform fluxes, is arguably the most important consideration in
the design of an upscaling method, and the best strategy will depend
on many factors including the type of numerical solver used. For this
study, a distribution of topographic data from z grid cells along each U
grid edge were computed to support flux estimation, as shown in Eq.
(4), to enable implementation of a Godunov-type finite volume scheme.
Another approach used for an implicit finite volume scheme is to inte-
grate all of the z data within a staggered U grid cell (Volp et al., 2013),
and Hodges (2015) proposed processing z values to identify topographic
objects that control the spreading of flood water, and subsequently as-
signing z values to edges to preserve the functionality of the object. Oth-
ers have pursued adaptive meshing approaches that increase resolution
around key topographic features (Hou et al., 2018). Indeed, many op-
tions are possible and more research is needed to deepen understanding
of strengths and weaknesses and to develop improved methods.
Results presented here point to several strengths including the abil-
ity to resolve flooding over a wide range of topographic and flow con-
ditions, the ability to resolve flow in channels resolved on the z grid
but finer that the resolution of the U grid, the ability to improve accu-
racy by making 𝜂 piecewise linear instead of piecewise constant, and the
ability of the scheme to support ideal parallel scaling which allows the
method to be efficiently applied at any scale. Another major advantage
of this upscaling method is that a single parameter, 𝛼, can be changed to
best meet accuracy and run time requirements. That is, there is no need
for time consuming re-gridding to make a model run faster. The main
weakness of the method is that blockage features may not be correctly
resolved which leads to artificial spreading of flood water. Additionally,
the method does not overcome a well-known limitation of finite-volume
schemes whereby flow down steep slopes is modeled as a series of wa-
terfalls (Kim et al., 2012; Xia et al., 2017). In fact, use of piecewise linear
𝜂 changes the representation of this waterfall dynamic, and additional
research is needed to develop improved representation of flow down
steep slopes in conjunction with upscaling methods. Moreover, while
the scheme conserves mass to numerical precision based on the direct
discretization of conservative equations and the consistent updating of
the discrete solution based on globally computed flux values, the scheme
does not guarantee positivity. The prediction of negative depth is pos-
sible if the volume of water predicted to leave a cell over one time step
exceeds volume stored in the cell (Begnudelli and Sanders, 2006; Brad-
ford and Sanders, 2002; Brufau et al., 2004). In the test cases considered,
negative depth predictions were minimal and presented no noticeable
impacts on accuracy or stability, but strategies to address this limitation
in conjunction with upscaling methods could be considered the future
whereby tradeoffs between accuracy, stability and computational effort
in parallel execution are examined.
It is unclear whether PRIMo would have scaled better or run faster
had it been implemented on a GPU or a cluster of GPUs instead of a
cluster of multi-core compute nodes. Several studies point to fast flood
inundation model performance on GPUs (Brodtkorb et al., 2012; Castro
et al., 2011; Neal et al., 2010; Vacondio et al., 2014), and as the size of
applications increases, memory and communication demands increase
and ultimately this controls the parallel scaling of the model. Additional
research is needed to address this possibility and improve understanding
93
Advances in Water Resources 126 (2019) 79–95
of the strengths and weaknesses of GPUs and multi-core compute nodes
to guide development of future models.
A broader issue raised by this research is the potential to de-
liver timely flood hazard information at fine resolution over whole-
city scales. World population has more than doubled since 1960, from
around 3 billion people to around 7 billion today, and the exposure
and vulnerability of communities to flooding has been magnified with
over half of the global population concentrated in cities that are lo-
cated along coastlines or major rivers (Hinkel et al., 2014; Wolsko and
Marino, 2016). This motivates a pressing need for flood risk informa-
tion that is responsive to the wide web of decision-making causing
the increase in exposure and vulnerability, or actionable information
(Morss et al., 2005; Spiekermann et al., 2015). Fine-scale visualiza-
tions of flooding have tremendous power for communicating flood haz-
ards in intuitive ways that anyone can understand (Lane et al., 2011;
Luke et al., 2018), and collaboration with stakeholders on visualiza-
tions can meet site-specific decision-making needs, in a context-sensitive
way, for all phases of the disaster management cycle including plan-
ning, preparedness, early warning, emergency response and recovery
(Dawson et al., 2011; Evers et al., 2012; Luke et al., 2018; Pasche et al.,
2009; Steinführer, 2009).
What is demonstrated here is the computational feasibility of a fast-
response, fine-resolution, hydrodynamic flood hazard simulation over a
large metropolitan area. Transitioning from a feasibility to a reality intro-
duces several more research challenges deserving attention including:
(a) the ability to accurately force flood inundation models with rainfall,
streamflow and ocean height data, (b) capturing all important hydrolog-
ical processes that contribute to the distribution of flood water (e.g., ur-
ban drainage infrastructure) and (c) managing the enormous volumes of
data associated with time-dependent, fine-resolution spatial fields over
large spatial extents. Lessons learned from fine-resolution flood hazard
modeling over smaller spatial extents are relevant here. For example,
DTMs can be conditioned to improve the representation of drainage
pathways and representation of flow barriers such a levees, but models
have generally relied on manual processing or at best semi-automated
methods (e.g. Gallegos et al., 2009; Gallien et al., 2011; Schubert and
Sanders, 2012) which is tedious over the scale of entire metropolitan
areas like Los Angeles. Nevertheless, DTM processing is clearly needed
in this case based on the amount of ponding predicted to occur behind
highways, where flood extent is surely overpredicted given that highway
underpasses are not resolved by the model. Poor channel bathymetry ac-
curacy in the DTM is also likely to be a contributor to the overprediction
of flood depths and flood extent in coastal areas, and motivates the need
for coastal DTMs that merge the best available topographic and bathy-
metric data, such as those developed by NOAA (NOAA National Center
for Environmental Information, 2018).
5. Conclusions
A Godunov-type finite volume scheme for solving the shallow-water
equations, PRIMo, is implemented with an innovative sub-grid model
that enables flow computations at scale ΔU that is a multiple 𝛼 of the
DTM resolution, Δz . The method reduces the number of flux computa-
tions by at least a factor of 𝛼 per time step, and the number of solution
updates by a factor of 𝛼 2 per time step, compared with a flow model
that performs flow computations at a resolution of Δz . Additionally, the
maximum allowable time step for stability is increased by a factor of 𝛼
compared to a model run at a resolution of Δz , and thus fluxes and solu-
tion updates are reduced by a factor of 𝛼 2 and 𝛼 3 overall, respectively.
Execution time is further reduced by a sampling technique that limits
the number of times the approximate Riemann solver is used each time
step to compute fluxes. Additionally, piecewise linear treatment of the
free surface elevation 𝜂 is combined with upscaling to reduce numerical
dissipation and improve the accuracy and visualization of downscaled
flood depth.
B.F. Sanders and J.E. Schubert
PRIMo is shown to simulate subcritical, supercritical and transcrit-
ical flows which is expected of a Godunov-type flow solver that uses an
approximate Riemann solver suited to all classes of shallow-water flow.
PRIMo is also shown to propagate flow through channels that are re-
solved on the z grid but finer than the U grid. Additionally, piecewise
linear treatment of the water surface 𝜂 is shown to improve accuracy of
downscaled flood depths over piecewise linear models (𝛿𝜂 = 0). Appli-
cation of PRIMo to the Baldwin Hills dam-break flood showed the po-
tential to upscale up to 𝛼 = 10 in an urban area (using Δ𝑧 = 1.5 m) with
negligible impact on flood extent accuracy, while further increases in 𝛼
lead to overprediction of flood extent. The Los Angeles test case showed
practical benefits such as the ability to simulate flow over spillways and
weirs, flow down steep slopes, and backwater effects such as those that
occur near the coastline. Additionally, the Los Angeles test case showed
the potential to simulate compound flooding involving pluvial, fluvial
and coastal flood hazards.
PRIMo is shown to achieve ideal parallel scaling with up to 1024
processes (the largest problem tested). The combined effects of upscaled
flow computations and parallel execution enable metric resolution sim-
ulations over thousands of km2 and hourly time scales in a matter of
seconds to minutes, depending on flow conditions (depth and velocity)
which control the maximum allowable time step. This capacity is well
suited to supporting time-sensitive applications such as nowcasting and
forecasting, and implies that hydrodynamic simulations can be run in
real time to assist with several phases of disaster management including
early warning, emergency response and recovery. Fast execution times
also increase the feasibility of Monte Carlo simulation to characterize
uncertainties for planning, preparedness and mitigation aspects of dis-
aster management.
Acknowledgements
The upscaling method adopted by PRIMo was conceived by B.
Sanders and on several occasions, was discussed with L. Begnudelli
who was involved in the development of a separate model that uses
a similar upscaling technique. The authors acknowledge L. Begnudelli
for valuable dialogue and correspondence which contributed to this
work. The authors also acknowledge B.M. Ginting for valuable dia-
logue and correspondence regarding parallelization strategies, A. Luke
for valuable dialogue regarding the practical value of fast simulations,
and J. Vrugt for suggestions to improve the paper. This research was
made possible by a grant from the National Science Foundation (DMS-
1331611). We also like acknowledge the use of the UCI High Perfor-
mance Computing System (http://hpc.oit.uci.edu) and computational
resources (doi:10.5065/D6RX99HX) at the NCAR-Wyoming Supercom-
puting Center provided by the National Science Foundation and the
State of Wyoming, and supported by NCAR’s Computational and Infor-
mation Systems Laboratory. Finally, we thank the anonymous reviewers
for comments and questions that helped us to improve the paper.
References
Bates, P.D., 2012. Integrating remote sensing data with flood inundation models: how far
have we got? Hydrol. Process. 26 (16), 2515–2521.
Bates, P.D., Horritt, M.S., Fewtrell, T.J., 2010. A simple inertial formulation of the shallow
water equations for efficient two-dimensional flood inundation modelling. J. Hydrol.
387 (1–2), 33–45.
Begnudelli, L., 2016. 2D Shallow Water Model with Cells-Aggregation Technique for Large
Scale Flood Modelling Using High Resolution Terrain Models. AGU Fall Meeting Ab-
stracts.
Begnudelli, L., Sanders, B.F., 2006. Unstructured grid finite-volume algorithm for shal-
low-water flow and scalar transport with wetting and drying. J. Hydraul. Eng. 132
(4), 371–384.
Bilskie, M.V., Coggin, D., Hagen, S.C., Medeiros, S.C., 2015. Terrain-driven unstructured
mesh development through semi-automatic vertical feature extraction. Adv. Water
Resour. 86, 102–118.
Bradford, S.F., Sanders, B.F., 2002. Finite-volume model for shallow-water flooding of
arbitrary topography. J. Hydraul. Eng. 128 (3), 289–298.
Brodtkorb, A.R., Sætra, M.L., Altinakar, M., 2012. Efficient shallow water simulations on
GPUs: implementation, visualization, verification, and validation. Comput. Fluids 55,
1–12.
94
Advances in Water Resources 126 (2019) 79–95
Brufau, P., García-Navarro, P., Vázquez-Cendón, M., 2004. Zero mass error using unsteady
wetting–drying conditions in shallow flows over dry irregular topography. Int. J. Nu-
mer. Methods Fluids 45 (10), 1047–1082.
Castro, M.J., Ortega, S., De la Asuncion, M., Mantas, J.M., Gallardo, J.M., 2011. Gpu
computing for shallow water flow simulation based on finite volume schemes. Acad.
Sci. Comptes Rendus. Mec. 339 (2–3), 165–184.
Casulli, V., Stelling, G.S., 2011. Semi-implicit subgrid modelling of three-dimensional
free-surface flows. Int. J. Numer. Methods Fluids 67 (4), 441–449.
Cea, L., Bladé, E., 2015. A simple and efficient unstructured finite volume scheme for
solving the shallow water equations in overland flow applications. Water Resour. Res.
51 (7), 5464–5486.
CRED, U., 2015. The Human Cost of Weather-Related Disasters, 1995–2015. United Na-
tions, Geneva.
Dawson, R.J., Ball, T., Werritty, J., Werritty, A., Hall, J.W., Roche, N., 2011. Assessing the
effectiveness of non-structural flood management measures in the thames estuary un-
der conditions of socio-economic and environmental change. Global Environ. Change
21 (2), 628–646.
Defina, A., 2000. Two-dimensional shallow flow equations for partially dry areas. Water
Resour. Res. 36 (11), 3251–3264.
Di Baldassarre, G., Montanari, A., Lins, H., Koutsoyiannis, D., Brandimarte, L., Blöschl, G.,
2010. Flood fatalities in africa: from diagnosis to mitigation. Geophys. Res. Lett. 37
(22).
Djordjević, S., Butler, D., Gourbesville, P., Mark, O., Pasche, E., 2011. New policies to deal
with climate change and other drivers impacting on resilience to flooding in urban
areas: the CORFU approach. Environ. Sci. Policy 14 (7), 864–873.
Evers, M., Jonoski, A., Maksimovič, Č., Lange, L., Ochoa Rodriguez, S., Teklesadik, A.,
Cortes Arevalo, J., Almoradie, A., Eduardo Simões, N., Wang, L., et al., 2012. Collab-
orative modelling for active involvement of stakeholders in urban flood risk manage-
ment. Nat. Hazards Earth Syst. Sci. 12 (9), 2821–2842.
Few, R., Ahern, M., Matthies, F., Kovats, S., 2004. Floods, Health and Climate Change: A
Strategic Review.
Gallegos, H.A., Schubert, J.E., Sanders, B.F., 2009. Two-dimensional, high-resolution mod-
eling of urban dam-break flooding: a case study of Baldwin Hills, California. Adv.
Water Resour. 32 (8), 1323–1335.
Gallien, T., Schubert, J., Sanders, B., 2011. Predicting tidal flooding of urbanized embay-
ments: a modeling framework and data requirements. Coastal Eng. 58 (6), 567–577.
Gettleman, J., 2017. More than 1,000 died in south asia floods this summer. New York
Times 29.
Goodridge, J.D., 1994. A study of 1000 year storms in california. In: Predicting Heavy
Rainfall Events in California: A Symposium to Share Weather Pattern Knowledge,
pp. 3–72.
Guinot, V., Sanders, B.F., Schubert, J.E., 2017. Dual integral porosity shallow water model
for urban flood modelling. Adv. Water Resour. 103, 16–31.
Hagen, S., Westerink, J., Kolar, R., Horstmann, O., 2001. Two-dimensional, unstructured
mesh generation for tidal models. Int. J. Numer. Methods Fluids 35 (6), 669–686.
Hallegatte, S., Green, C., Nicholls, R.J., Corfee-Morlot, J., 2013. Future flood losses in
major coastal cities. Nat. Clim. Chang 3 (9), 802.
Heimhuber, V., Hannemann, J.-C., Rieger, W., 2015. Flood risk management in remote
and impoverished areas–a case study of Onaville, Haiti. Water 7 (7), 3832–3860.
Hénonin, J., Hongtao, M., Zheng-Yu, Y., Hartnack, J., Havnø, K., Gourbesville, P.,
Mark, O., 2015. Citywide multi-grid urban flood modelling: the july 2012 flood in
beijing. Urban Water J. 12 (1), 52–66.
Hinkel, J., Lincke, D., Vafeidis, A.T., Perrette, M., Nicholls, R.J., Tol, R.S., Marzeion, B.,
Fettweis, X., Ionescu, C., Levermann, A., 2014. Coastal flood damage and adaptation
costs under 21st century sea-level rise. Proc. Natl. Acad. Sci. 111 (9), 3292–3297.
Hodges, B., 2015. Representing hydrodynamically important blocking features in coastal
or riverine lidar topography. Nat. Hazards Earth Syst. Sci. 15 (5), 1011–1023.
Hou, J., Wang, R., Liang, Q., Li, Z., Huang, M.S., Hinkelmann, R., 2018. Efficient surface
water flow simulation on static cartesian grid with local refinement according to key
topographic features. Comput. Fluids 176, 117–134.
Jongman, B., Ward, P.J., Aerts, J.C., 2012. Global exposure to river and coastal flooding:
long term trends and changes. Global Environ. Change 22 (4), 823–835.
Kim, J., Warnock, A., Ivanov, V.Y., Katopodes, N.D., 2012. Coupled modeling of hydro-
logic and hydrodynamic processes including overland and channel flow. Adv. Water
Resour. 37, 104–126.
Lane, S.N., Odoni, N., Landström, C., Whatmore, S.J., Ward, N., Bradley, S., 2011. Doing
flood risk science differently: an experiment in radical scientific method. Trans. Inst.
Br. Geogr. 36 (1), 15–36.
Luke, A., Sanders, B.F., Goodrich, K.A., Feldman, D.L., Boudreau, D., Eguiarte, A., Ser-
rano, K., Reyes, A., Schubert, J.E., AghaKouchak, A., Basolo, V., Matthew, R.A.,
2018. Going beyond the flood insurance rate map: insights from flood haz-
ard map co-production. Nat. Hazards Earth Syst. Sci. 18 (4), 1097–1120.
https://doi.org/10.5194/nhess-18-1097-2018.
Mackay, E., Wilkinson, M., Macleod, C.J., Beven, K., Percy, B.J., Macklin, M., Quinn, P.F.,
Stutter, M., Haygarth, P.M., 2015. Digital catchment observatories: a platform for
engagement and knowledge exchange between catchment scientists, policy makers,
and local communities. Water Resour. Res. 51 (6), 4815–4822.
Maskrey, S.A., Mount, N.J., Thorne, C.R., Dryden, I., 2016. Participatory modelling for
stakeholder involvement in the development of flood risk management intervention
options. Environ. Modell. Softw. 82, 275–294.
Meyer, V., Kuhlicke, C., Luther, J., Fuchs, S., Priest, S., Dorner, W., Serrhini, K., Pardoe, J.,
McCarthy, S., Seidel, J., et al., 2012. Recommendations for the user-specific enhance-
ment of flood maps. Nat. Hazards Earth Syst. Sci. 12 (5), 1701–1716.
Morss, R.E., Wilhelmi, O.V., Downton, M.W., Gruntfest, E., 2005. Flood risk, uncertainty,
and scientific information for decision making: lessons from an interdisciplinary
project. Bull. Am. Meteorol. Soc. 86 (11), 1593–1602.
B.F. Sanders and J.E. Schubert
Neal, J., Schumann, G., Bates, P., 2012. A subgrid channel model for simulating river
hydraulics and floodplain inundation over large and data sparse areas. Water Resour.
Res. 48 (11).
Neal, J., Villanueva, I., Wright, N., Willis, T., Fewtrell, T., Bates, P., 2012. How much
physical complexity is needed to model flood inundation? Hydrol. Process. 26 (15),
2264–2282.
Neal, J.C., Fewtrell, T.J., Bates, P.D., Wright, N.G., 2010. A comparison of three paral-
lelisation methods for 2D flood inundation models. Environ. Modell. Softw. 25 (4),
398–411.
NOAA, 2018. Billion-Dollar Weather and Climate Disasters: Overview.
NOAA National Center for Environmental Information, 2018. Coastal Elevation Models.
https://www.ngdc.noaa.gov/mgg/coastal/, Last accessed on 2018-10-7.
Özgen, I., Liang, D., Hinkelmann, R., 2016. Shallow water equations with depth-dependent
anisotropic porosity for subgrid-scale topography. Appl. Math. Model. 40 (17–18),
7447–7473.
Pasche, E., Manojlovic, N., Schertzer, D., Deroubaix, J., Tchguirinskaia, I., El Tabach, E.,
Ashley, R., Newman, R., Douglas, I., Lawson, N., et al., 2009. The use of non structural
measures for reducing the flood risk in small urban catchments. In: Samuels, et al.
(Eds.), Flood Risk Management: Research and Practice. Taylor and Francis Group,
London.
Sampson, C.C., Fewtrell, T.J., Duncan, A., Shaad, K., Horritt, M.S., Bates, P.D., 2012. Use of
terrestrial laser scanning data to drive decimetric resolution urban inundation models.
Adv. Water Resour. 41, 1–17.
Sanders, B.F., 2017. Hydrodynamic modeling of urban flood flows and disaster risk reduc-
tion. In: Cutter, S.L. (Ed.), Natural Hazard Science. Oxford University Press.
Sanders, B.F., Bradford, S.F., 2006. Impact of limiters on accuracy of high-resolution flow
and transport models. J. Eng. Mech. 132 (1), 87–98.
Sanders, B.F., Schubert, J.E., Detwiler, R.L., 2010. ParBreZo: a parallel, unstructured grid,
Godunov-type, shallow-water code for high-resolution flood inundation modeling at
the regional scale. Adv. Water Resour. 33 (12), 1456–1467.
Sanders, B.F., Schubert, J.E., Gallegos, H.A., 2008. Integral formulation of shallow-water
equations with anisotropic porosity for urban flood modeling. J. Hydrol. 362 (1–2),
19–38.
Schubert, J.E., Monsen, W.W., Sanders, B.F., 2015. Metric-resolution 2D river modeling
at the macroscale: computational methods and applications in a braided river. Front.
Earth Sci. 3, 74.
Schubert, J.E., Sanders, B.F., 2012. Building treatments for urban flood inundation models
and implications for predictive skill and modeling efficiency. Adv. Water Resour. 41,
49–64.
Schubert, J.E., Sanders, B.F., Smith, M.J., Wright, N.G., 2008. Unstructured mesh gener-
ation and landcover-based resistance for hydrodynamic modeling of urban flooding.
Adv. Water Resour. 31 (12), 1603–1621.
Schumann, G., Bates, P.D., Horritt, M.S., Matgen, P., Pappenberger, F., 2009. Progress
in integration of remote sensing–derived flood extent and stage data and hydraulic
models. Rev. Geophys. 47 (4).
95
Advances in Water Resources 126 (2019) 79–95
Schumann, G.J.-P., Bates, P.D., Neal, J.C., Andreadis, K.M., 2014. Technology: fight floods
on a global scale. Nature 507 (7491), 169.
Smith, P.J., Brown, S., Dugar, S., 2017. Community-based early warning systems for flood
risk mitigation in nepal. Nat. Hazards Earth Syst. Sci. 17 (3), 423–437.
Soares-Frazão, S., Lhomme, J., Guinot, V., Zech, Y., 2008. Two-dimensional shallow-water
model with porosity for urban flood modelling. J. Hydraul. Res. 46 (1), 45–64.
Southgate, R., Roth, C., Schneider, J., Shi, P., Onishi, T., Wenger, D., Amman, W.,
Ogallo, L., Beddington, J., Murray, V., 2013. Using science for disaster risk reduction
Report of the UNISDR Scientific and Technical Advisory Group, UNISDR, Geneva,
Switzerland.
Spiekermann, R., Kienberger, S., Norton, J., Briones, F., Weichselgartner, J., 2015. The
disaster-knowledge matrix–reframing and evaluating the knowledge challenges in dis-
aster risk reduction. Int. J. Disaster Risk Reduct. 13, 96–108.
Steinführer, A., 2009. Recommendations for Flood Risk Management with Communities
at Risk. T11-07-14.
Stelling, G.S., 2012. Quadtree flood simulations with sub-grid digital elevation models.
Proc. Inst. Civ. Eng. 165 (10), 567.
Toro, E.F., 2001. Shock-Capturing Methods for Free-Surface Shallow Flows. John Wiley
New York.
U.S. Census Bureau, 2010. 2010 Census, Los Angeles County, California.
https://www.census.gov/data.html, Last accessed on 2018-10-8.
USGS, 2006. 2006 10-foot Digital Elevation Model (DEM) - LARIAC - Public Domain.
ftp://ftpext.usgs.gov/pub/wr/ca/san.diego/ddecker/, Last accessed on 2018-10-8.
Vacondio, R., Dal Palù, A., Mignosa, P., 2014. GPU-enhanced finite volume shallow water
solver for fast flood simulations. Environ. Modell. Softw. 57, 60–75.
Valiani, A., Begnudelli, L., 2006. Divergence form for bed slope source term in shallow
water equations. J. Hydraul. Eng. 132 (7), 652–665.
Volp, N., Van Prooijen, B., Stelling, G., 2013. A finite volume approach for shallow water
flow accounting for high-resolution bathymetry and roughness data. Water Resour.
Res. 49 (7), 4126–4135.
Watts, N., Adger, W.N., Agnolucci, P., Blackstock, J., Byass, P., Cai, W., Chaytor, S., Col-
bourn, T., Collins, M., Cooper, A., et al., 2015. Health and climate change: policy
responses to protect public health. Lancet 386 (10006), 1861–1914.
Wilkinson, M.E., Mackay, E., Quinn, P.F., Stutter, M., Beven, K.J., MacLeod, C.J., Mack-
lin, M.G., Elkhatib, Y., Percy, B., Vitolo, C., et al., 2015. A cloud based tool for knowl-
edge exchange on local scale flood risk. J. Environ. Manage. 161, 38–50.
Wing, O.E., Bates, P.D., Smith, A.M., Sampson, C.C., Johnson, K.A., Fargione, J., More-
field, P., 2018. Estimates of present and future flood risk in the conterminous united
states. Environ. Res. Lett. 13 (3), 034023.
Wolsko, C., Marino, E., 2016. Disasters, migrations, and the unintended consequences of
urbanization: what’s the harm in getting out of harms way? Popul. Environ. 37 (4),
411–428.
Xia, X., Liang, Q., Ming, X., Hou, J., 2017. An efficient and stable hydrodynamic model
with novel source term discretization schemes for overland flow and flood simula-
tions. Water Resour. Res. 53 (5), 3730–3759.