Database Management Course Content

Systems
Fall 2001
CMPUT 391: Query Processing & Optimization
Dr. Osmar R. Zaïane
University of Alberta
• Introduction
• Database Design Theory
• Query Processing and Optimisation
• Concurrency Control
• Data Base Recovery and Security
• Object-Oriented Databases
• Inverted Index for IR
• XML
• Data Warehousing
• Data Mining
• Parallel and Distributed Databases
Chapters 12, 13
• Other Advanced Database Topics
&16 of Textbook

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 1  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 2

Query Processing and

Objectives of Lecture 3 Optimization
Query Processing and Optimization
• Query Processing and Planning
• Get a glimpse on query processing and • System Catalog
evaluation. • Evaluation of Relational Operations
• Introduce the issue of query planning and • Cost Estimation and Plan Selection
plan selection. • Physical Database Design Issues
• Understand the importance of good • Database Tuning
database design for good performance.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 3  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 4
 Overview of Query Processing
• The aim is to transform a query in a high-level
The Need for Optimization
declarative language (SQL) into a correct and
Consider:
efficient execution strategy
SELECT name, address
• Query Decomposition FROM Customer, Account
– Analysis WHERE Customer.name = Account.name
AND Balance > 2000
– Conjunctive and disjunctive normalization
– Semantic analysis There are different possibilities for execution:
πC.name,C.address(σC.name=A.name ∧ A.balance>2000(C×A))
• Query Optimization
πC.name,C.address(σC.name=A.name (C× σ A.balance>2000 (A))
• Query Evaluation (Execution)
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 5  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 6

Architecture for DBMS Query

General Approaches to Optimization Processing
• Heuristic-based query optimization SQL Query
– Given a query expression, perform selections and SQL Parser
projections as early as possible.
Relational Algebra Expression
– Eliminate duplicate computations.
Query Optimizer
• Cost-based query optimization Query Plan Cost
System
Generator Estimator Catalog
– Estimate the cost of different equivalent query
expressions (using the heuristics and algebra Query Execution Plan
manipulation) and choose the execution plan with Query Plan
the lowest cost estimation. Interpreter
Query Result
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 7  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 8
 Heuristic Transformations
General Guidelines Selection and projection-based transformations
• Cascading Selection
• Perform Selections and projections as early as
σ cond1∧ cond2 (R) ≡σ cond1 (σ cond2 (R))
possible
– Splitting selection formula if necessary • Commutativity of selection
– Adding projections to eliminate unused columns σ cond1 (σ cond2 (R)) ≡ σ cond2 (σ cond1 (R))
• Eliminating or reducing if possible repeated • Cascading of Projection
computations πAttribs1(πAttribs2(…(πAttribsn(R)…)) ≡ πAttribs1(R)
• Combine unary operators with binary operators • Commutativity of Selection and Projection
πAttribs(σ cond (R)) ≡ σ cond (πAttribs(R))
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 9  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 10

Heuristic Transformations Query Trees

Pushing selections and projections through joins • A query tree is a tree structure that corresponds to a
σ cond (R × S) ≡ R S relational algebra expression such that:
cond
– Each leaf node represents an input relation;
if conditions cond relate to the attributes of both R and S
– Each internal node represents a relation obtained by applying
σ cond (R × S) ≡ σ cond (R) × S one relational operator to its child nodes
if attributes in cond all belong to R (idem with joins) – The root relation represents the answer to the query
πAttribs1(R × S) ≡ πAttribs1(πAttribs2(R) × S) • Two query trees are equivalent if their root relations are
Where attribs1 ⊆ attribs2 ⊆ (R) the same (query result)
πAttribs1(R S) ≡ πAttribs1(πAttribs2(R) S) • A query tree may have different execution plans
cond cond
Attribs2 should contain all attributes in cond • Some query trees and plans are more efficient to execute
than others.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 11  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 12

Example of Query Tree
 Dr. Osmar R. Zaïane, 2001
Overview of Query Optimization
• Plan: Tree of Relational Algebra operators with choice of
algorithms for each operation.
– Each operator typically implemented using a `pull’ interface:
sname when an operator is `pulled’ for the next output tuples, it `pulls’
on its inputs and computes them.
bid=100 rating > 5
• Two main issues:
– For a given query, what plans are considered?
• Algorithm to search plan space for cheapest (estimated) plan.
sid=sid
– How is the cost of a plan estimated?
Reserves Sailors • Ideally: Want to find best plan.
• Practically: Avoid worst plans!
Database Management Systems University of Alberta 13  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 14

Query Processing and

Optimization System Catalog
• Query Processing and Planning
• A Database system maintains information
• System catalog about every relation and view it contains.
• Evaluation of Relational Operations • This information is stored in special
• Cost Estimation and Plan Selection relations called catalog relations or data
• Physical Database Design Issues dictionary
• Database Tuning • The data in the data dictionary is
extensively used for query optimization

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 15  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 16

System Catalog Information
• For each relation
– Relation name, file name, file structure
– Attribute name and type for all attributes
– Index name for all indexes on the relation
– Integrity constrains on the relation
• For each index
– Index name and structure
– Search key attributes
• For each view
– View name and definition
 Dr. Osmar R. Zaïane, 2001 Database Management Systems
Query Processing and
Optimization
• Query Processing and Planning
• System catalog
• Evaluation of Relational Operations
• Cost Estimation and Plan Selection
• Physical Database Design Issues
• Database Tuning
 Dr. Osmar R. Zaïane, 2001 Database Management Systems
Statistics Stored
• Cardinality (Ntuples(R)): number of tuples in each relation
• Size (Npages(R)): number of pages for each relation
• Index Cardinality (Nkeys(I)): number of distinct key values
• Index Size (INPages(I)): number of pages for each index
• Index Height (IHeight(I)): number of nonleaf levels for each tree
index
• Index Range: number of minimum (ILow(I)) and maximum
(IHigh(I)) present key values for each index
• Catalogs updated periodically.
– Updating whenever data changes is too expensive; lots of approximation
anyway, so slight inconsistency ok.
• More detailed information (e.g., histograms of the values in some
field, or attribute weight, etc.) are sometimes stored.
University of Alberta 17  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 18
Estimating the Result Size
• Typical optimizers estimate the size of the
relation resulting from a relational operation.
• The result size estimation plays an important
role in cost estimation because the output of an
operation can be the input of another operation.
• In a SELECT-FROM-WHERE query, the size of the
result is typically the product of the cardinality
of the relations in the FROM clause, adjusted by
the reduction effect by the conditions in the
WHERE clause.
University of Alberta 19  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 20
 Reduction Factor
Evaluating Relational Operators
• Reduction effect depends upon the terms in the condition
• Column=Value Î reduction factor estimated by • Selection (σ)
r ≈ 1/Nkeys(I). A better estimate is possible if histograms
are available. • Projection (π)
• Column1=Column2 Î reduction factor estimated by • Join ( )
r ≈ 1/(MAX(Nkeys(I1),Nkeys(I2)) • Is there more than one way to execute these
• Column > Value Î reduction factor is estimated by operations? Can we take advantage of some
r ≈ (High(I)-Value)/(High(I)-Low(I)) factors such as indexes, ordering, etc.
• Column IN (list of Values) reduction factor is estimated
• Other operators (difference, union, aggregation,
by the factor for Column=Value for all values in the list.
group by, etc.)
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 21  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 22

Evaluating the Selection Evaluating the Projection

• Size of result approximated as size of R * reduction
factor. • Projections can generate duplicate tuples after removing
unnecessary attributes.
• With no index, unsorted: Must essentially scan the
whole relation; cost is M (#pages in R). • Removing duplicates is difficult Î different approaches
• With an index on selection attribute: Use index to find • Projection based on sorting
qualifying data entries, then retrieve corresponding – Produce the set of tuples with desired attributes
data records. (Hash index useful only for equality – Sort tuples with all remaining attributes
selections.) – Scan sorted result comparing adjacent tuples
• Retrieval cost depends also upon clustering • Projection based on Hashing
– Partition result with hash function (if enough buffers)
• Complex conditions Î conjunctive normal form
– Eliminate duplicates in partitions

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 23  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 24
 Evaluating the Join
Schema for Examples
Sailors (sid: integer, sname: string, rating: integer, age: real)
• Simple Nested Loop Join Reserves (sid: integer, bid: integer, day: dates, rname: string)
• Block Nested Loop Join R  S is very
Common Î Must be
• Index Nested Loop Join carefully optimized.
• Reserves:
R × S is large; so, R × S
• Sort-Merge Join – Each tuple is 40 bytes long, 100 tuples per page, 1000 pages.
followed by a selection
• Hash Join is inefficient • Sailors:
– Each tuple is 50 bytes long, 80 tuples per page, 500 pages.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 25  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 26

Simple Nested Loops Join M pages N pages

Block Nested Loops Join
for R for S
foreach tuple r in R do • Use one page as an input
foreach tuple s in S do foreach block of B-2 of R do
if ri == sj then add <r, s> to result
buffer for scanning the
pR tuples foreach page of S do
inner S, one page as the forall matching in memory tuples
• For each tuple in the outer relation R, we scan the entire output buffer, and use all r in R-blocks and s in S-Page
add <r, s> to result
inner relation S. remaining pages to hold
``block’’ of outer R.
– Cost: M + pR * M * N = 1000 + 100*1000*500 I/Os. ≈(50 M) R&S Join Result
– For each matching tuple r Hash table for block of R
• Page-oriented Nested Loops join: For each page of R, get in R-blocks, s in S-page,
(k < B-1 pages)

...
each page of S, and write out matching pairs of tuples add <r, s> to result.
<r, s>, where r is in R-page and s is in S-page. Then read next R-block, ... ...
– Cost: M + M*N = 1000 + 1000*500 I/Os. ≈(501 103) scan S, etc.
Input buffer for S Output buffer

– If smaller relation (S) is outer, cost = 500 + 500*1000 I/Os.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 27  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 28
 Examples of Block Nested Loops Index Nested Loops Join
foreach tuple r in R do
• Cost: Scan of outer + #outer blocks * scan of inner foreach tuple s in S where ri == sj do
– #outer blocks = # of pages of outer / blocksize  add <r, s> to result
• With Reserves (R) as outer, and 100 pages of R: • If there is an index on the join column of one relation
– Cost of scanning R is 1000 I/Os; a total of 10 (B-2) blocks. (say S), can make it the inner and exploit the index.
– Î we scan Sailors (S); 10*500 I/Os. – Cost: M + ( (M*pR) * cost of finding matching S tuples)
– If space for just 90 pages of R, we would scan S 12 times (1000/90).
• For each R tuple, cost of probing S index is about 1.2 for
• With 100-page block of Sailors as outer: hash index, 2-4 for B+ tree. Cost of then finding S tuples
– Cost of scanning S is 500 I/Os; a total of 5 blocks. that match depends on clustering.
– Per block of S, we scan Reserves; 5*1000 I/Os. – Clustered index: 1 I/O (typical since all matching tuples would
• With sequential reads considered, analysis changes: may be together), unclustered: up to 1 I/O per matching S tuple since
be best to divide buffers evenly between R and S. they are scattered.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 29  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 30

Examples of Index Nested Loops Sort-Merge Join (R i=j S)

• Hash-index on sid of Sailors (as inner):
– Scan Reserves: 1000 page I/Os, 100*1000 tuples.
– For each Reserves tuple: 1.2 I/Os to get data entry in index, plus
1 I/O to get (the exactly one) matching Sailors tuple. Total:
100,000 * 1.2 + 100,000 = 220,000 I/Os.
• Hash-index on sid of Reserves (as inner):
– Scan Sailors: 500 page I/Os, 80*500 tuples.
– For each Sailors tuple: 1.2 I/Os to find index page with data
entries, plus cost of retrieving matching Reserves tuples.
Assuming uniform distribution, 2.5 reservations per sailor
(100,000 / 40,000). Cost of retrieving them is 1 or 2.5 I/Os
depending on whether the index is clustered.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta
• Sort R and S on the join column, then scan them to do a
``merge’’ (on join col.), and output result tuples.
– Advance scan of R until current R-tuple >= current S tuple, then
advance scan of S until current S-tuple >= current R tuple; do this
until current R tuple = current S tuple.
– At this point, all R tuples with same value in Ri (current R group)
and all S tuples with same value in Sj (current S group) match;
output <r, s> for all pairs of such tuples.
– Then resume scanning R and S.
• R is scanned once; each S group is scanned once per
matching R tuple. (Multiple scans of an S group are likely
to find needed pages in buffer.)
31  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 32
 Original
Example of Sort-Merge Join Hash-Join Relation OUTPUT
1
Partitions

sid bid day rname 1

• Partition both INPUT 2

sid sname rating age 28 103 12/4/96 guppy relations using hash fn hash 2
function

22 dustin 7 45.0 28 103 11/3/96 yuppy h: R tuples in ... h B-1

partition i will only
35.0 31 101 10/10/96 dustin B-1
28 yuppy 9 match S tuples in
31 lubber 8 55.5 31 102 10/12/96 lubber
partition i.
Disk B main memory buffers Disk

44 guppy 5 35.0 31 101 10/11/96 lubber Partitions

• Read in a partition of R, Join Result
58 rusty 10 35.0 58 103 11/12/96 dustin hash it using h2 (≠ h).
of R & S
Hash table for partition
Ri (k < B-1 pages)
• Cost: M log M + N log N + (M+N) Scan matching partition hash
fn
of S, search for matches. h2
– The cost of scanning, M+N, could be M*N (very unlikely!)
• Cost: Partitioning R/W
h2
• With 35, 100 or 300 buffer pages, both Reserves and once R and S= 2(M+N).

Î
Phase 2: read partitions Input buffer Output
Sailors can be sorted in 2 passes; total join cost: 7500 I/Os. once M+N. Total for Si buffer

However with BNL join could be less I/Os with 100 buffers 3(M+N) Disk B main memory buffers Disk
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 33  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 34

Query Processing and

Optimization Highlights of System R Optimizer
• Impact:
• Query Processing and Planning – Most widely used currently; works well for < 10 joins.
• System catalog • Cost estimation: Approximate art at best.
– Statistics, maintained in system catalogs, used to estimate cost of
• Evaluation of Relational Operations
operations and result sizes.
• Cost Estimation and Plan Selection – Considers combination of CPU and I/O costs.
• Physical Database Design Issues • Plan Space: Too large, must be pruned.
– Only the space of left-deep plans is considered.
• Database Tuning • Left-deep plans allow output of each operator to be pipelined into the next
operator without storing it in a temporary relation.
– Cartesian products avoided.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 35  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 36
 sname

Motivating Example
Schema for Examples SELECT S.sname
bid=100 rating > 5

FROM Reserves R, Sailors S RA Tree:

Sailors (sid: integer, sname: string, rating: integer, age: real) WHERE R.sid=S.sid AND
sid=sid
Reserves (sid: integer, bid: integer, day: dates, rname: string) R.bid=100 AND S.rating>5
Reserves Sailors
(On-the-fly)
sname
• Cost: 500+500*1000 I/Os
Plan:
• Reserves: • By no means the worst plan!
– Each tuple is 40 bytes long, 100 tuples per page, 1000 pages. bid=100 rating > 5 (On-the-fly) • Misses several opportunities:
selections could have been `pushed’
• Sailors: earlier, no use is made of any
(Simple Nested Loops) available indexes, etc.
– Each tuple is 50 bytes long, 80 tuples per page, 500 pages. sid=sid
• Goal of optimization: Find more
efficient plans that compute the same
Reserves Sailors answer.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 37  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 38

Alternative Plans 1 sname

(On-the-fly)
Alternative Plans 2 sname
(On-the-fly)

(No Indexes) (Sort-Merge Join) With Indexes rating > 5 (On-the-fly)

sid=sid • With clustered index on bid of

(Scan;
Reserves, (100 boats) we get (Index Nested Loops,
with pipelining )
(Scan; sid=sid
write to bid=100
temp T1)
rating > 5 write to
temp T2)
100,000/100 = 1000 tuples on 1000/100
• Main difference: push selects. = 10 pages for each boat. (Use hash
index; do bid=100 Sailors
Reserves Sailors
• With 5 buffers, cost of plan: • INL with pipelining (outer is not
not write
result to
temp)
– Scan Reserves (1000) + write temp T1 (10 pages, if we have 100 boats, uniform materialized). Reserves
distribution). –Projecting out unnecessary fields from outer doesn’t help.
– Scan Sailors (500) + write temp T2 (250 pages, if we have 10 ratings). ❖ Join column sid is a key for Sailors.
– Sort T1 (2*2*10), sort T2 (2*4*250), merge (10+250) –At most one matching tuple, unclustered index on sid OK.
– Total: 4060 page I/Os.
❖ Decision not to push rating>5 before the join is based on
• If we used BNL join, join cost = 10+4*250, total cost = 2770. availability of sid index on Sailors.
• If we `push’ projections, T1 has only sid, T2 only sid and sname: ❖ Cost: Selection of Reserves tuples (10 I/Os); for each,
– T1 fits in 3 pages, cost of BNL drops to under 250 pages, total < 2000. must get matching Sailors tuple (1000*1.2); total 1210 I/Os.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 39  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 40
 Cost Estimation Size Estimation and Reduction Factors
• For each plan considered, must estimate cost: SELECT attribute list
– Must estimate cost of each operation in plan tree. FROM relation list
• Depends on input cardinalities. • Consider a query block: WHERE term1 AND ... AND termk
• We’ve already discussed how to estimate the cost of operations
(sequential scan, index scan, joins, etc.) • Maximum # tuples in result is the product of the
– Must estimate size of result for each operation in tree! cardinalities of relations in the FROM clause.
• Use information about the input relations. • Reduction factor (RF) associated with each term reflects
• For selections and joins, assume independence of predicates.
the impact of the term in reducing result size. Result
• The System R cost estimation approach. cardinality = Max # tuples * product of all RF’s.
– Very inexact, but works OK in practice. – Implicit assumption that terms are independent!
– More sophisticated techniques known now. – Term col=value has RF 1/NKeys(I), given index I on col
• Query plans estimated at run-time or estimated once and – Term col1=col2 has RF 1/MAX(NKeys(I1), NKeys(I2))
elected plan stored and revisited for re-evaluation. – Term col>value has RF (High(I)-value)/(High(I)-Low(I))
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 41  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 42

Query Processing and

Summary
Optimization
• Query optimization is an important task in a relational
DBMS. • Query Processing and Planning
• Must understand optimization in order to understand the
• System catalog
performance impact of a given database design (relations,
indexes) on a workload (set of queries). • Evaluation of Relational Operations
• Two parts to optimizing a query: • Cost Estimation and Plan Selection
– Consider a set of alternative plans.
• Physical Database Design Issues
• Must prune search space; typically, left-deep plans only.
– Must estimate cost of each plan that is considered. • Database Tuning
• Must estimate size of result and cost for each plan node.
• Key issues: Statistics, indexes, operator implementations.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 43  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 44
 Overview
• After ER design, schema refinement, and the definition
Understanding the Workload
of views, we have the conceptual and external schemas • For each query in the workload:
for our database. – Which relations does it access?
• The next step is to choose indexes, make clustering – Which attributes are retrieved?
decisions, and to refine the conceptual and external – Which attributes are involved in selection/join conditions? How
schemas (if necessary) to meet performance goals. selective are these conditions likely to be?
• We must begin by understanding the workload: • For each update in the workload:
– The most important queries and how often they arise. – Which attributes are involved in selection/join conditions? How
– The most important updates and how often they arise. selective are these conditions likely to be?
– The desired performance for these queries and updates. – The type of update (INSERT/DELETE/UPDATE), and the attributes
that are affected.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 45  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 46

Decisions to Make
Choice of Indexes
• What indexes should we create?
– Which relations should have indexes? What field(s) should be • One approach: consider the most important queries
the search key? Should we build several indexes? in turn. Consider the best plan using the current
• For each index, what kind of an index should it be? indexes, and see if a better plan is possible with an
– Clustered? Hash/tree? Dynamic/static? Dense/sparse? additional index. If so, create it.
• Should we make changes to the conceptual schema?
• Before creating an index, must also consider the
– Consider alternative normalized schemas? (Remember, there are
many choices in decomposing into BCNF, etc.) impact on updates in the workload!
– Should we ``undo’’ some decomposition steps and settle for a – Trade-off: indexes can make queries go faster, updates
lower normal form? (Denormalization.) slower. Require disk space, too.
– Horizontal partitioning, replication, views ...
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 47  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 48
 Issues to Consider in Index Issues in Index Selection (Contd.)
Selection • Multi-attribute search keys should be considered when a
• Attributes mentioned in a WHERE clause are candidates for WHERE clause contains several conditions.
index search keys. – If range selections are involved, order of attributes should be
carefully chosen to match the range ordering.
– Exact match condition suggests hash index.
– Such indexes can sometimes enable index-only strategies for
– Range query suggests tree index.
important queries. (no need to access the relation)
• Clustering is especially useful for range queries, although it can help on
equality queries as well in the presence of duplicates. • For index-only strategies, clustering is not important!

• Try to choose indexes that benefit as many queries as • When considering a join condition:
possible. Since only one index can be clustered per – Hash index on inner is very good for Index Nested Loops.
• Should be clustered if join column is not key for inner, and inner tuples
relation, choose it based on important queries that would need to be retrieved.
benefit the most from clustering. – Clustered B+ tree on join column(s) good for Sort-Merge.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 49  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 50

SELECT E.ename, D.dname

SELECT E.ename, D.mgr
FROM Emp E, Dept D
Example 1 FROM Emp E, Dept D
WHERE D.dname=‘Toy’ AND E.dno=D.dno
Example 2 WHERE E.sal BETWEEN 10000 AND 20000
AND E.hobby=‘Stamps’ AND E.dno=D.dno
• Hash index on D.dname supports ‘Toy’ selection. • Clearly, Emp should be the outer relation.
– Given this, index on D.dno is not needed. Nothing is gained by an
– Suggests that we build a hash index on D.dno.
index on D.dno since Dept tuples are retrieved with dname index
• What index should we build on Emp?
• Hash index on E.dno allows us to get matching (inner) Emp
– B+ tree on E.sal could be used, OR an index on E.hobby could be
tuples for each selected (outer) Dept tuple. used. Only one of these is needed, and which is better depends
• What if WHERE included: “ ... AND E.age=25” ? upon the selectivity of the conditions.
– Could retrieve Emp tuples using index on E.age, then join with • As a rule of thumb, equality selections more selective than range selections.
Dept tuples satisfying dname selection. Comparable to strategy • As both examples indicate, our choice of indexes is guided
that used E.dno index. by the plan(s) that we expect an optimizer to consider for a
– So, if E.age index is already created, this query provides much query. Have to understand optimizers!
less motivation for adding an E.dno index.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 51  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 52
 Examples of Clustering Clustering and Joins
SELECT E.dno
FROM Emp E SELECT E.ename, D.mgr
• B+ tree index on E.age can be used to
WHERE E.age>40 FROM Emp E, Dept D
get qualifying tuples. WHERE D.dname=‘Toy’ AND E.dno=D.dno
– How selective is the condition? SELECT E.dno, COUNT (*) • Clustering is especially important when accessing inner
– Is the index clustered? FROM Emp E
tuples in INL.
WHERE E.age>10
• Consider the GROUP BY query. – Should make index on E.dno clustered.
GROUP BY E.dno
– If many tuples have E.age > 10, using • Suppose that the WHERE clause is instead:
E.age index and sorting the retrieved
WHERE E.hobby=‘Stamps AND E.dno=D.dno
tuples may be costly.
– If many employees collect stamps, Sort-Merge join may be worth
– Clustered E.dno index may be better!
SELECT E.dno considering. A clustered index on D.dno would help.
• Equality queries and duplicates: FROM Emp E
• Summary: Clustering is useful whenever many tuples are
– Clustering on E.hobby helps! WHERE E.hobby=Stamps
to be retrieved.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 53  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 54

SELECT D.mgr
FROM Dept D, Emp E
Multi-Attribute Index Keys Index-Only Plans <E.dno> WHERE D.dno=E.dno

SELECT D.mgr, E.eid

<E.dno,E.eid>
• To retrieve Emp records with age=30 AND sal=4000, an • A number of FROM Dept D, Emp E
Tree index! WHERE D.dno=E.dno
index on <age,sal> would be better than an index on age or queries can be
an index on sal. answered without SELECT E.dno, COUNT(*)
<E.dno> FROM Emp E
– Such indexes also called composite or concatenated indexes. retrieving any GROUP BY E.dno
– Choice of index key orthogonal to clustering etc. tuples from one
<E.dno,E.sal> SELECT E.dno, MIN(E.sal)
• If condition is: 20<age<30 AND 3000<sal<5000: or more of the FROM Emp E
Tree index!
relations GROUP BY E.dno
– Clustered tree index on <age,sal> or <sal,age> is best.
• If condition is: age=30 AND 3000<sal<5000: involved if a <E. age,E.sal> SELECT AVG(E.sal)
suitable index is or FROM Emp E
– Clustered <age,sal> index much better than <sal,age> index! <E.sal, E.age> WHERE E.age=25 AND
available. E.sal BETWEEN 3000 AND 5000
• Composite indexes are larger, updated more often. Tree!
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 55  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 56
 Summary Summary (Contd.)
• Database design consists of several tasks: requirements
• Indexes must be chosen to speed up important queries (and
analysis, conceptual design, schema refinement, physical
perhaps some updates!).
design and tuning.
– Index maintenance overhead on updates to key fields.
– In general, have to go back and forth between these tasks to refine
a database design, and decisions in one task can influence the – Choose indexes that can help many queries, if possible.
choices in another task. – Build indexes to support index-only strategies.
• Understanding the nature of the workload for the – Clustering is an important decision; only one index on a given
relation can be clustered!
application, and the performance goals, is essential to
– Order of fields in composite index key can be important.
developing a good design.
– What are the important queries and updates? What • Static indexes may have to be periodically re-built.
attributes/relations are involved? • Statistics have to be periodically updated.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 57  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 58

Query Processing and

Tuning the Conceptual Schema
Optimization • The choice of conceptual schema should be guided by the
workload, in addition to redundancy issues:
• Query Processing and Planning – We may settle for a 3NF schema rather than BCNF.
• System catalog – Workload may influence the choice we make in decomposing a
relation into 3NF or BCNF.
• Evaluation of Relational Operations – We may further decompose a BCNF schema!
• Cost Estimation and Plan Selection – We might denormalize (i.e., undo a decomposition step), or we
might add fields to a relation.
• Physical Database Design Issues
– We might consider horizontal decompositions.
• Database Tuning • If such changes are made after a database is in use, called
schema evolution; might want to mask some of these
changes from applications by defining views.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 59  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 60
 Example Schemas
Contracts (Cid, Sid, Jid, Did, Pid, Qty, Val)
Depts (Did, Budget, Report)
Suppliers (Sid, Address)
Parts (Pid, Cost)
Projects (Jid, Mgr)
• We will concentrate on Contracts, denoted as CSJDPQV.
The following ICs are given to hold:JP → C, SD → P,
C is the primary key.
– What are the candidate keys for CSJDPQV?
– What normal form is this relation schema in?
 Dr. Osmar R. Zaïane, 2001 Database Management Systems
Settling for 3NF vs BCNF
• CSJDPQV can be decomposed into SDP and CSJDQV,
and both relations are in BCNF. (Which FD suggests that
we do this?)
– Lossless decomposition, but not dependency-preserving.
– Adding CJP makes it dependency-preserving as well.
• Suppose that this query is very important:
– Find the number of copies Q of part P ordered in contract C.
– Requires a join on the decomposed schema, but can be answered
by a scan of the original relation CSJDPQV.
– Could lead us to settle for the 3NF schema CSJDPQV.
University of Alberta 61  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 62

Denormalization Choice of Decompositions

• Suppose that the following query is important: • There are 2 ways to decompose CSJDPQV into BCNF:
– Is the value of a contract less than the budget of the department? – SDP and CSJDQV; lossless-join but not dep-preserving.
• To speed up this query, we might add a field budget B to – SDP, CSJDQV and CJP; dep-preserving as well.
Contracts. • The difference between these is really the cost of enforcing
– This introduces the FD D→ B wrt Contracts. the FD JP → C.
– Thus, Contracts is no longer in 3NF. – 2nd decomposition: Index on JP on relation CJP.
• We might choose to modify Contracts thus if the query is – 1st: CREATE ASSERTION CheckDep
CHECK ( NOT EXISTS ( SELECT *
sufficiently important, and we cannot obtain adequate FROM PartInfo P, ContractInfo C
performance otherwise (i.e., by adding indexes or by WHERE P.sid=C.sid AND P.did=C.did
choosing an alternative 3NF schema.) GROUP BY C.jid, P.pid
HAVING COUNT (C.cid) > 1 ))
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 63  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 64
 Choice of Decompositions (Contd.)
• The following ICs were given to hold:
JP→ C, SD → P, C is the primary key.
• Suppose that, in addition, a given supplier always charges
the same price for a given part: SPQ → V.
• If we decide that we want to decompose CSJDPQV into
BCNF, we now have a third choice:
– Begin by decomposing it into SPQV and CSJDPQ.
– Then, decompose CSJDPQ (not in 3NF) into SDP, CSJDQ.
– This gives us the lossless-join decomp: SPQV, SDP, CSJDQ.
– To preserve JP → C, we can add CJP, as before.
• Choice: { SPQV, SDP, CSJDQ } or { SDP, CSJDQV } ?
 Dr. Osmar R. Zaïane, 2001 Database Management Systems
Horizontal Decompositions
• Our definition of decomposition: Relation is
replaced by a collection of relations that are
projections. Most important case.
• Sometimes, might want to replace relation by a
collection of relations that are selections.
– Each new relation has same schema as the original, but a
subset of the rows.
– Collectively, new relations contain all rows of the
original. Typically, the new relations are disjoint.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems
Decomposition of a BCNF
Relation
• Suppose that we choose { SDP, CSJDQV }. This is in
BCNF, and there is no reason to decompose further
(assuming that all known ICs are FDs).
• However, suppose that these queries are important:
– Find the contracts held by supplier S.
– Find the contracts that department D is involved in.
• Decomposing CSJDQV further into CS, CD and CJQV
could speed up these queries. (Why?)
• On the other hand, the following query is slower:
– Find the total value of all contracts held by supplier S.
University of Alberta 65  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 66
Horizontal Decompositions (Contd.)
• Suppose that contracts with value > 10000 are subject to
different rules. This means that queries on Contracts will
often contain the condition val>10000.
• One way to deal with this is to build a clustered B+ tree
index on the val field of Contracts.
• A second approach is to replace contracts by two new
relations: LargeContracts and SmallContracts, with the
same attributes (CSJDPQV).
– Performs like index on such queries, but no index overhead.
– Can build clustered indexes on other attributes, in addition!
University of Alberta 67  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 68
 Masking Conceptual Schema Changes Tuning Queries and Views
CREATE VIEW Contracts(cid, sid, jid, did, pid, qty, val)
AS SELECT * • If a query runs slower than expected, check if an index
FROM LargeContracts needs to be re-built, or if statistics are too old.
UNION
• Sometimes, the DBMS may not be executing the plan you
SELECT *
FROM SmallContracts
had in mind. Common areas of weakness:
– Selections involving null values.
• The replacement of Contracts by LargeContracts and – Selections involving arithmetic or string expressions.
SmallContracts can be masked by the view. – Selections involving OR conditions.
• However, queries with the condition val>10000 must be – Lack of evaluation features like index-only strategies or certain
join methods or poor size estimation.
asked wrt LargeContracts for efficient execution: so users
concerned with performance have to be aware of the • Check the plan that is being used! Then adjust the choice
change. of indexes or rewrite the query/view.
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 69  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 70

More Guidelines for Query Tuning Guidelines for Query Tuning (Contd.)
SELECT * INTO Temp
FROM Emp E, Dept D
• Minimize the use of DISTINCT: don’t need it if duplicates
• Avoid using intermediate WHERE E.dno=D.dno
are acceptable, or if answer contains a key. AND D.mgrname=‘Joe’
• Minimize the use of GROUP BY and HAVING:
relations:
SELECT E.dno, AVG(E.sal)
and
SELECT MIN (E.age) FROM Emp E, Dept D
SELECT MIN (E.age)
vs. WHERE E.dno=D.dno SELECT T.dno, AVG(T.sal)
FROM Employee E FROM Employee E
AND D.mgrname=‘Joe’ FROM Temp T
GROUP BY E.dno WHERE E.dno=102
GROUP BY E.dno GROUP BY T.dno
HAVING E.dno=102
❖ Does not materialize the intermediate reln Temp.
❖ Consider DBMS use of index when writing arithmetic ❖ If there is a dense B+ tree index on <dno, sal>, an index-only
expressions: E.age=2*D.age will benefit from index on E.age, plan can be used to avoid retrieving Emp tuples in the second
but might not benefit from index on D.age! query!
 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 71  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 72
 Summary of Database Tuning Summary (Contd.)
• The conceptual schema should be refined by considering • Over time, indexes have to be fine-tuned (dropped, created,
performance criteria and workload: re-built, ...) for performance.
– May choose 3NF or lower normal form over BCNF. – Should determine the plan used by the system, and adjust the
– May choose among alternative decompositions into BCNF (or choice of indexes appropriately.
3NF) based upon the workload. • System may still not find a good plan:
– May denormalize, or undo some decompositions. – Only left-deep plans considered!
– May decompose a BCNF relation further! – Null values, arithmetic conditions, string expressions, the use of
– May choose a horizontal decomposition of a relation. ORs, etc. can confuse an optimizer.
– Importance of dependency-preservation based upon the • So, may have to rewrite the query/view:
dependency to be preserved, and the cost of the IC check.
• Can add a relation to ensure dep-preservation (for 3NF, not BCNF!); or
– Avoid nested queries, temporary relations, complex conditions,
else, can check dependency using a join. and operations like DISTINCT and GROUP BY.

 Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 73  Dr. Osmar R. Zaïane, 2001 Database Management Systems University of Alberta 74