BasicAspectsofMetaAnalysis

HannahR.Rothstein
BaruchCollege CityUniversityofNew
York
MichaelBorenstein
Biostat
Split,CroatiaJune2014 1
 Agenda
Primarystudies,systematicreviewsandmeta
analysis
Basiccomponentsofametaanalysis
EffectSizes
BasicCalculations
Heterogeneityofeffects
Comparisonoffixedeffectandrandomeffects
models
Split,CroatiaJune2014 2
PartI

PRIMARYSTUDIES,SYSTEMATIC
REVIEWSANDMETAANALYSIS

Split,CroatiaJune2014 3
 Primarystudy
Hypothesis
Inclusion/exclusioncriteria
Collectdata
Statisticalanalysis(metaanalysis)
Report

Split,CroatiaJune2014 4
 Systematicreview
Hypothesis
Inclusion/exclusioncriteria
Collectdata(informationretrievalanddata
extraction)
Statisticalanalysis(metaanalysis)
Report

Split,CroatiaJune2014 5
 Metaanalysis
Metaanalysisisthequantitativedataanalysis
componentofasystematicreview.
Notallsystematicreviewsincludeametaanalysis.
Notallmetaanalysesareprecededbytheearlier
componentsofasystematicreview.
Inpractice,researcherswilloftenusesystematic
review,metaanalysisandresearchsynthesisas
synonyms.

Split,CroatiaJune2014 6
 ASystematicReviewAimsToBe
Explicit(e.g.initsstatementofobjectives,
materialsandmethods)
Systematic(e.g.initsidentificationof
literature)
Transparent(e.g.initscriteriaanddecisions)
Reproducible(e.g.initsmethodologyand
conclusions)
Unbiased

Split,CroatiaJune2014 7
Themovetosystematicreviews
andawayfromnarrativereviews

Split,CroatiaJune2014 8
Narrativereviewsarenotscientifically
rigorous
Theyuseinformalandsubjectivemethodsto
collectandinterpretinformation
Theygenerallyprovideanarrativesummaryof
theresearchliterature
Differentexpertsmayperformareviewonthe
samequestionandcometodifferentconclusions
Sometimesduetoreviewofdifferentsetsofstudies
Butcanhappenevenwhenthesamestudiesare
reviewed

Split,CroatiaJune2014 9
 Narrativereviewsbecomeless
efficientwithmoredata
Researchermaybeabletocombinethe
resultsofafewstudiesinhisorherhead
Becomesincreasinglydifficulttodosoasthe
numberofstudiesincrease
Weusestatisticstocombineinformation
withinasinglestudy.

Split,CroatiaJune2014 10
 Problemswhenthetreatment
effectvaries
Asthenumberofstudiesgrow,theyoften
examinedifferentpopulations
Thesizeoftherelationshipofinterestmay
varyindifferentpopulations
Thenarrativereviewer,whohasenough
troublesummarizingstudieswhentheyareall
doneinsimilarsituationsnowhasamuch
hardertask

Split,CroatiaJune2014 11
 Toomuchliterature
Amongtheearliestmetaanalysesweresynthesisof:
345studiesoftheeffectsofinterpersonalexpectationson
behavior(Rosenthal&Rubin,1978),
725estimatesoftherelationbetweenclasssizeand
academicachievement(Glass&Smith,1979),
833testsoftheeffectivenessofpsychotherapy(Smith&
Glass,1977),
866comparisonsofthedifferentialvalidityof
employmenttestsforBlackandWhiteworkers(Hunter,
Schmidt&Hunter,1979)

WeekOneIntroduction 12
 Isclasssizerelatedtostudent
achievement?
By1978therewerehundredsofstudieson
thistopic

WeekOneIntroduction 13
 Narrativereview
Thompsonconcludedthattherelationship
...involvedtoomanycomplexissuestobe
reducedtoasingletestablehypothesis,and
thatresearchfindingswerenecessarily
inconclusive.

WeekOneIntroduction 14
 Metaanalysis
SmithandGlass(1978)
80studies
700comparisonsofsmallerandlargerclasses
Resultsshowedclearlythatsmallerclassesare
betteron
studentachievement,
classroomprocesses
teacherandstudentattitudes.

WeekOneIntroduction 15
 Systematicvs.NarrativeReviews
Scientificapproach(models Influencedbyauthorspoint
itselfonprimaryresearch) ofview(bias)
Criteriadeterminedapriori Authordoesnotneedto
Comprehensivesearchfor justifycriteriaforinclusion
relevantinformation Searchfordatadoesnot
Explicitmethodsofdata needtobecomprehensive
extractionandcoding Methodsnotusually
Metaanalysisgenerallybe specified
usedtocombinedata Narrativesummaryorvote
Replicable count
Cantreplicatereview

Split,CroatiaJune2014 16
 Anattemptedfix:Votecounting
Collectasetofstudies
Examinethetestsofstatisticalsignificance
TallytheproportionsignificantasYesvotes
TallytheproportionnotsignificantasNovotes
Majoritywins
(Therearevariantswiththreecategories:positive,
,negativeandnonsignificant)

Split,CroatiaJune2014 17
Warr andPerry(1982)Psych.Bulletin

Split,CroatiaJune2014 18
 Votecounting
Focusesonthestatisticalsignificanceofthe
primarystudies
Votecountingtreatsanonsignificantpvalue
asevidencethataneffectisabsent.Infact,
though,small,moderate,andevenlargeeffect
sizesmayyieldanonsignificantpvaluedueto
inadequatestatisticalpower.
Oftenwrong,duetolowpowerofprimary
studies

Split,CroatiaJune2014 19
VoteCountingincorrectconclusions
HedgesandOlkin(1980)showedthatthe
powerofvotecountingcannotonlybelower
thanthatofthestudiesonwhichitisbased,
butcantendtowardzeroasthenumberof
studiesincreases.
With20studiesofN=30andaneffectofd=.5
avotecountwillfailtodetecttheeffect75%
ofthetime
Inotherwords,votecountingisnotonly
misleading,ittendstobemoremisleadingas
theamountofevidenceincreases!
Split,CroatiaJune2014 20
 VoteCounting
Evenwhencorrect,doesntprovide
informationaboutthesizeofeffectsorthe
consistencyofeffectsacrossstudies
Doesntgivemoreweighttomoreprecise
studies

Split,CroatiaJune2014 21
SpecialSituations:Combinedsignificancetestand
signtest
OTHERAPPROACHESTOSYNTHESIS

WeekOneIntroduction 22
 Combinedsignificancetestmeta
analysis(Rosenthalsearlyattempt)
Advantage:liesintheincreasedpoweroftheoverall
comparison.
Ifseveraltestsconsistentlyfavortheresearchquestion
butfailtoreachthelevelofsignificance,duetosmall
samplesizes,theoveralltestismorelikelytoreach
significancebecausethepooledsamplesizeismuch
larger.
Thehypothesisbeingtestedisthatthenullhypothesisis
trueineverystudy.Ifthishypothesisisrejected,wecan
concludethatthereisatleastonestudywithanonnull
effect.
Ittellsyounothingaboutthemagnitudeofeffectsize(s)

WeekOneIntroduction 23
JCCP1987ShohamSalomon&Rosenthal,Paradoxical
Interventions
 Combiningp Values
Collectasetofstudies
Extractthep valuesfromallofthetestsofsignificance
(whethersignificantornot)
Computeanoverallp value
Youobtaintheoverallstrengthofevidencethatan
effect exists
ImportantNote:Thistestdoesnottellusanything
aboutthevalueofanoveralleffect,oritsstatistical
significance
Thiswaspopularintheearlydaysofmetaanalysis;
Rosenthalrecommendeditin1978.
Wehavebettermethodsnow,butsometimesthisis
thebestwecando.
 Combiningpvalues
Advantages
Youcanusethisifallyouhaveistheresultsof
significancetests
Youcancombinepvaluesfromanyteststatistic
representingthesubstantivehypothesisof
interest,evenifthestudiesvaryindesignor
analysis
Apvalueisapvalueisapvalue

Split,CroatiaJune2014 26
 Combiningpvalues
Disadvantages
Sameissuesasinprimarystudies
Mayhaveeffectthatislargebutnot
statisticallysignificant
particularlyinprimarystudies
Mayhaveeffectthatistrivial,yetstatistically
significant
aparticularproblemforoverallmetaanalysis

Split,CroatiaJune2014 27
 Alastresort:Thesigntest
Thesigntestisusedtocountthenumberof
studieswithfindingsinonedirection
comparedtothenumberoffindingsinthe
otherdirection,withoutregardtowhether
thefindingsarestatisticallysignificant
Ifatreatmentiscompletelyineffective,we
expectthathalfofthestudieswillfalloneach
sideofthenoeffectline
 Thesigntest
Teststhehypothesisthattheeffectsizesfrom
acollectionofKindependentstudiesareall
zero
Simplytestifproportionofpositiveresultsis
50%
Ifthetreatmenthasaneffect,theprobabilityof
gettingapositiveresultisgreaterthan.5
Ifithasnoeffect,theprobabilityofgettinga
positiveresultis.5
Split,CroatiaJune2014 29
 Signtestadvantages
Thesigntestisusefulwhen
Nonumericdataareprovidedfromstudies,but
directionsofeffectsareprovided
NumericdataareSOdifferentthattheycannotbe
combinedstatistically
Studiesaresodiverseintheirpopulationsorother
characteristicsthatapooledeffectsizeis
meaningless,butstudiesareaddressinga
questionssufficientlysimilarthatthedirectionof
effectismeaningful.
Resultscanbetestedforstatisticalsignificance
usingthestandard binomialtest
 Signtestdisadvantages
Doesntincorporatesamplesize(givemore
weighttomoreprecisestudies)
Doesnotprovideanestimateofeffectsize

Split,CroatiaJune2014 31
 Signtest:Example
Healthrelatedqualityoflifeafterlivertransplantation:
ametaanalysis(Bravata etal.,1999,LiverTransp
Surg).
Performedasignteston49studiestoevaluatethe
direction(positiveornegative)oftheeffectof
transplantationonQOL.
Thesigntestshowedsignificantimprovementin
posttransplantation physicalhealth(P<.0004),sexual
functioning(P<.008),dailyactivities(P<.02),general
HRQL(P<.02),andsocialfunctioning(P<.05),butnot
psychologicalhealth(P<.08).

Split,CroatiaJune2014 32
COMPONENTSOFAMETAANALYSIS

Split,CroatiaJune2014 33
Split,CroatiaJune2014 34
Split,CroatiaJune2014 35
 Effectsizeandweight
Effectsizecanbebasedonmeans,
proportions,andsoon
Weightsbasedonamountofinformation
carriedbyeachstudy

Split,CroatiaJune2014 36
Split,CroatiaJune2014 37
 SummaryeffectandCI
Summaryeffectisweightedmean
Confidenceintervalbasedonsamestatistics
asprimarystudy
Takesaccountofmultiplelevelsofsampling

Split,CroatiaJune2014 38
Split,CroatiaJune2014 39
 Ametaanalysisisasynthesis
Howweapproachthesynthesisdependson
ourresearchquestion
Whatkindsofstudiesweincludedependson
ourresearchquestion
Thekindofanalysisdependsonourresearch
question
Theconclusionsdependonourresearch
questionandtheinformationavailable

Split,CroatiaJune2014 40
 Comparedtoprimarystudy
Providesacontext
Thetreatmenteffectissimilaracrossthesetof
studies
Thetreatmenteffectvariesinunknownways
Thetreatmenteffectvariesinwayswecanmodel

Split,CroatiaJune2014 41
 Whenthetreatmenteffectis
consistentacrossstudies
Metaanalysis
Providesamorepreciseestimateofeffectsize
Providesincreasedstatisticalpower

Split,CroatiaJune2014 42
 Streptokinase
Anowclassicsystematicreviewandmeta
analysis

Split,CroatiaJune2014 43
Split,CroatiaJune2014 44
 BasicAnalysis

Treatmenteffectwasconsistent
Combinedeffectwassubstantial
Only6of33studiesmetcriterionfor
significance

Split,CroatiaJune2014 45
Split,CroatiaJune2014 46
 CumulativeAnalysis

Metaanalysisin1977couldhavebeen
definitive
40,000patientsrandomizedafter1977
Additionalmillionsnottreated
Evenin1992,narrativereviewwasnot
definitive

Split,CroatiaJune2014 47
 Studiesvaryinunknownways
Metaanalysis
Providesmorepreciseestimateofeffectsize
Providesincreasedstatisticalpower
Allowsustomeasuretheheterogeneityofthe
effect

Split,CroatiaJune2014 48
Split,CroatiaJune2014 49
 Prevalence
Basedontheheterogeneityindex(tau
squared),mostofthevariationhereisreal;
thatisitisnotduetochance.
Prevalencevariedfrom5%to40%across
studies;thereasonsforthisheterogeneity
werenotinvestigated.

Split,CroatiaJune2014 50
 Prevalence of
Prevalence
Pevalence of PTSD
of PTSD
PTSD
Study
Study name
Study name
name Statistics
Statistics
Statistics for study
for study
for each each each study Event
Event
Event rate
rateand
rate and and
95%
95% CICI 95% CI
EventEvent
LowerLower
Event Upper
Lower Upper Upper
rate rate limit limitlimit limit
rate limit limit
Pelcovitz, 19961996 0.2160.2160.112 0.112
Pelcovitz, 0.376 0.376
Pelcovitz,
Barakat, 1996
1997
Barakat,
0.216
1997 0.0900.0900.066 0.066
0.1120.122 0.376
0.122
Barakat,
Manne, 1997
1998
Manne, 1998 0.090
0.0500.0500.021 0.0210.0660.115 0.122
0.115
Manne,
Fuemmeler, 1998 0.050
200120010.4000.400
Fuemmeler, 0.243 0.2430.0210.581 0.115
0.581
Fuemmeler,
Libov, 20022002 2001
Libov, 0.400
0.1880.1880.116 0.1160.2430.288 0.581
0.288
Libov,
Brown, 2002
2003
Brown, 2003 0.188
0.2350.2350.157 0.1570.1160.337 0.288
0.337
Brown,
Kazak, 2003
2004
Kazak, 2004 0.235
0.1300.1300.090 0.0900.1570.184 0.337
0.184
Kazak,
Manne, 2004
2004
Manne, 2004 0.130
0.1080.1080.064 0.0640.0900.178 0.184
0.178
Manne,
Kazak, 2004
2005
Kazak, 2005 0.108
0.2400.2400.186 0.1860.0640.304 0.178
0.304
Landolt,
Kazak, 2005
2005
Landolt, 2005 0.2140.2140.134 0.134
0.240 0.326
0.1860.326 0.304
Glover
Landolt,
Glover2005 0.3100.3100.227 0.227
0.214 0.407
0.1340.407 0.326
Glover 0.1780.1800.158 0.130
0.310 0.201
0.2270.244 0.407
0.180 0.130 -1.000.244
-1.00 -0.50-0.50 0.00
0.00 0.50
0.50 1.001.00
-0.50 -0.25 0.00 0.25 0.50

Split,CroatiaJune2014 51
Random-effects
Meta Analysis model
Random-effects model
Studiesvaryinwayswecanmodel
Metaanalysis
Allowsustoassesstreatmenteffect(and
variation)withinsubgroups
Assessustomeasuretheimpactof
moderator(s)onthetreatmenteffect

Split,CroatiaJune2014 52
 ImpactoftwoDrugs
OnWeightLoss

Split,CroatiaJune2014 53
Split,CroatiaJune2014 54
 Outcomesmatter
Thesetwoobesitydrugshavedifferent
mechanisms,differentsideeffectsand
differentcontraindications.
Ifwehadlookedatoneofthesideeffectsas
anoutcome,wemighthaveadifferent
picture.

Split,CroatiaJune2014 55
Split,CroatiaJune2014 56
 Caffeine by Analgesic | Relief

Group by Study name Statistics for each study Risk ratio and 95% CI
Analgesic
Risk Lower Upper
ratio limit limit p-Value

Ibuprofen 1.294 1.119 1.496 0.001

Paracetamol 1.111 1.060 1.165 0.000

Overall 1.127 1.078 1.179 0.000

0.5 1 2

Favours Placebo Favours Caffeine

Meta Analysis

Split,CroatiaJune2014 57
Split,CroatiaJune2014 58
ImpactofStatinsbySmoking

Split,CroatiaJune2014 59
 Bottomline
Canlimittheanalysistostudiesthatare
essentiallyidentical,andgetmoreprecise
estimateofthecommoneffect
Canincludestudiesthatvaryinrandomways,
assessvariationineffect
Canincludestudiesthatvaryonpotentially
importantfactors,assesstheimpactofthese
factors

Split,CroatiaJune2014 60
 Cautionaboutmoderatoranalysis
Moderatorvariablesuseuppower
ToomanymoderatoranalysesleadtoTypeI
errors
Variablesareoftenconfoundedwitheach
other
Moderatoranalysesarealwaysobservational
theydontsupportcausalinferences

Split,CroatiaJune2014 61
Effectsizeindices

Split,CroatiaJune2014 62
 Indices
ContinuousData
Dichotomous(Binary)Data
Correlations
Others(E.g.prevalence,meaninonegroup)

Split,CroatiaJune2014 63
 Indicesformeans
RawmeandifferenceD
Standardizedmeandifferenced andg

Split,CroatiaJune2014 64
Rawmeandifference

D = X1 X 2

D = 550 500 = 50

Split,CroatiaJune2014 65
StandardizedMeanDifference

X1 X 2
d=
SWithin

550 500
d= = 0.50
100

Split,CroatiaJune2014 66
 Rawvs.StandardizedDifference

D d (org)

Scaleisnaturalorknown Required Not required

Allstudiesmustusesame scale Required Notrequired

Standarddeviationmust be Required Notrequired

consistent

Split,CroatiaJune2014 67
 Indicesforbinaryoutcomes
Definitionofrisk
Definitionofodds
RiskdifferenceRD
Riskratio(relativerisk)RR
OddsratioOR

Split,CroatiaJune2014 68
Split,CroatiaJune2014 69
 Meaningofrisk
Thenumberofpeoplewiththe
event/conditionweareinterestedin
comparedwiththetotalnumberofpeople
whocouldhavehadit.
ExamplefromCochrane
24peopledrankCoffee;6developedaheadache.
TheRisk ofdevelopingaheadache(givencoffee
drinking)is6/24or25%

Split,CroatiaJune2014 70
 MeaningofOdds
Thenumberofpeoplewith theeventof
interestcomparedwiththenumberwithout
theeventofinterest.
IntherunningCochraneexample,if24people
drankcoffeeand6developedaheadache
TheOddsofdevelopingaheadache(given
coffeedrinking)are6/18,or1in3.

Split,CroatiaJune2014 71
 RiskversusOdds
INGENERAL
Whenaneventisrare,therewontbetoo
muchdifferencebetweentheriskandthe
odds
Whenaneventiscommon,therecanbeaBIG
differencebetweentheriskandtheodds.

Split,CroatiaJune2014 72
 RiskDifference
Computetheriskoftheeventinthetreated
group
Computetheriskoftheeventinthecontrol
group
TheRiskdifferenceistheRiskinthetreated
groupMINUStheRiskinthecontrolgroup
ARiskdifferenceofZEROmeansthatthereis
nodifferencebetweenthegroups

Split,CroatiaJune2014 73
 RiskDifference

A C
RD =
n1 n2

5 10
RD = = 0.05
100 100

Split,CroatiaJune2014 74
 RiskRatio
Computetheriskinthetreatedgroup
Computetheriskinthecontrolgroup
TheRiskRatioistheriskinthetreatedgroup
dividedbytheriskinthecontrolgroup
Whentheriskratiois1,thatmeansthereis
NOdifferencebetweenthegroups
Whentheriskratioisabove1,itmeansthat
theriskishigherinthetreatedgroupthanin
thecontrolgroup.
Split,CroatiaJune2014 75
RiskRatio(RelativeRisk)

A / n1
RR =
C / n2

5 / 100
RR = = 0.50
10 / 100

Split,CroatiaJune2014 76
 Riskratio

5 / 100
= 0.50
10 / 100 0.50 + 2.00
= 1.25
2
10 / 100
= 2.00
5 / 100

Split,CroatiaJune2014 77
 LogRiskratio

ln(0.50) = 0.693
0.693 + 0.693
= 0.00
2
ln( 2.00) = +0.693

Split,CroatiaJune2014 78
Split,CroatiaJune2014 79
 Computetheoddsinthetreatedgroup
Computetheoddsinthecontrolgroup
TheOddsRatioistheoddsinthetreated
groupdividedbytheoddsinthecontrolgroup
Whentheoddsratiois1,thatmeansthereis
NOdifferencebetweenthegroups

Split,CroatiaJune2014 80
 OddsRatio

AD
OR =
BC

5 90
OR = = 0.47
95 10

Split,CroatiaJune2014 81
Split,CroatiaJune2014 82
Split,CroatiaJune2014 83
 Ifeverydiabeticnowtaking
Avandiawereinsteadgivena
similarpillnamedActos,about
500heartattacksand300
casesofheartfailurewouldbe
avertedeverymonth
Split,CroatiaJune2014 84
Split,CroatiaJune2014 85
 Inahighlyunusual
coordinatedannouncement,
drugregulatorsintheUnited
StatesandEuropesaid
ThursdaythatAvandiawould
nolongerbewidelyavailable

Split,CroatiaJune2014 86
Choiceofeffectsizeindex:Avandia
RRis1.43 yourchancesofdyingincreaseby
almost50%
RDis.005 theriskgoesupfromoneina
thousand,versus1perthousand.

Split,CroatiaJune2014 87
 RDvs.RRvs.OR
RD RR OR

Type of Absolute Relative Relative

difference
Intuitivefor Researchers Researchers Statisticians
Special Prob.offailure Prob offailure Prob.of
statisticalissues andprob of andofsuccess failureand
successare NOT prob of
reciprocal reciprocal,and successare
weightsvary reciprocal
bychoice
Metric for Raw Log Log
analyses 88
Split,CroatiaJune2014
 CochranelikestheRiskDifference
Itiseasytocompute
Itisanabsolutemeasureofactualchangein
risk
Itiseasytoconverttonaturalfrequenciesand
toNNT
HOWEVER:TheRiskDifferenceismore
variablethantheRRorOR acrossdifferent
populations,whenbaselineriskvaries.

Split,CroatiaJune2014 89
 Correlation
Usedwhenbothpredictorandoutcome
variablearecontinuous.
Usedmostlyinobservationalstudies.

Split,CroatiaJune2014 90
Split,CroatiaJune2014 91
 EffectSizesConventions
Usesandcautions

Split,CroatiaJune2014 92
 Effectsizes
ratherthanpvalues

Split,CroatiaJune2014 93
 pvalueandeffectsize
Approachesareconsistent

Split,CroatiaJune2014 94
 Example1
Onestudyp=.119
Onestudyp=.001

Whichstudyhadthelargereffectsize?

Split,CroatiaJune2014 95
Example1

Split,CroatiaJune2014 96
 Example2
Onestudyp=.012
Onestudyp=.012
Onestudyp=.012

Whichstudyhadthelargereffectsize?

Split,CroatiaJune2014 97
Example2

Split,CroatiaJune2014 98
 Example3
Onestudyp=.057
Onestudyp=.035

Whichstudyhadthelargereffectsize?

Split,CroatiaJune2014 99
Example3

Split,CroatiaJune2014 100
 Keypoint
AlwaysimportanttoworkwithES
Inmetaanalysis,especiallyso

Split,CroatiaJune2014 101
 Keypoint
pvalueispoorsurrogateforeffectsize
Inprimarystudies,shouldreportES
InmetaanalysesmustworkwithES

Split,CroatiaJune2014 102
 Threestudies,nonsignificant
Focusonpvalues,synthesizeconclusionsand
mightconcludeisnoevidenceofeffect
Focusoneffectsize,synthesizeeffects,asin
nextslide

Split,CroatiaJune2014 103
MetaAnalysis
withconsistenteffects

FIXEDEFFECTMETAANALYSIS

Split,CroatiaJune2014 104
 Samplingerror
whenstudiesshareacommoneffectsize

Split,CroatiaJune2014 105
 Weights
whenstudiesshareacommoneffectsize
W = 1/ (V1 )

Split,CroatiaJune2014 106
Split,CroatiaJune2014 107
Split,CroatiaJune2014 108
Split,CroatiaJune2014 109
Split,CroatiaJune2014 110
Precisionofthe
mean effect
Split,CroatiaJune2014 111
Split,CroatiaJune2014 112
Impactonprecision

Split,CroatiaJune2014 113
Split,CroatiaJune2014 114
Split,CroatiaJune2014 115
Split,CroatiaJune2014 116
Split,CroatiaJune2014 117
Split,CroatiaJune2014 118
Impactonpvalue

Split,CroatiaJune2014 119
Split,CroatiaJune2014 120
Split,CroatiaJune2014 121
Split,CroatiaJune2014 122
Split,CroatiaJune2014 123
Split,CroatiaJune2014 124
Split,CroatiaJune2014 125
Split,CroatiaJune2014 126
Heterogeneityineffectsizes

Split,CroatiaJune2014 127
WhatwemeanbyHeterogeneity

Itsthevarianceintrueeffects (not
observedeffects)thatwecareabout
Split,CroatiaJune2014 128
Whyestimateheterogeneity?

Split,CroatiaJune2014 129
Itaffectstheweights

Split,CroatiaJune2014 130
 WeightswhenT2=0
W = 1/ (V1 )

Split,CroatiaJune2014 131
 WeightswhenT2>0
(
W = 1/ V1 + T 2 )

Split,CroatiaJune2014 132
Itaffectsthemean

Split,CroatiaJune2014 133
 Itaffectsthestandarderror

theconfidenceinterval,andthepvalue

Split,CroatiaJune2014 134
Itaffectstheutilityofthetreatment

Isthetreatmenteffectiveforeveryone,or
effectiveforsomeandharmfulforothers?
Split,CroatiaJune2014 135
ImpactonStandardError

Split,CroatiaJune2014 136
Split,CroatiaJune2014 137
Split,CroatiaJune2014 138
Split,CroatiaJune2014 139
Split,CroatiaJune2014 140
Precisionofthe
mean effect
Split,CroatiaJune2014 141
Precisionofthe
mean effect
Split,CroatiaJune2014 142
Precisionofthe
mean effect
Split,CroatiaJune2014 143
 Impactonmean
Asheterogeneityincreases,meaneffectshifts
awayfromlargerstudiesandtowardssmaller
studies

Split,CroatiaJune2014 144
Split,CroatiaJune2014 145
Split,CroatiaJune2014 146
Impactonsubstantiveutility

Split,CroatiaJune2014 147
Split,CroatiaJune2014 148
Split,CroatiaJune2014 149
Split,CroatiaJune2014 150
Split,CroatiaJune2014 151
Split,CroatiaJune2014 152
Split,CroatiaJune2014 153
Split,CroatiaJune2014 154
 Dispersionofthe
individual effects
Split,CroatiaJune2014 155
 Dispersionofthe
individual effects
Split,CroatiaJune2014 156
Thatswhy weneedtoquantify

Split,CroatiaJune2014 157
How dowequantifyheterogeneity?

Split,CroatiaJune2014 158
 Twostepprocess
Isolatetherealdispersion
Translatethisintousefulindices

Split,CroatiaJune2014 159
Part1 Isolatetherealdispersion

Split,CroatiaJune2014 160
Whatwedliketoseeifthetrueeffectis
thesameinallstudies

Split,CroatiaJune2014 161
Whatwemightseeifthetrueeffectisthe
sameinallstudies

Split,CroatiaJune2014 162
Isthererealdispersion?

Split,CroatiaJune2014 163
Itdependsontheprecision

Split,CroatiaJune2014 164
Itdependsontheprecision

Split,CroatiaJune2014 165
 Keypoint
Wecaneasilycomputevarianceofobserved
effects
Butthisisduepartlytorealdifferencesin
effectsandpartlytosamplingerrorwithin
studies
Weneedtoisolatethebetweenstudies
variance

Split,CroatiaJune2014 166
Split,CroatiaJune2014 167
Split,CroatiaJune2014 168
Split,CroatiaJune2014 169
 Toassessheterogeneity
Computeobservedvariance
Estimatehowmuchvariancewouldbe
expectediftrueeffectisidenticalinallstudies
Observedminusexpectedisestimateoftrue
variance

Split,CroatiaJune2014 170
Isolatingtherealdispersion

Q df Qdf

Split,CroatiaJune2014 171
Indicesrelatedtoheterogeneity

Qdf Basis forallindices

p Testofnull
T Standard deviationoftrueeffects
T2 Variance oftrueeffects
I2 Proportion oftrue/totalvariance

Split,CroatiaJune2014 172
 Pvalue
Canweconcludethatthereissomevariance
intrueeffects
Dependsonamountofexcessvarianceand
theamountofevidence

Split,CroatiaJune2014 173
Split,CroatiaJune2014 174
Split,CroatiaJune2014 175
Split,CroatiaJune2014 176
Statisticsapplytobothmodels

Split,CroatiaJune2014 177
 Caution!
I2 isNOT ameasureofabsoluteheterogeneity
I2 tellsuswhatproportionoftheobserved
dispersionreflectsdifferencesintruescores
ratherthanrandomsamplingerror

Split,CroatiaJune2014 178
I2 Whatproportion
oftheobservedvarianceisreal?

Split,CroatiaJune2014 179
Fixedeffectvs.Randomeffects

Split,CroatiaJune2014 180
 Fixedeffectvs.Randomeffects
Fixedeffect
Samplingtakesplaceatonelevelonly
Anybetweenstudyvariancewillbeignoredwhen
assigningweights
Randomeffects
Samplingtakesplaceattwolevels
Anybetweenstudyvariancewillbeusedwhen
assigningweights

Split,CroatiaJune2014 181
 Fixedeffect
Whenthereisreasontobelievethatallthe
studiesarefunctionallyidentical
Whenourgoalistocomputethecommon
effectsize,forthestudiesintheanalysis
Exampleofdrugcompanyhasrunfivestudies
toassesstheeffectofadrug.

Split,CroatiaJune2014 182
 Randomeffects
Whennotlikelythatallthestudiesare
functionallyequivalent.
Whenthegoalofthisanalysisistogeneralize
toarangeofpopulations.
Exampleofstudiesculledfrompublications

Split,CroatiaJune2014 183
Split,CroatiaJune2014 184
Split,CroatiaJune2014 185
 Definitionofcombinedeffect
Fixedeffectmodel
Thereisonetrueeffect
Summaryeffectisestimateofthisvalue
Randomeffectsmodel
Thereisadistributionofeffects
Summaryeffectismeanofdistribution

Split,CroatiaJune2014 186
Howweightsareassigned

Split,CroatiaJune2014 187
 Fixedeffectmodel
W = 1/ (V1 )

Split,CroatiaJune2014 188
 Randomeffectsmodel
(
W = 1/ V1 + T 2 )

Split,CroatiaJune2014 189
 Howweightsshift
Ifwithinstudyvarianceonly,W=1/V
Ifbetweenstudyvarianceonly,W=1/T2
Ifboth,W=1/(V+T2)

Split,CroatiaJune2014 190
Split,CroatiaJune2014 191
Split,CroatiaJune2014 192
Randomvs.Fixed

Split,CroatiaJune2014 193
 Randomvs.Fixed

RE weights are more balanced

Split,CroatiaJune2014 194
 Randomvs.Fixed

RE confidence interval is wider

Split,CroatiaJune2014 195
LargestudyhaslessimpactunderRE

Split,CroatiaJune2014 196
SmallstudyhasmoreimpactunderRE

Split,CroatiaJune2014 197
FE
Split,CroatiaJune2014 198
 Summaryeffect

FE 101.171
= 0.414
244.215
Split,CroatiaJune2014 199
 Varianceofsummaryeffect

FE 1
= 0.004
244.215
Split,CroatiaJune2014 200
RE
Split,CroatiaJune2014 201
 Summaryeffect

RE 32.342
= 0.358
90.284
Split,CroatiaJune2014 202
 Varianceofsummaryeffect

RE 1
= 0.011
90.284
Split,CroatiaJune2014 203
 Whydoesitmatter?
Onematchesthesampling
Onedoesnot
Wrongmodelyieldsincorrectweights
Estimateofmeaniswrong
EstimateofCIiswrong

Split,CroatiaJune2014 204
 Fixedvs.Random
MUSTchoosebasedonsamplingmodel
ThemeaningoftheESisdifferent
RelativeweightsarecloserunderRE(effect
sizewillshift)
AbsoluteweightsaresmallerunderRE(CIwill
becomewider)
pvaluewillchange(lesssignificantinlongrun
butcangoeitherway)
Split,CroatiaJune2014 205
 Testofnull

M One
Fixed Z= source
SEM of error

*
M Two
Random Z* = sources
SEM * of error
Split,CroatiaJune2014 206
 Question!
Supposewehadfourstudies,eachwith
N=1,000,000,andatrue(mean)effectsizeof
0.50.Underthetwomodels,

Whatwouldtheforestplotlooklike?
Whatwouldthediamondlooklike?

Split,CroatiaJune2014 207
 2
SEM =
k n

Split,CroatiaJune2014 208
 2
T 2
SEM * = +
k n k
Split,CroatiaJune2014 209
Statisticalpower,Fixedeffect

d
Power = f
SEd

2
SEd =
k n

Split,CroatiaJune2014 210
Statisticalpower,Randomeffects

d
Power = f
SEd

2 2
SEd = +
kn k

Split,CroatiaJune2014 211
 Needtoknowthesourceofthe
variance
Ifonesource,thenerrorisV/n
Iftwosources,thenerrorisV/n+T2/k

Split,CroatiaJune2014 212
 Whichmodelshouldweuse?
Basedecisiononthemodelthatmatchedthe
waythedatawerecollected
Not ontestofhomogeneity

Split,CroatiaJune2014 213
Split,CroatiaJune2014 214
 Whatyoumayhear
Fixedeffectissimplemodel
Randomeffectsismorecomplicated

Split,CroatiaJune2014 215
 Actually
Fixedeffectismorerestrictedmodel
Randomeffectsmakeslessassumptions

Split,CroatiaJune2014 216
 Analternateview
Randomeffectsmodelonlymakessenseifwe
haveaclearpictureofthesamplingframe
Otherwise,weshouldreportthemeanandCI
forthestudiesinoursamplewithoutattempt
togeneralizetoalargeruniverse
Thisisafixedeffectsanalysis(intheplural)
wherefixedmeanssetratherthan
common

Split,CroatiaJune2014 217
Predictionintervals

Split,CroatiaJune2014 218
Precisionofthe Dispersionofthe
mean effect individual effects
Split,CroatiaJune2014 219
ConfidenceandPrediction

Split,CroatiaJune2014 220
 ConfidenceInterval
Measureofprecision
Rangeforthetruevalueofthemean
Analogoustostandarderror
Appliestofixedorrandomeffectsmodel

Split,CroatiaJune2014 221
 PredictionInterval
Measureofdispersion
Rangeforthetrueeffectindifferentstudies
Analogoustostandarddeviation

Split,CroatiaJune2014 222
ConfidenceIntervalvs.
PredictionInterval

Split,CroatiaJune2014 223