4/25/2017 Re:st:hausmanandxthausmanafterpanelfe,re

[DatePrev][DateNext][ThreadPrev][ThreadNext][Dateindex][Threadindex]

Re:st:hausmanandxthausmanafterpanel
fe,re
From"MarkSchaffer"<M.E.Schaffer@hw.ac.uk>
Tostatalist@hsphsun2.harvard.edu
SubjectRe:st:hausmanandxthausmanafterpanelfe,re
DateTue,23Aug200516:55:03+0100

AnexceedinglyrarecorrectiontoapostingbyVinceseebottom,
wheremynamecomesup.

From: vwiggins@stata.com(VinceWiggins,StataCorp)
To: statalist@hsphsun2.harvard.edu
Subject: Re:st:hausmanandxthausmanafterpanelfe,re
Datesent: Tue,23Aug200510:17:260500
Sendreplyto: statalist@hsphsun2.harvard.edu

>CarlNelson<chnelson3@insightbb.com>askswhyhegetsdifferentresultsfrom
>thehausmancommandandthedeprecatedxthausmancommand.
>
>>Thisquestionconcernsproblem10.9inJeffWooldridge'sbook
>>EconometricAnalysisofCrossSectionandPanelData.Inthis
>>exercise,whichIgavetosomestudentsinacoursethissummer,
>>usingCornwell.datstudentsareaskedtoestimatextreg,feand
>>xtreg,reandperformthehausmantest.Usingtheoldxthausman
>>syntaxtheresultisasignificantteststatistic(approximately121
>>forachisquared(11)rv).Usingthenewerhausmansyntaxtheresult
>>isanegativechisquaredstatisticandwarningaboutviolationof
>>assumptions.Iconstructedthestatisticfromthesavedresults
>>e(b)ande(V)andIgotthesameresultasthenewerhausmansyntax.
>>[...]
>
>Itisrarethathausmanandxthausmanproducedifferentstatistics,butI
>recommendthatCarlbelievetheresultsfromhausmanandnotxthausman.
>Themainreasonxthausmanwasundocumented(andnowworksonlyunderversion
>control)wasthatthatitcouldbefooledbynonpositivedefinite(PD)
>differencedcovariancematricesorbyvariableswithdegeneratepanel
>behavior.
>
>IpostedaratherlengthydiscussionoftheissuesbackinMarchof2002.
>Thispostpredatessomeofthestatalistarchives,soattheriskofbeing
>longwindedyetagain,letmequotefromthatposting.
>
>Beginexcerpts
>
>EricNeumayer<E.Neumayer@lse.ac.uk>askswhyheisgettingdifferentresults
http://www.stata.com/statalist/archive/200508/msg00762.html 1/6
4/25/2017 Re:st:hausmanandxthausmanafterpanelfe,re

>fromxthausandhausmanwhentestingforfixedvs.randomeffectsafter
>estimationwithxtreg.[...]
>
>IbelievethereareopenquestionsaboutHausmantestsinsituationslike
>Eric's,seetheexplanationthatfollows.
>
>
>Preliminaries
>
>
>ItishardtodiscusstheHausmantestwithoutbeingspecificabouthowthe
>testisperformed.LetBbetheparameterestimatesfromafullyefficient
>estimator(randomeffectsregressioninthiscase)andbbetheestimatesfrom
>alessefficientestimator(fixedeffectsregression),butonethatis
>consistentinthefaceofoneormoreviolatedassumptions,inthiscasethat
>theeffectsarecorrelatedwithoneormoreoftheregressors.Ifthe
>assumptionisviolatedthenweexpectthattheestimatesfromthetwo
>estimatorswillnotbethesame,b~=B.
>
>TheHausmantestisessentiallyaWaldtestthat(bB)==0forallcoefficients
>wherethecovariancematrixforbBistakenasthedifferenceofthe
>covariancematrices(VCEs)forbandB.Whatisamazingaboutthetestis
>thatwecanjustsubtractthesetwocovariancematricestogetanestimateof
>thecovariancematrixof(bB)withoutevenconsideringthattheVCEsofthe
>twoestimatorsmightbecorrelatedtheyareafterallestimatedonthesame
>data.Wecanjustsubtract,butonlybecausethetheVCEofthefully
>efficientestimatorisuncorrelatedwiththeVCEsofallotherestimators,see
>HausmanandTaylor(1981),"paneldataandunobservableindividualeffects",
>econometrica,49,13371398).TheVCEoftheefficientestimatorwillalsobe
>smallerthanthelessefficientestimator.Takentogether,theseresults
>implythatthesubtractionofthetwoVCE(V_bV_B)willbepositivedefinite
>(PD)andthatweneednotconsiderthecovariancebetweenthetwoVCEs.
>
>Theseresults,however,holdonlyasymptotically.Foranygivenfinitesample
>wehavenoreasontobelievethat(V_bV_B)willbePD.So,itisamazing
>thatwecanjustsubtractthesetwomatrices,butthepricewepayisthatwe
>canonlydososafelyifwehaveaninfiniteamountofdata.TheHausman
>test,unlikemosttests,reliesonasymptoticargumentsnotonlyforits
>distribution,butforitsabilitytobecomputed!Let'sdiscusswhatwedo
>whatwedowhen(V_bV_B)innotPDinthecontextofEric'sresults.
>
>Aside:IfanyoneisinterestedinaHausmanliketestthatdropsthe
>assumptionthateitherestimatorisfullyefficient,actuallyestimatesthe
>covariancebetweentheVCEs,andcanalwaysbecomputed,seeWeesie(2000)
>"Seeminglyunrelatedest.andclusteradjustedsandwichestimator",STB
>ReprintsVol9,pp231248.Thetestunfortunatelyrequiresthescoresfrom
>theestimator,andxtreg,fedoesnotdirectlyproducethese.
>
><Note,aversionofsuestcommandisnowofficial,butisstillunavailable
>afterxtreg>
>
>
>OfInversesandHausmanStatistics
>
>
>ThereasonthatxthausandhausmanproducedifferentstatisticsonEric's2/6
http://www.stata.com/statalist/archive/200508/msg00762.html
4/25/2017 Re:st:hausmanandxthausmanafterpanelfe,re

>ThereasonthatxthausandhausmanproducedifferentstatisticsonEric's
>modelsisthattheytakedifferentinversesofthisnonPDmatrix.xthaus
>usesStata'ssyminv()whichzerosoutcolumnsandrowstoformasubmatrix
>thatisPDandinvertsthatmatrix,whereashausmanusesaMoorePenrose
>generalizedinverse.MostoftheliteratureonHausmantestssuggeststhata
>generalizedinversesuchasMoorePenrosebeusedwhenthematrixisnotPD,
>however,Ihavenotseenafoundationofthissuggestion(andwould
>appreciationareferenceifanyoneknowsofone).
>
>TwoofusatStatahaveindependentlyrunsomeinformalsimulations,where
>nonPDmatricesarecommon,todetermineifeitheroftheseinverseshas
>nominalcoverageforatruenull.Whilethesesimulationsarenotcomplete
>enoughtoshareorpublish,webothfoundthatneitherinverseperformswell.
>Thisdoesn'tseemtoosurprisingtome,iftheinformationinoursampleis
>insufficienttoproduceaPD"VCE"thenthebasisofthetestwouldseemtobe
>inquestion.
>
>xthausdoesnotmakeitclearwhenthematrixisnotPD.Irecallhaving
>read,thoughIcannotnowfindthereference,thatinthecaseofrandomvs.
>fixedeffectsthatthematrixwaseitheralwaysPD.Thismayhavebeenthe
>thinkinginexcludingthischeckfromxthausman.Regardless,itisclearly
>notimpossibleandisnotevenunlikely.SimulationsshowthatnonPD
>matricesarequitecommon.
>
>
>AnAlternative
>
>
>Evenintheirearlywork,HausmanandTaylor(1981)discussanasymptotically
>equivalenttestforrandomvs.fixedeffectsusinganaugmentedregression.
>Thereareactuallyseveralformsoftheaugmentedregression,allofwhichare
>asymptoticallyequivalenttotheHausmantest.Alloftheseaugmented
>regressiontestsarebasedonestimatinganaugmentedregressionthatnests
>boththerandomandfixedeffectsmodels.Theyareparameterizedinsucha
>waythatwecanperformasimpleWaldtestofasetofthejointlyestimated
>coefficients.Theyhavefewerofthemechanicalandinterpretationproblems
>associatedwiththeHausmantest.Theirresultswilldiffernumericallyfrom
>theHausmantestinfinitesamplesbecausetheyareonlyasymptotically
>equivalent.
>
>Ihaveincludebelowablockofcodethatwillperformanaugmentedregression
>testforEric'smodel(italsoperformstheHausmantestusingxthausand
>hausman).Itcaneasilybeadaptedtoanymodelbychangingthedepvarand
>varlistmacros.
>
>IfIhavegiventheimpressionthatIdon'tmuchcarefortheHausmantest,
>good.Idon't.InadhocsimulationsIhavefoundthatinadditiontoits
>proclivitytobeuncomputable,thetesthaslowpowerforthecurrentproblem,
>fortestsofendogeneityininstrumentalvariablesregression,andfortests
>ofindependenceofirrelevantalternatives(IIA)inchoicemodels.
>
>Regardless,thetestisastapleineconometricsanditwillstayinStata.
>
>
><Note:Carlshouldbeabletoeasilyadaptthiscodebyspecifyingtheid
>variable,dependentvariable,andvarlist.>
http://www.stata.com/statalist/archive/200508/msg00762.html 3/6
4/25/2017 Re:st:hausmanandxthausmanafterpanelfe,re

>variable,dependentvariable,andvarlist.>
>
>BEGINforeric.doCUTHERE
>localidmyid
>localdepvarlnuncs
>localvarlistlngdpecriseecfallurbanlnhousefemalepamale1544/*
> */lndiscrlnfreelnptslatinssadeathprulelawprotestcathol/*
> */muslimtransitilnethvoecdwaryear89year92year95
>
>xtreg`depvar'`varlist',re
>hausman,save
>version7:xthausman
>
>xtreg`depvar'`varlist',fe
>hausman,less
>
>tokenize`varlist'
>locali1
>while"``i''"!=""{
> quiby`id':gendoublemean`i'=sum(``i'')/_n
> quiby`id':replacemean`i'=mean`i'[_N]
> quiby`id':gendoublediff`i'=``i''mean`i'
> localnewlist`newlist'mean`i'diff`i'
>
> locali=`i'+1
>}
>
>xtreg`depvar'`newlist',re
>tempnameb
>matrix`b'=e(b)
>
>quitestmean1=diff1,notest /*cleartest*/
>locali2
>while"``i''"!=""{
> if`b'[1,colnumb(`b',"mean`i'")]!=0& /*
> */`b'[1,colnumb(`b',"diff`i'")]!=0{
> quitestmean`i'=diff`i',accumnotest
> }
> locali=`i'+1
>}
>test
>
>ENDforeric.doCUTHERE
>
>Endexcerpts
>
>Asnotedintheexcerpt,Whenxthausmanwaswrittenwewereswayedby
>published"proofs"thatthedifferencematrixwasrequiredmathematicallyto
>bepositivedefinitewhencomparingFEandRElinearregression.AsEric's
>andCarl'sexamplesshow,thisisnottrue.IwouldliketothankMark
>Schaffer<M.E.Schaffer@hw.ac.uk>forremindingmeofoneofthe"proofs",
>
>
>"ThisappendixprovesthattheAvar(q_hat)in(5.2.21)is
>positivedefiniteandtheHausmanstatistic(5.2.22)is
>guaranteedtobenonnegativeinanyfinitesamples."
http://www.stata.com/statalist/archive/200508/msg00762.html 4/6
4/25/2017 Re:st:hausmanandxthausmanafterpanelfe,re

>guaranteedtobenonnegativeinanyfinitesamples."
>
>Hayashi,Econometrics(2000),Appendix5.A,pp.346349and334335.

Theproofiscorrect.Theissueisthatitdependsontheuseofa
singleestimateoftheerrorvarianceforboththerandomeffectsVCE
andthefixedeffectsVCE.Asymptotically,itdoesn'tmatterwhich
oneisused,andeitherwillguaranteeapositiveteststatistic.
Whatthismeansinpracticeisthat,say,therandomeffectsVCEis
multipliedbys2_fe/s2_re,sothatitincorporatesthefixedeffects
s2insteadoftheoriginalrandomeffects2,beforetheHausmantest
isapplied.

Infact,ifthepanelisbalanced,usingthefixedeffectsestimate
oftheerrorvarianceinaHausmantestisnumericallyequivalentto
theartificialregressionversionofthetestwhengroupmeansare
addedasregressors.

ThesituationisverysimilartoanIVendogeneitytest(discussedin
theStatamanuals,ifIrecall).

Theoldxthausmandidn'tuseasingleestimateoftheerrorvariance
theuntransformedVCEswereused,andthiscouldsometimesgenerate
negativeteststats.Thesameappliedtothehausmancommandasof
version8.2.Theproblemwasthatthesigmamoreorsigmaless
hausmanoptions,whichforcetheusageofasingleerrorvariancein
ahausmantest,wouldn'tworkwhentheestimationresultsweredone
byxtreg.I*still*don'thaveStata9sighandcan'tcheckif
thishasbeenrectified.

Inanycase,theartificialregressionapproachwillalwayswork.

Cheers,
Mark

>Toavoidbreakinguser'sdofiles,wewerereluctanttoremovexthausman
>whenhausmanwasfirstintroduced.Sufficienttimehaspassed,andasof
>version9ofStata,xthausmanworksonlywhenyourversionissetto8or
>lower.
>
>
>Vince
>vwiggins@stata.com
>
>*
>*Forsearchesandhelptry:
>*http://www.stata.com/support/faqs/res/findit.html
>*http://www.stata.com/support/statalist/faq
>*http://www.ats.ucla.edu/stat/stata/

Prof.MarkE.Schaffer
Director
CentreforEconomicReformandTransformation
DepartmentofEconomics
SchoolofManagement&Languages
HeriotWattUniversity,EdinburghEH144ASUK
http://www.stata.com/statalist/archive/200508/msg00762.html 5/6
4/25/2017 Re:st:hausmanandxthausmanafterpanelfe,re

HeriotWattUniversity,EdinburghEH144ASUK
441314513494direct
441314513296fax
441314513485CERTadministrator
http://www.sml.hw.ac.uk/cert

*
*Forsearchesandhelptry:
*http://www.stata.com/support/faqs/res/findit.html
*http://www.stata.com/support/statalist/faq
*http://www.ats.ucla.edu/stat/stata/

References:
Re:st:hausmanandxthausmanafterpanelfe,re
From:vwiggins@stata.com(VinceWiggins,StataCorp)

PrevbyDate:Re:st:Howtofindamacrocontainingagivenstring
NextbyDate:st:estoutafteradjust
Previousbythread:Re:st:hausmanandxthausmanafterpanelfe,re
Nextbythread:Re:st:hausmanandxthausmanafterpanelfe,re
Index(es):
Date
Thread

Copyright19962017StataCorpLLC|Termsofuse|Privacy|Contactus|What'snew|Siteindex

http://www.stata.com/statalist/archive/200508/msg00762.html 6/6