Subject: [QuantStart]Lesson3:SuccessfulBacktestingofAlgorithmicTradingStrategies(PartI)

From:

MichaelHallsMoore(mike@quantstart.com)

To:

darekstepien@ymail.com

Date:

Sunday,4January2015,6:18

HiDarek,
Inthesecondlessonofourquantitativefinanceemailcoursewediscussedhowtoidentify
algorithmictradingstrategiesand,inparticular,createastrategypipeline.
Intoday'slessonwe'regoingtodigdeeperintobacktestingandconsideraspectsthatareoften
ignoredbymuchofthealgorithmictradingliterature.
Algorithmicbacktestingrequiresknowledgeofmanyareas,includingpsychology,mathematics,
statistics,softwaredevelopmentandmarket/exchangemicrostructure.
Icouldn'thopetocoverallofthosetopicsinoneemail,soI'mgoingtosplitthemintotwoorthree
smallerpieces.Whatwillwediscussinthissection?I'llbeginbydefiningbacktestingandthenIwill
describethebasicsofhowitiscarriedout.
ThenIwillelucidateuponthebiaseswetoucheduponinthefirstemail(Beginner'sGuideto
QuantitativeTrading).NextIwillpresentacomparisonofthevariousavailablebacktestingsoftware
options.
Insubsequentemailswewilllookatthedetailsofstrategyimplementationsthatareoftenbarely
mentionedorignored.
Wewillalsoconsiderhowtomakethebacktestingprocessmorerealisticbyincludingthe
idiosyncrasiesofatradingexchange.Thenwewilldiscusstransactioncostsandhowtocorrectly
modeltheminabacktestsetting.
Let'sbeginbydiscussingwhatbacktestingisandwhyweshouldcarryitoutinouralgorithmic
trading.

WhatisBacktesting?
Algorithmictradingstandsapartfromothertypesofinvestmentclassesbecausewecanmorereliably
provideexpectationsaboutfutureperformancefrompastperformance,asaconsequenceof
abundantdataavailability.Theprocessbywhichthisiscarriedoutisknownasbacktesting.
Insimpleterms,backtestingiscarriedoutbyexposingyourparticularstrategyalgorithmtoastream
ofhistoricalfinancialdata,whichleadstoasetoftradingsignals.
Eachtrade,whichwewillmeanheretobea'roundtrip'oftwosignals,willhaveanassociatedprofit
orloss.Theaccumulationofthisprofit/lossoverthedurationofyourstrategybacktestwillleadtothe
totalprofitandloss(alsoknownasthe'P&L'or'PnL').Thatistheessenceoftheidea,althoughof
coursethe"devilisalwaysinthedetails"!
Whatarekeyreasonsforbacktestinganalgorithmicstrategy?
FiltrationIfyourecallfromthepreviousemailStrategyIdentification,ourgoalattheinitial
researchstagewastosetupastrategypipelineandthenfilteroutanystrategythatdidnot
meetcertaincriteria.Backtestingprovidesuswithanotherfiltrationmechanism,aswecan
eliminatestrategiesthatdonotmeetourperformanceneeds.
ModellingBacktestingallowsusto(safely!)testnewmodelsofcertainmarketphenomena,
suchastransactioncosts,orderrouting,latency,liquidityorothermarketmicrostructure
issues.
OptimisationAlthoughstrategyoptimisationisfraughtwithbiases,backtestingallowsusto
increasetheperformanceofastrategybymodifyingthequantityorvaluesoftheparameters
associatedwiththatstrategyandrecalculatingitsperformance.

VerificationOurstrategiesareoftensourcedexternally,viaourstrategypipeline.
Backtestingastrategyensuresthatithasnotbeenincorrectlyimplemented.Althoughwewill
rarelyhaveaccesstothesignalsgeneratedbyexternalstrategies,wewilloftenhaveaccess
totheperformancemetricssuchastheSharpeRatioandDrawdowncharacteristics.Thuswe
cancomparethemwithourownimplementation.
Backtestingprovidesahostofadvantagesforalgorithmictrading.However,itisnotalways
possibletostraightforwardlybacktestastrategy.
Ingeneral,asthefrequencyofthestrategyincreases,itbecomeshardertocorrectlymodelthe
microstructureeffectsofthemarketandexchanges.
Thisleadstolessreliablebacktestsandthusatrickierevaluationofachosenstrategy.Thisisa
particularproblemwheretheexecutionsystemisthekeytothestrategyperformance,aswithultra
highfrequencyalgorithms.
Unfortunately,backtestingisfraughtwithbiasesofalltypes.Wehavetoucheduponsomeofthese
issuesinpreviousemails,butwewillnowdiscussthemindepth.

BiasesAffectingStrategyBacktests
Therearemanybiasesthatcanaffecttheperformanceofabacktestedstrategy.
Unfortunately,thesebiaseshaveatendencytoinflatetheperformanceratherthandetractfromit.
Thusyoushouldalwaysconsiderabacktesttobeanidealisedupperboundontheactual
performanceofthestrategy.Itisalmostimpossibletoeliminatebiasesfromalgorithmictradingsoitis
ourjobtominimisethemasbestwecaninordertomakeinformeddecisionsaboutouralgorithmic
strategies.
TherearefourmajorbiasesthatIwishtodiscuss:OptimisationBias,LookAheadBias,Survivorship
BiasandPsychologicalToleranceBias.

OptimisationBias
Thisisprobablythemostinsidiousofallbacktestbiases.
Itinvolvesadjustingorintroducingadditionaltradingparametersuntilthestrategyperformanceon
thebacktestdatasetisveryattractive.However,oncelivetheperformanceofthestrategycanbe
markedlydifferent.Anothernameforthisbiasis"curvefitting"or"datasnoopingbias".
Optimisationbiasishardtoeliminateasalgorithmicstrategiesofteninvolvemanyparameters.
"Parameters"inthisinstancemightbetheentry/exitcriteria,lookbackperiods,averagingperiods(i.e
themovingaveragesmoothingparameter)orvolatilitymeasurementfrequency.
Optimisationbiascanbeminimisedbykeepingthenumberofparameterstoaminimumand
increasingthequantityofdatapointsinthetrainingset.
Infact,onemustalsobecarefulofthelatterasoldertrainingpointscanbesubjecttoapriorregime
(suchasaregulatoryenvironment)andthusmaynotberelevanttoyourcurrentstrategy.
Onemethodtohelpmitigatethisbiasistoperformasensitivityanalysis.Thismeansvaryingthe
parametersincrementallyandplottinga"surface"ofperformance.
Sound,fundamentalreasoningforparameterchoicesshould,withallotherfactorsconsidered,lead
toasmootherparametersurface.
Ifyouhaveaveryjumpyperformancesurface,itoftenmeansthataparameterisnotreflectinga
phenomenaandisanartefactofthetestdata.
Thereisavastliteratureonmultidimensionaloptimisationalgorithmsanditisahighlyactiveareaof
research.Iwon'tdwellonithere,butkeepitinthebackofyourmindwhenyoufindastrategywitha
fantasticbacktest!

LookAheadBias
Lookaheadbiasisintroducedintoabacktestingsystemwhenfuturedataisaccidentallyincluded
atapointinthesimulationwherethatdatawouldnothaveactuallybeenavailable.
IfwearerunningthebacktestchronologicallyandwereachtimepointN,thenlookaheadbias
occursifdataisincludedforanypointN+k,wherek>0.Lookaheadbiaserrorscanbeincredibly
subtle.Herearethreeexamplesofhowlookaheadbiascanbeintroduced:

TechnicalBugsArrays/vectorsincodeoftenhaveiteratorsorindexvariables.Incorrect
offsetsoftheseindicescanleadtoalookaheadbiasbyincorporatingdataatN+kfornon
zerok.
ParameterCalculationAnothercommonexampleoflookaheadbiasoccurswhen
calculatingoptimalstrategyparameters,suchaswithlinearregressionsbetweentwotime
series.Ifthewholedataset(includingfuturedata)isusedtocalculatetheregression
coefficients,andthusretroactivelyappliedtoatradingstrategyforoptimisationpurposes,then
futuredataisbeingincorporatedandalookaheadbiasexists.
Maxima/MinimaCertaintradingstrategiesmakeuseofextremevaluesinanytimeperiod,
suchasincorporatingthehighorlowpricesinOHLCdata.However,sincethese
maximal/minimalvaluescanonlybecalculatedattheendofatimeperiod,alookaheadbias
isintroducedifthesevaluesareusedduringthecurrentperiod.Itisalwaysnecessarytolag
high/lowvaluesbyatleastoneperiodinanytradingstrategymakinguseofthem.
Aswithoptimisationbias,onemustbeextremelycarefultoavoiditsintroduction.Itisoftenthemain
reasonwhytradingstrategiesunderperformtheirbacktestssignificantlyin"livetrading".

SurvivorshipBias
Survivorshipbiasisaparticularlydangerousphenomenonandcanleadtosignificantlyinflated
performanceforcertainstrategytypes.
Itoccurswhenstrategiesaretestedondatasetsthatdonotincludethefulluniverseofpriorassets
thatmayhavebeenchosenataparticularpointintime,butonlyconsiderthosethathave"survived"
tothecurrenttime.
Asanexample,considertestingastrategyonarandomselectionofequitiesbeforeandafterthe
2001marketcrash.Sometechnologystockswentbankrupt,whileothersmanagedtostayafloatand
evenprospered.
Ifwehadrestrictedthisstrategyonlytostockswhichmadeitthroughthemarketdrawdownperiod,
wewouldbeintroducingasurvivorshipbiasbecausetheyhavealreadydemonstratedtheirsuccess
tous.
Infact,thisisjustanotherspecificcaseoflookaheadbias,asfutureinformationisbeingincorporated
intopastanalysis.
Therearetwomainwaystomitigatesurvivorshipbiasinyourstrategybacktests:
SurvivorshipBiasFreeDatasetsInthecaseofequitydataitispossibletopurchase
datasetsthatincludedelistedentities,althoughtheyarenotcheapandonlytendtobeutilised
byinstitutionalfirms.Inparticular,YahooFinancedataisNOTsurvivorshipbiasfree,andthis
iscommonlyusedbymanyretailalgotraders.Onecanalsotradeonassetclassesthatare
notpronetosurvivorshipbias,suchascertaincommodities(andtheirfuturederivatives).
UseMoreRecentDataInthecaseofequities,utilisingamorerecentdatasetmitigatesthe
possibilitythatthestockselectionchosenisweightedto"survivors",simplyasthereisless
likelihoodofoverallstockdelistinginshortertimeperiods.Onecanalsostartbuildinga
personalsurvivorshipbiasfreedatasetbycollectingdatafromcurrentpointonward.After34
years,youwillhaveasolidsurvivorshipbiasfreesetofequitiesdatawithwhichtobacktest
furtherstrategies.
Wewillnowconsidercertainpsychologicalphenomenathatcaninfluenceyourtradingperformance.

PsychologicalToleranceBias
Thisparticularphenomenaisnotoftendiscussedinthecontextofquantitativetrading.However,itis
discussedextensivelyinregardtomorediscretionarytradingmethods.
Ithasvariousnames,butI'vedecidedtocallit"psychologicaltolerancebias"becauseitcapturesthe
essenceoftheproblem.
Whencreatingbacktestsoveraperiodof5yearsormore,itiseasytolookatanupwardlytrending
equitycurve,calculatethecompoundedannualreturn,Sharperatioandevendrawdown
characteristicsandbesatisfiedwiththeresults.
Asanexample,thestrategymightpossessamaximumrelativedrawdownof25%andamaximum
drawdowndurationof4months.Thiswouldnotbeatypicalforamomentumstrategy.
Itisstraightforwardtoconvinceoneselfthatitiseasytotoleratesuchperiodsoflossesbecausethe
overallpictureisrosy.However,inpractice,itisfarharder!

Ifhistoricaldrawdownsof25%ormoreoccurinthebacktests,theninalllikelihoodyouwillsee
periodsofsimilardrawdowninlivetrading.
Theseperiodsofdrawdownarepsychologicallydifficulttoendure.Ihaveobservedfirsthandwhatan
extendeddrawdowncanbelike,inaninstitutionalsetting,anditisnotpleasantevenifthe
backtestssuggestsuchperiodswilloccur.
ThereasonIhavetermedita"bias"isthatoftenastrategywhichwouldotherwisebesuccessfulis
stoppedfromtradingduringtimesofextendeddrawdownandthuswillleadtosignificant
underperformancecomparedtoabacktest.
Thus,eventhoughthestrategyisalgorithmicinnature,psychologicalfactorscanstillhaveaheavy
influenceonprofitability.
Thetakeawayistoensurethatifyouseedrawdownsofacertainpercentageanddurationinthe
backtests,thenyoushouldexpectthemtooccurinlivetradingenvironments,andwillneedto
persevereinordertoreachprofitabilityoncemore.

SoftwarePackagesforBacktesting
Thesoftwarelandscapeforstrategybacktestingisvast.
Solutionsrangefromfullyintegratedinstitutionalgradesophisticatedsoftwarethroughto
programminglanguagessuchasC++,PythonandRwherenearlyeverythingmustbewrittenfrom
scratch(orsuitable'plugins'obtained).
Asquanttraders,weareinterestedinthebalanceofbeingableto"own"ourtradingtechnologystack
versusthespeedandreliabilityofourdevelopmentmethodology.Herearethekeyconsiderationsfor
softwarechoice:
ProgrammingSkillThechoiceofenvironmentwillinalargepartcomedowntoyourability
toprogramsoftware.Iwouldarguethatbeingincontrolofthetotalstackwillhaveagreater
effectonyourlongtermP&Lthanoutsourcingasmuchaspossibletovendorsoftware.Thisis
duetothedownsideriskofhavingexternalbugsoridiosyncrasiesthatyouareunabletofixin
vendorsoftware,whichwouldotherwisebeeasilyremediedifyouhadmorecontroloveryour
"techstack".Youalsowantanenvironmentthatstrikestherightbalancebetweenproductivity,
libraryavailabilityandspeedofexecution.Imakemyownpersonalrecommendationbelow.
ExecutionCapability/BrokerInteractionCertainbacktestingsoftware,suchas
TradeStation,tiesindirectlywithabrokerage.Iamnotafanofthisapproachasreducing
transactioncostsareoftenabigcomponentofgettingahigherSharperatio.Ifyou'retiedinto
aparticularbroker(andTradestation"forces"youtodothis),thenyouwillhaveahardertime
transitioningtonewsoftware(oranewbroker)iftheneedarises.InteractiveBrokersprovide
anAPIwhichisrobust,albeitwithaslightlyobtuseinterface.
CustomisationAnenvironmentlikeMATLABorPythongivesyouagreatdealofflexibility
whencreatingalgostrategiesastheyprovidefantasticlibrariesfornearlyanymathematical
operationimaginable,butalsoallowextensivecustomisationwherenecessary.
StrategyComplexityCertainsoftwarejustisn'tcutoutforheavynumbercrunchingor
mathematicalcomplexity.Excelisonesuchpieceofsoftware.Whileitisgoodforsimpler
strategies,itcannotreallycopewithnumerousassetsormorecomplicatedalgorithms,at
speed.
BiasMinimisationDoesaparticularpieceofsoftwareordatalenditselfmoretotrading
biases?Youneedtomakesurethatifyouwanttocreateallthefunctionalityyourself,thatyou
don'tintroducebugswhichcanleadtobiases.
SpeedofDevelopmentOneshouldn'thavetospendmonthsandmonthsimplementinga
backtestengine.Prototypingshouldonlytakeafewweeks.Makesurethatyoursoftwareis
nothinderingyourprogresstoanygreatextent,justtograbafewextrapercentagepointsof
executionspeed.C++isthe"elephantintheroom"here!
SpeedofExecutionIfyourstrategyiscompletelydependentuponexecutiontimeliness(as
inHFT/UHFT)thenalanguagesuchasCorC++willbenecessary.However,youwillbe
vergingonLinuxkerneloptimisationandFPGAusageforthesedomains,whichisoutsidethe
scopeofthisemail!
CostManyofthesoftwareenvironmentsthatyoucanprogramalgorithmictradingstrategies
witharecompletelyfreeandopensource.Infact,manyhedgefundsmakeuseofopen
sourcesoftwarefortheirentirealgotradingstacks.Inaddition,ExcelandMATLABareboth
relativelycheapandthereareevenfreealternativestoeach.

Nowthatwehavelistedthecriteriawithwhichweneedtochooseoursoftwareinfrastructure,Iwant
torunthroughsomeofthemorepopularpackagesandhowtheycompare:
Note:Iamonlygoingtoincludesoftwarethatisavailabletomostretailpractitionersandsoftware
developers,asthisisthereadershipofthesiteandtheemaillist.Whileothersoftwareisavailable,
suchasthemoreinstitutionalgradetools,Ifeelthesearetooexpensivetobeeffectivelyusedina
retailsettingandIpersonallyhavenoextensiveexperiencewiththem.

MS
Excel

BacktestingSoftwareComparison
Description:WYSIWYG(whatyouseeiswhatyouget)
spreadsheetsoftware.Extremelywidespreadinthefinancial
industry.Dataandalgorithmaretightlycoupled.
Execution:Yes,Excelcanbetiedintomostbrokerages.
Customisation:VBAmacrosallowmoreadvancedfunctionalityat
theexpenseofhidingimplementation.
StrategyComplexity:Moreadvancedstatisticaltoolsareharder
toimplementasarestrategieswithmanyhundredsofassets.
BiasMinimisation:Lookaheadbiasiseasytodetectviacell
highlightingfunctionality(assumingnoVBA).
DevelopmentSpeed:Quicktoimplementbasicstrategies.
ExecutionSpeed:Slowexecutionspeedsuitableonlyforlower
frequencystrategies.
Cost:Cheaporfree(dependinguponlicense).
Alternatives:OpenOffice

Description:Programmingenvironmentoriginallydesignedfor
computationalmathematics,physicsandengineering.Verywell
suitedtovectorisedoperationsandthoseinvolvingnumericallinear
algebra.Providesawidearrayofpluginsforquanttrading.In
widespreaduseinquantitativehedgefunds.
Execution:Nonativeexecutioncapability,MATLABrequiresa
separateexecutionsystem.
Customisation:Hugearrayofcommunitypluginsfornearlyall
areasofcomputationalmathematics.
MATLAB StrategyComplexity:Manyadvancedstatisticalmethodsalready
availableandwelltested.
BiasMinimisation:Hardertodetectlookaheadbias,requires
extensivetesting.
DevelopmentSpeed:Shortscriptscancreatesophisticated
backtestseasily.
ExecutionSpeed:Assumingavectorised/parallelisedalgorithm,
MATLABishighlyoptimised.Poorfortraditionaliteratedloops.
Cost:~1,000USDforalicense.
Alternatives:Octave,SciLab
Description:Highlevellanguagedesignedforspeedof
development.Widearrayoflibrariesfornearlyanyprogrammatic
taskimaginable.Gainingwideracceptanceinhedgefundand
investmentbankcommunity.NotquiteasfastasC/C++for
executionspeed.
Execution:Pythonpluginsexistforlargerbrokers,suchas
InteractiveBrokers.Hencebacktestandexecutionsystemcanall
bepartofthesame"techstack".
Customisation:Pythonhasaveryhealthydevelopment
communityandisamaturelanguage.NumPy/SciPyprovidefast

Python

C++

scientificcomputingandstatisticalanalysistoolsrelevantforquant
trading.
StrategyComplexity:Manypluginsexistforthemainalgorithms,
butnotquiteasbigaquantcommunityasexistsforMATLAB.
BiasMinimisation:Samebiasminimisationproblemsexistasfor
anyhighlevellanguage.Needtobeextremelycarefulabout
testing.
DevelopmentSpeed:Python'smainadvantageisdevelopment
speed,withrobustinbuiltintestingcapabilities.
ExecutionSpeed:NotquiteasfastasC++,butscientific
computingcomponentsareoptimisedandPythoncantalktonative
Ccodewithcertainplugins.
Cost:Free/OpenSource
Alternatives:Ruby,Erlang,Haskell
Description:Environmentdesignedforadvancedstatistical
methodsandtimeseriesanalysis.Widearrayofspecificstatistical,
econometricandnativegraphingtoolsets.Largedeveloper
community.
Execution:Rpossessespluginstosomebrokers,inparticular
InteractiveBrokers.Thusanendtoendsystemcanwrittenentirely
inR.
Customisation:Rcanbecustomisedwithanypackage,butits
strengthslieinstatistical/econometricdomains.
StrategyComplexity:Mostlyusefulifperformingeconometric,
statisticalormachinelearningstrategiesduetoavailableplugins.
BiasMinimisation:Similarlevelofbiaspossibilityforanyhigh
levellanguagesuchasPythonorC++.Thustestingmustbe
carriedout.
DevelopmentSpeed:Risrapidforwritingstrategiesbasedon
statisticalmethods.
ExecutionSpeed:RisslowerthanC++,butremainsrelatively
optimisedforvectorisedoperations(aswithMATLAB).
Cost:Free/OpenSource
Alternatives:Stata
Description:Mature,highlevellanguagedesignedforspeedof
execution.Widearrayofquantitativefinanceandnumerical
libraries.Hardertodebugandoftentakeslongertoimplementthan
PythonorMATLAB.Extremelyprevalentinboththebuyandsell
side.
Execution:MostbrokerageAPIsarewritteninC++andJava.
Thusmanypluginsexist.
Customisation:C/C++allowsdirectaccesstounderlyingmemory,
henceultrahighfrequencystrategiescanbeimplemented.
StrategyComplexity:C++STLprovideswidearrayofoptimised
algorithms.Nearlyanyspecialisedmathematicalalgorithm
possessesafree,opensourceC/C++implementationontheweb.
BiasMinimisation:Lookaheadbiascanbetrickytoeliminate,but
noharderthanotherhighlevellanguage.Gooddebuggingtools,
butonemustbecarefulwhendealingwithunderlyingmemory.
DevelopmentSpeed:C++isquiteverbosecomparedtoPythonor
MATLABforthesamealgorithmm.Morelinesofcode(LOC)often

leadstogreaterlikelihoodofbugs.
ExecutionSpeed:C/C++hasextremelyfastexecutionspeedand
canbewelloptimisedforspecificcomputationalarchitectures.This
isthemainreasontoutiliseit.
Cost:Variouscompilers:Linux/GCCisfree,MSVisualStudiohas
differinglicenses.
Alternatives:C#,Java,Scala
Differentstrategieswillrequiredifferentsoftwarepackages.HFTandUHFTstrategieswillbe
writteninC/C++(thesedaystheyareoftencarriedoutonGPUsandFPGAs),whereaslowfrequency
directionalequitystrategiesareeasytoimplementinTradeStation,duetothe"allinone"natureof
thesoftware/brokerage.
MypersonalpreferenceisforPythonasitprovidestherightdegreeofcustomisation,speedof
development,testingcapabilityandexecutionspeedformyneedsandstrategies.
IfIneedanythingfaster,Ican"dropin"toC++directlyfrommyPythonprograms.Onemethod
favouredbymanyquanttradersistoprototypetheirstrategiesinPythonandthenconverttheslower
executionsectionstoC++inaniterativemanner.EventuallytheentirealgoiswritteninC++andcan
be"leftalonetotrade"!
Inthenextemailwewilltakeanextensivelookattransactioncostmodellingaswellasstrategy
specificimplementation.
Thanksforreading,
Mike.

QuarkGluonLtd,Flat481,ImperialCourt,225KenningtonLane,London,GreaterLondonSE115QN,UNITED
KINGDOM
Unsubscribe|ChangeSubscriberOptions