COLLEGETESTPREPTAKESATEST

AReviewofTenOnlineSATTestPreparationServices

byWarrenBuckleitner,Ph.D. MediatechFoundation May1,2006

ConsumerReportsWebWatch 101TrumanAvenue Yonkers,NY107031057 Tel.:914.378.2600 www.ConsumerWebWatch.org

TABLEOFCONTENTS
EXECUTIVESUMMARY________________________________________________ 3
KEYFINDINGS_________________________________________________________________ 5 CREDITS_______________________________________________________________________ 6

ABOUTCONSUMERREPORTSWEBWATCH _____________________________ 7 INTRODUCTIONANDMETHODOLOGY _________________________________ 8

ANewWaytoPrep:Online ________________________________________________________ 8

METHODOLOGY__________________________________________________________ 9
SelectionofOnlineTestPrepServices ________________________________________________ 9 SelectionofStudentTesters________________________________________________________ 10 TheTestingProcess ______________________________________________________________ 11 DataCollection _________________________________________________________________ 11

MAJORFINDINGSANDCONCERNS ___________________________________ 13
OnlineServicesComewithAggressiveAdvertising_____________________________________ 13 HigherPriceandReputationDontNecessarilyMakeforBetterProducts ____________________ 14 StudentsPrivacyProtectionQuestioned______________________________________________ 14 SiteFunctionalityInfluencesPerceptionof Trust_______________________________________ 15 InteractivityDistinguishesOnlineServices,NeedsImprovement___________________________ 15 ErrorsAbound __________________________________________________________________ 16 SiteIdentityOftenConfusing ______________________________________________________ 16 TechnicalGlitchesHamperSitePerformance __________________________________________ 16 CustomerServiceGenerallyGood___________________________________________________ 17

RESULTSBYSERVICE________________________________________________ 18
Barron'sTestPrep_______________________________________________________________ 18 BostonTestPrep________________________________________________________________ 18 Kaplan'sSATOnlinePrep _________________________________________________________ 19 Number2.com___________________________________________________________________ 21 Peterson'sOnlineSATCourse______________________________________________________ 23 PrepMe________________________________________________________________________ 23 SATSecrets ____________________________________________________________________ 25 TestPreparationProgramOnlineTestPrep____________________________________________ 25 TheOfficialSATOnlineCourse ____________________________________________________ 26 ThePrincetonReview____________________________________________________________ 26

TIPSFORSTUDENTSANDFAMILIES __________________________________ 29 RECOMMENDATIONSFORPUBLISHERS ______________________________ 31 Appendix1:OnlineSATTestPreparationEvaluationInstrument ______________ 32 AppendixII:StudentResponses__________________________________________ 38 AppendixIII:AcademicProfileofStudentTesters ___________________________ 41 AppendixIV:StudentTesterSATScores,BeforeandAfterStudy,byService _____ 43

Page2

EXECUTIVESUMMARY
TakingtheSATexamhasbecomeariteofpassageformorethan1million collegeboundAmericanhighschoolstudentseachyear,anumberthathas steadilyincreasedsincetheSATwasfirstadministeredbythenonprofitCollege Boardin1926. Notsurprisingly,demandforSATtestpreparationserviceshasgrownintandem, fueledbythecompetitivenatureofcollegeadmissionsandtheabilitytotakethe SATrepeatedlytotryforabetterscore.Therearealsomorestudentswith collegeplans,accordingtostatisticsprovidedbyTheCollegeBoard.Thenumber ofcollegestudentsrose15percentinthedecadeendingin2003,whilethe numberofstudentstakingtheSATgrew35percent,toalmost1.5millionthis yearthe15thconsecutiveyearofgrowth.1 Additionally,Americansappetitefortestpreparationassistancegrewnoticeably thisyear,whenthenewSATformatwasunveiled.Forthefirsttime,thetest featuredanopenended,essaywritingcomponentinadditiontothetraditional multiplechoicetestformat.Thescoringsystemalsowasrevised,withanew highscoreof2400ratherthanthetraditional1600. Asaresultofthisconfluenceoffactors,themarketfortestpreparationservices ingeneral,notjustfortheSAT,hasballoonedtomorethan$4billion,according toEduventuresInc.,aBostonbasedmarketresearchfirm.Astudentfacinga forthcomingSATexamcanpurchaseavarietyofproductsthatpromiseabetter score,includingWebbasedprograms,books,flashcards,handheldelectronic devices,software,personaltutorsandclasses.Therearealsoavarietyoffreeor lowcostoptions,suchhighschoolcourses,andfreeresourcesprovidedatthe siteofthetestmaker,TheCollegeBoardandothers.Internetdeliveredtest preparationproductsareamongthenewestformoftestpreptoemergeinrecent years,representingagrowingsliceofthemarketestimatedat$50million, accordingtoEduventures. Duetotheexpandingtestpreparationservicesmarketandtheincreasingrole theInternetisplayingindeliveringtheseservicestostudentsandfamilies, ConsumerReportsWebWatchspentfiveweeksinthesummerof2005 examining10WebsitesprovidingonlineSATtestprepservices.WebWatch wantedtolearnhowthemarketslargest,bestknownand,often,mostexpensive optionssuchasservicesfromKaplanandThePrincetonReviewfared againstsmaller,lesserknownservices,suchasthefreesiteNumber2.com.The fieldof10sites,therefore,wasselectedwithafocusoncostvariationandmarket share,toassureamixofsiteswasincluded.Thisreportreviewedonlythose
1

TheCollegeBoard.Availableonlineat http://www.collegeboard.com/press/article/0,,46851,00.html Page3

servicesofferedonline,notthosefeaturinglivetutors,books,classesorother offlinestudyaids. The10servicesevaluatedwere:BarronsTestPrep,BostonTestPrep,Kaplan's SATOnlinePrep,Number2.com,Peterson'sSATOnlineCourse,PrepMe,SAT Secrets,TestPreparationProgram,TheOfficialSATOnlineCourse,andThe PrincetonReview. Theevaluationprocessconsistedofbothexpertandstudenttesting.WebWatch, inconjunctionwithMediatechFoundation,recruited20highschooljuniorswho hadalreadytakentheSATtestinthespringof2005andplannedtotakethe examagaininthefall.Eachofthe10siteswasreviewedbytwostudentsduring aminimumoffive,fourhoursessionsinacontrolledlabsettingunderthe supervisionofthestudyauthors,bothexperiencededucators.Testersspent morethan400combinedhoursreviewingthesesites, Whiletestersdeemedsevenofthe10sitesgenerallyeffectiveintheirproduct deliveryandtheoverallqualityoftheirservices,WebWatchnotedsometroubling trends,particularlyregardingtheblendingofadvertisingandeducationalcontent andaggressivemarketingofpromotionalmaterialstocollegeboundstudents.In onecase,astudentwhoexpressedinterestinfreescholarshipinformation receivedanemailcontainingalinktoanAirForceReserveOfficerTraining Corpsenrollmentform.Studentsusingtheseservicesaremostoftenlegal minors,andgreatercareshouldbetakentonotpressurethem,eitherdirectlyor indirectly,tosharetheirpersonalinformationortopromptthemtopurchase additionalproductswiththepromiseofahigherscore. Moreover,marketingpracticesofsomesitescanbeconsideredquestionable, suchastheimpliedpromiseofaccesstoIvyLeagueeducators,orcombining testreminderswithaddonproducts.Peterson'sSATOnlineCourse,for example,promisedaccesstoaHarvardeducatedessayreaderwhoseidentity wasneverrevealed,andwhosecredentials,therefore,couldnotbeverified. SometestersweresurprisedtolearnThePrincetonReviewhadnoactual connectiontoPrincetonUniversityortheEducationalTestingService,whichis locatedinPrinceton,N.J.Moretroublingwasthepracticeofusingtestanxietyas amotivatortopurchasemorebooksorproductsbycombiningemailtest preparationreminderswithproductadvertisements. Inaddition,testersfoundtyposandgrammaticalerrorsonsomesites,prompting concernabouttheinconsistentqualityofpurchasedonlinematerials.These discoveriesnotonlychippedawayattheperceivedcredibilityofonlineproducts designedtohelpstudentsexcelonascholasticaptitudetest,butalsoprompted questionsregardingthecredentialsofthoseprovidingandchargingforthese services.

Page4

Overall,however,thetestersappeartohavebenefitedfromtheprocessof spendinghoursonthesetestpreparationsites.The20highschooljuniorswho servedastesterstooktheSATforthesecondtimeinthefallof2005,after participatinginthisstudy,andsawtheirtestscoresimprovebyanaverageof38 pointsperstudenta1.6%gain.Whilethisscoringincreasemaybetheresultof manyfactors,includinggreaterfamiliaritywiththeSATtest,thestudentsdid appeartogaingreatertestingconfidenceafterdedicatinghourstotest preparation.

KEYFINDINGS
OnlineServicesComewithAggressiveAdvertising.Byregistering forservices,includingKaplan's,TheCollegeBoard,BostonTestPrep andThePrincetonReview,studentsweresuffusedwithmessages fromcollegeandmilitaryrecruitersoroffersforfinancialaidorstudy aids.OneemailmessagefromThePrincetonReviewcontainedalink toaUnitedStatesAirForcerecruitingform.Additionally,TheCollege Boardmarketedits Webbasedonlinetestprepserviceinadvertising spacenotclearlydistinguishedfromfreetestresources. ErrorsAbound.Mistakesinonlinesampletestsincluding grammaticalproblems,questionswithnoanswers,missingsectionsof text,fontproblems,orpoorlyconstructedquestionswereconsistently presentin6ofthe10servicesevaluated,butwereparticularlyevident inBarron'sTestPrep,Peterson'sSATOnlineCourse,PrepMeand TestPreparationProgramsites.Onlyoneofthe10servicestested BostonTestPrepofferedameansforreportingerrors. HigherPriceandReputationDontNecessarilyMakeforBetter Products.Number2.comwastheonlyfreeofchargeservicetested, anditperformedexceptionallywellagainstexpensive,betterknown products,suchasThePrincetonReviewandKaplan's.Theprep serviceofferedbyTheCollegeBoard,thecreatoroftheSAT,was buggyandlackedinteractivefeaturescommononothersites.Testers alsofoundTheCollegeBoardscomputerizedessaygradingservice toogeneraltobeuseful,comparedwithThePrincetonReviewsmore specificfeedbackprovidedbyhumaneditors. TechnicalIssuesHamperedPerformance.Kaplan'sSATOnline Prep,ThePrincetonReviewandBostonTestPrepmaynotworkon olderbrowserswithvirusprotectionorpopupblockingfeatures activated,testersdiscovered.Newversionsofbrowsersare recommendedorrequired,withcurrentpluginsandtheabilitytouse cookiesturnedon.Unfortunately,thisalsomakesabrowsermore vulnerabletopopupadvertisingandadware.

Page5

StudentsPrivacyProtectionQuestioned.Somesitescontained onlyvaguedescriptionsaboutwhatstudentinformationwouldbe shared,andwithwhom.ThePrincetonReview,TheCollegeBoard, andKaplan'sSATOnlinePrepwerenotforthcomingaboutwhotheir partnerswereandwhatoptinactuallymeans,definitionsthat shouldbeclearlyprovidedandeasytounderstand,giventhetarget audiencefortheseproductsisteenagers.

CREDITS
ThisprojectwaswrittenanddirectedbyWarrenBuckleitner,Ph.D.,aconsultant andadvisertoConsumerReportsWebWatch.BuckleitneriseditorofChildren's TechnologyReview,aperiodicalcoveringchildrensinteractivemedia.Healsois thefounderandinterimdirectoroftheMediatechFoundation,anonprofitpublic communitytechnologycenterbasedinFlemington,N.J.,wheretheresearchwas conducted.ThisstudywascoordinatedbyLindaBradway,aretiredpublicschool principalandresearcheratMediatech. Theprojectwasfundedatacostofapproximately$33,000byConsumerReports WebWatch,withfundingandstaffcontributionfromConsumersUnionsStatistics &QualityManagementDepartment.TheresearchreportwaseditedbyTracy Ziemer,WebWatchsresearcherand Websiteproduceratthetimeofthetesting, andBeauBrendler, WebWatchsdirector.ConsumersUnionandMediatechare notaffiliatedwithanySATtestprepservicereviewedinthisstudy.

Page6

ABOUTCONSUMERREPORTSWEBWATCH
ConsumerReportsWebWatchisaprojectofConsumersUnion,thenonprofit publisherofConsumerReportsmagazineandConsumerReports.org.No ConsumersUnionpublicationacceptsanyfundswhatsoeverfromadvertising. WebWatchissupportedbyThePewCharitableTrusts,whichinvestsinideas thatfueltimelyactionandresulttheJohnS.andJamesL.KnightFoundation, whichpromotesexcellenceinjournalismworldwideandinvestsinthevitalityof 26U.S.communitiesandtheOpenSocietyInstitute,whichencouragesdebate inareasinwhichoneviewofanissuedominatesallothers. ConsumerReportsWebWatch'smissionistoimprovethecredibilityofWeb sites,throughresearch,newsinvestigations,andarticulationofbestpractices guidelinesinspecificsectorsofWebpublishing.WebWatch'sresearch, investigations,guidelines,conferencetranscripts,alistofsitesthatcomplywith WebWatch'sbestpracticesguidelines,andothermaterialsareavailableforfree athttp://www.consumerwebwatch.org.

Page7

INTRODUCTIONANDMETHODOLOGY
Whatcosts$41.50,lastsalmostfourhours,andgetsalmost1.5million teenagersactually,1,475,623in2005outofbedearlyonaSaturday morning?TheansweristheSAT,anoftenmindnumbingriteofpassagefor almosteveryU.S.studentwhowantstoattendcollege. TheSATReasoningTestwascommissionedin1926byanassociationof colleges,knowntodayasTheCollegeBoard,withtheintentionofmeasuringa studentscriticalreading,mathandwritingskillsneededforacademicsuccessin college.In1947,TheCollegeBoardbeganworkingwiththeEducationalTesting Service(ETS),alarge,independent,nonprofitorganizationinPrinceton,N.J.,to createandadministertheSAT.Today,ETSisathrivinghubofstandardized testing,creatorofmorethan30wellknownteststhatincludetheGRE,APand TOEFL.TheCollegeBoardremainsETSslargestclient.Whilegroupssuchas theNationalCenterforFair&OpenTestingdebatethevalueofstandardized testing,thereisnodenyingtheSATisentrenchedinAmericanacademicculture. TheSAT,givenseventimesperyearacrossthecountry,isdesignedto challengeeventhebrighteststudent,especiallythosewhohavenotcomeface tofacewithaquadraticequationforsometime.Thetestisalmostentirely multiplechoice,thoughtheformatoftheSATchangedrathersignificantlyin 2005toincludeanopenendedessaywritingsection.Inaddition,therearenow threesectionsinsteadoftwoandamoredifficultmathsection.Studentscan achieveatopscoreof2400insteadof1600. Acommonstrategyformanystudents,includingthe20thatparticipatedinthis study,istotakethetestatleasttwiceonceduringtheirjunioryear,typicallythe spring,thenagainthefirstsemesteroftheirsenioryeartotryforabetterscore. StudentscantaketheSATasmanytimesastheyarewillingtopayforit, tormentingsomestudentswiththenotionthat,aswithagolfswing,onecan alwaysdobetter.

ANewWaytoPrep:Online
Theincreasinglycompetitivenatureofcollegeadmissions,combinedwiththe abilitytotakethetestmorethanonce,hascreatedathrivingSATtestprep business.Inthepast,testprepproductshaveconsistedofbooks,personaltutors andclasses.Morerecently,subscriptionbasedonlinetestprepcourseshave becomeavailable.Requiringacomputer,creditcardandInternetconnection, theseservicesdifferconsiderablyinpriceandservice. WhiletheexactsizeoftheonlineSATmarketisdifficulttodefine,ithasbeen estimatedat$50millionbyEduventuresInc.,aBostonbasededucationmarket researchfirm.Oneofthemarketleaders,ThePrincetonReview,providesprep

Page8

servicesforavarietyoftests,includingtheSAT,andreporteda15%increasein testpreprevenue,from$61.2millionin2004to$70.3millionin2005. Itssignificanttorecognizethelureof1.5millionSATtakersandfutureU.S. collegestudentsasamarketdemographicgrouptocommercialand,potentially, militaryinterests.Thisisapopulationonthevergeofspendingtensofthousands dollarsoncollegetuition,housing,books,computersandotherelectronics. Moreover,somefamiliesaremakinglongtermdecisionsaboutloans,mobile phoneservices,scholarshipsandfirsttimecreditcardownership. Giventhepotentialfinancialsignificanceofthisdemographic,WebWatchalso examinedrelationshipsbetweenthesebusinessinterestsandtestpreparation sitesanddiscoveredtroublingissuesregardingthehandlingofstudentsprivate informationandadvertisingtominors.Thismatterisdiscussedatgreaterlength intheFindingsandConcernssectionofthisreport.

METHODOLOGY
SelectionofOnlineTestPrepServices
Inanefforttobetterunderstandthenatureoftheonlinetestpreparation business,WebWatchsurveyedbothcommercial,forprofitservicesandnon profitoptions,andselected10sitesthatrepresentedamixofpricesandmarket sharetoassurethefullspectrumofoptionsavailabletostudentswascovered. Thesitesselectedincludedbothhouseholdnameprograms,suchasThe CollegeBoard,Kaplan'sandPrincetonReview,aswellaslesserknown services,suchasPrepMeandSATSecrets. Inordertobeconsidered,asitehadtomeetthefollowingcriteria: ActivelyadvertisetestprepservicesforthenewSATformatontheirInternet sites BeavailableduringatimeperiodofJuly15Aug15,2005,whenWebWatch conductedtesting BedeliveredprimarilyviatheInternetratherthanthroughbooks,classesor personaltutors. Thefollowingserviceswereselectedforthisstudy,listedherewiththe subscriptionpricespaidbyWebWatchonoraroundJuly15,2005: 1. Barron'sTestPrepfromBarron'sEducationalSeries,Inc.,$10.95for6 monthswww.barronstestprep.com. 2. BostonTestPrepfromBostonTestPrep,$99.00for3months http://sat.bostontestprep.com. 3. Kaplan'sSATOnlinePrep fromKaplan,Inc.,$399.00for6months www.kaptest.com.

Page9

4. Number2.comfromXapCorp.,free,withnotimelimitwww.number2.com. 5. Peterson'sSATOnlineCoursefromTheThompsonCorporation,$169.95 for4monthswww.petersons.com. 6. PrepMefromPrepMeCorporation,$499.99for4months www.prepme.com. 7. SATSecretsfromMorrisonMediaLLC,$49.95 www.secretsstudyguide.com/sat/ 8. TestPreparationProgramfromOnlineTestPrep,$29.95for3months www.onlinetestprep.com. 9. TheOfficialSATOnlineCoursefromTheCollegeBoard,$69.95for4 monthshttp://store.collegeboard.com. 10. ThePrincetonReviewSATOnlinefromthePrincetonReview,$399.00for 4monthswww.review.com

SelectionofStudentTesters
These10onlineSATtestpreparationserviceswereevaluatedby20highschool juniorsinNewJersey,whereMediatech,anonprofitcommunitytechnology center,islocated.Thestudents,recruitedthroughalocalnewspaperarticle, receivedfreeaccesstotworandomlyselectedservicesinexchangefortheirsite evaluationsandfutureSATtestscores.Thetestprepsessionstookplaceevery weekdaymorningfrom9amtonoononInternetenabledcomputerstationsover thecourseoffourweeks. Thisfieldof20students(11male,9female): WaswillingtosharepastandfutureSATscores,andhighschoolgrade pointaverages WasfromfourdifferentNewJerseyhighschools(threepublic,one private) HadalreadytakentheSATtest,withthenewessayformat,inthespring of2005andplannedtotakeitagaininthefall Wasdoingnoothertestpreparationoverthesummerinadditiontothe workatMediatechforthisstudy.

Itshouldbenoted20testersisnotlargealargeenoughsampletoclaim statisticalsignificancewithanyofthemeasures.Nostatisticalvalidityshouldbe assumedbythereaderwithregardtothequantitativemeasuresinthisreport. Instead,thestudentresponsesprovidequalitativeindicatorsofhowasmall groupofrandomlyassignedstudenttestersfeltaboutaparticularserviceduring aspecifictimeperiod.

Page10

TheTestingProcess
Studentswererandomlyassignedatestprepservicetoevaluate,thenwere registeredtothatserviceusingrealnamesandhomemailingaddresses.They werealsogivenanoperationalemailaddresscreatedspecificallyforthisproject. Thisemailaddresswascontinuallymonitoredbyoneoftheadultresearchersto trackongoingcorrespondencefromthesites,suchaspromotionaloffers, remindersorbillingissues. Whilestudentswereencouragedtostaywiththeirassignedprograms,fourofthe 20studentsrequestedpermissiontochangetoanotherserviceafteralerting adultresearchersastosignificantsiteperformanceand/orcontentissueswith theirassignedsites.Researchersmadenoteoftheissuesathandand,ifagreed thestudentstestpreparationneedswerenotbeingmet,grantedpermissionfor studentstoevaluateanotherservice. Eightofthe10serviceswereevaluatedbytwoormorestudentsforaminimum of10,fourhoursessionsforatotalof40hoursperservice.Thetwoexceptions werePrepMeandSATSecretsinthesecases,studentsrequestedpermission toterminateparticipationintheseservices.(Formoreinformation,seeResultsby Service).Inall,studentsloggedmorethan400hoursatMediatechevaluating services,withadditionalhoursloggedathome. Studentswereinstructedtodenypermissiontosharepersonalinformationwith outsidepartners,oroptoutofthisprocess,whenregistering.Thisoccurredforall buttwoservices:KaplansandThePrincetonReview.InthecasewithKaplans, astudentaccidentallyoptedinafterbecomingconfusedbyaregistrationform. WithThePrincetonReviewregistrations,astudentwhoneededtohave materialsmailedtoahomeaddresswasrequiredtooptin,inordertoreceivethe booklets.Inbothcases,studentscouldlateradjusttheirpersonalpreferencesto optoutoftheinformationexchange. AllregistrationfeeswerepaidbyWebWatch,andeachstudentwasgivena$5 cashincentiveattheendofeachtestpreparationsession.

DataCollection
Bothquantitativeandqualitativedatawerecollected.Studentstookscreen capturesofeachsiteduringtheregistrationprocess,thenusedapersonallogto documentanysiteerrors,eventsorcommentsduringthetestingprocess.Atthe endofthe20hourperiod,researchersusedachecklistdesignedtocapture strengthsandweaknessesabouteachtestpreparationservicetointervieweach student.Thesechecklistsweretalliedandscored,providingmorespecific informationabouteachservicesstrengthsandweaknesses(seeAppendix1).

Page11

Inaddition,sections611oftheinstrumentweredesignedspecificallytoexamine factorsthathavebeenassociatedwiththetrustworthinessofanInternetservice. Theseincludesuchissuesasidentity,disclosure,paymentandorder confirmation,privacyandtheuseofadvertising.

Page12

MAJORFINDINGSANDCONCERNS
Theonlinetestpreparationservicesevaluatedinthisstudyvariedsignificantlyin price,featuresandquality.While,inmostcases,studenttesterswerepleased withtheironlinetestprepexperienceandwouldrecommendthiscourseofstudy toothers,theresearchalsocastlightonseriousissuesofaggressiveadvertising andsiteperformanceusersofsuchonlineservicesshouldbeawareof. Beforedetailingtheseandotherobservations,thelimitationsofthisresearch shouldbenoted.First,Internetdeliveredservices,includingthe10SATtestprep servicesconsideredinthisstudy,areobjectsinmotion.Theobservationsand detailsdescribedhereoccurredbetweenJuly21andAugust15,2005. Second,thegroupof20testerswasnotlargeenoughtoprovidevalidstatistical evidence.Instead,thissurveyinformationprovidesqualitativeindicatorsofhowa smallgroupofrandomlyselectedstudenttestersfeltaboutaparticularservice. Third,whenitcomestotheissueofSATtestperformance,therearemany potentiallyconfoundingvariablesthatcanemerge,andsometimesdid.These canincludetestanxiety,exposuretooutsidetestprepmaterialsthatmaynot havebeendisclosed,andlearningdisabilities,tonameafew. Despitethesestatisticallimitations,thisresearchprovidesathoroughqualitative assessmentofonlinetestprepsiteservicesandfunctionality,whenthetarget audienceisyoungadultsunderage18makingmajordecisionsabouttheir futures. Researchersinterviewedeachstudenttocapturedetailedindividualreactionsto eachtestprepsitereviewed.Researchersusedtestersperceptions,aswellas culledobservationsduringthetestingprocess,toassessasitestrustworthiness andtoarriveatthefollowingfindings:

OnlineServicesComewithAggressiveAdvertising
Byregisteringforservices,includingKaplan's,TheCollegeBoard,BostonTest PrepandThePrincetonReview,studentsoftenreceivedmessagesfrombanks, collegeandmilitaryrecruiters,oroffersforfinancialaidorstudyaids.Oneemail messagefromThePrincetonReviewcontainedalinktoaUnitedStatesAir Forcerecruitingform. Additionally,TheCollegeBoard,themakeroftheSAT,mixedfreeofficialtest information,suchassampletestsandparentalguidance,withadvertisementsfor "TheOfficialSATOnlineCourse,"a$69.95product,withoutdisclosingtheprice upfront.

Page13

Ifstudentswerecarefultooptoutofpromotionswhenregistering,nearlyalle mailsolicitationswereavoided.However,thisprocesswasnotalwaysobvious. Insomecases,suchaswithThePrincetonReviewandKaplans,optingoutwas notaforthrightprocess.InthecaseofBostonTestPrep,promotionaloffers arrivedviaemaildespitestudentsrequestsnottoreceivesuchoffers.

HigherPriceandReputationDontNecessarilyMakeBetterProducts
Number2.comwastheonlycompletelyfreeservicetested,anditperformed exceptionallywellagainstexpensive,betterknownproducts,suchasThe PrincetonReviewandKaplan's,whichcost$399each. TheCollegeBoard,thecreatoroftheSAT,wasabetterknownbrandthan Number2.com,butwasbuggyduringtestingandlackedinteractivefeatures commononothersites.TestersalsofoundTheCollegeBoardscomputerized essaygradingservicetobetoogeneraltobeuseful,comparedwithThe PrincetonReviewsmorespecifichumanfeedback.

StudentsPrivacyProtectionQuestioned
Somesitescontainedonlyvaguedescriptionsaboutwhatstudentinformation wouldbeshared,andwithwhom.Studenttestersexpresseddifficultyfinding informationonbusinessandmarketingpartnershipsandwhatoptinactually meanswhenusingonlineproductsforThePrincetonReview,TheCollege Board,andKaplan'sSATOnlinePrep.Thesedefinitionsshouldbeclearly providedandeasytounderstand,consideringthetargetaudienceisteenagers. InallbutonecaseSATSecretstheprocessforstoringorhandlingstudent information,includingcreditcardinformation,wasnotclearlydefined,norfora student'spersonalmailingaddress,phone,email,passwordsandpersonal interests. Infact,withmostsitesevaluated,understandingwhohasaccesstostudents personalinformationisvagueatbest.Take,forexample,thistextfromThe PrincetonReviewsite: "Fromtimetotime,weobtainlistsofpeoplewhomaybeinterestedinour productsandservicesfromthirdpartymarketinglistsuppliers.Weuse theseliststosenddirectmarketingpromotionsbymailorbyemail.We doourbesttoonlyobtaintheselistsfromsupplierswhohaveoptin policiesforthirdpartymarketing,butwearenotresponsibleforthe conductandpoliciesofthesesuppliers.Pleasenotethatevenifyouare aregisteredcustomerofoursandhaveaskedusnottousethe informationyouprovidedtoustosendyoupromotionalmaterial,athird partymarketinglistsuppliercouldstillprovideuswithyournameand

Page14

address,inwhichcaseyouwillreceivethepromotionalmaterialsentby ustoothersonthesupplier'slist.Ifyouhavereceivedapromotionfrom usanddonotwishtoreceivefuturepromotions,pleasecontactusat: privacy@review.com." (http://www.princetonreview.com/footer/privacy_policy.asp) Giventhepriceoftheservicesbeingsoldsomeapproaching$500andthe confidentialnatureofstudentinformationbeingcollected,itisclearonlinetest prepservicesneedtodoabetterjobdisclosingwhoisbehindtheservice,what informationiscollected,andhowstudentinformationisused.

SiteFunctionalityInfluencesPerceptionofTrust
Theoveralldesignandfunctionalityofasitecontributednotonlytohigher studentsatisfaction,butahigherleveloftrustinthatservice,testerssaid. When askedtoreacttodesignrelatedsiteelements,suchaseaseofuse,interactive featuresandtechnicalrobustness(seeAppendix1),studentsgavehighmarksto Kaplan's,Number2.com,BostonTestPrep,PetersonsandThePrinceton Review.Ofthese,KaplansandThePrincetonReviewweresingledoutby testersforprovidingthebestcombinationofcontent,coachingmethodsand individualizedrecordkeeping.TheCollegeBoardsOfficialSATOnlineCourse wasdeemedtrustworthydespitehavingdesignproblems,because,asstudent testersnoted,TheCollegeBoardalsopublishestheSATtest. ThreeoftheservicesSATSecrets,PrepMeandtheTestPreparationProgram werefoundbystudentstohavesignificantdesignproblems,includinga frustratinglycomplexsigninprocessandpoorlydesigneduserinterface.These issuesinterferedwithsitefunctionalityand,therefore,usertrust.

InteractivityDistinguishesOnlineServices,NeedsImprovement
Makingonlinetestprepmorethanjustabookonacomputerwasbestachieved byKaplan'sSATOnlinePrepandNumber2.com,accordingtostudents.Testers particularlyappreciatedthesesitesimmediatefeedbackfunctionsandeffortsto organizecontentintoshortersections,whichmadematerialeasiertodigest. Testersalsolikedbeingabletotracktheirimprovementviarecordkeeping features,toolsthatheightenedtheonlineexperienceovertraditionaltestprep books.Asitesuseofaudio,Flashbasedvideoandothermultimedia componentsratherthanmerelylistinglongsectionsofquestionsasonemight findinabookwasvaluedbystudenttesters. However,allsitestestedlaggedbehindotherelectroniceducationaland entertainmentmaterialsintermsofdynamicinteractivefeatures.Whenasked abouteachsitesuseofinteractivetechniquesincludingcontextsensitivehelp, blogsbyotherstudents,rolloverhints,worddefinitions,Flashbasedgames,

Page15

autocorrectingandautolevelingstudentssaidallservicestestedcould improve.

ErrorsAbound
Mistakesinonlinesampletestsincludinggrammaticalproblems,questionswith noanswers,missingsectionsoftext,fontproblems,orpoorlyconstructed questionswereconsistentlypresentin6ofthe10servicesevaluated,butwere particularlyproblematicwithBarron'sTestPrep,Peterson'sOnlineSATCourse, PrepMeandOnlineTestPrep.Theseerrorsdiminishedstudentstrustinthese sitesasscholarlyresources. Onlyoneofthe10servicestestedBostonTestPrepofferedameansfor reportingerrors.

SiteIdentityOftenConfusing
ThreeoftheservicestestedNumber2.com,BostonTestPrepandBarron's providedmorethanonecompanynameduringtheregistrationandpayment process,raisingquestionsaboutwhichorganizationwasactuallybehindthese servicesandtherelationshipbetweenthefirms. Inthemosttroublingcase,researchersdiscoveredtheTestPreparationProgram wasregisteredtoNicoleSmithinSanAntonio,Texas,withanInternetservice providerlocatedoverseas(ZoomBangkok.com).Thesitedidnotrespondto,nor diditcomplywith,testersrequestsforrefunds. Inadditiontoconfirmingtheidentityofthebricksandmortarinstitutionsthat publisheachtestprepsite,researchersreviewedwhethersitesprovidedthe credentialsoridentitiesofhumanessayreaders,particularlysinceatleastone sitetoutedtheuseofHarvardeducatedreaders.Inonlyonecase,PrepMe, werereadersqualificationsandexpertisedetailed.

TechnicalGlitchesHamperSitePerformance
Overthecourseoffourweeksoftesting,studentsloggedhundredsofglitches anderrors.Thesecouldbecategorizedintwowayserrorsduetotheservice provider(serviceoutages,problemsloggingin,sluggishperformanceasaresult ofthepublisher'shardware)andthosecausedbythestudentscomputeror browser(fonterrors,missingplugins,popupblockers). Ingeneral,mostoftheservicesperformedfairlyreliablyonadaytodaybasis, aslongasanewerversionofMicrosoftInternetExplorerbrowserwasusedon aWindowsXPcomputersystem.Whensiteswerebrowsedusinganolder MacintoshcomputeroramoreobscurebrowsersuchasMozillaFirefox, NetscapeorAppleSafarithereweremoreproblems,includingscreenlayout

Page16

issues,missingfontsorsectionsoftext,oraninabilitytologin.Newversionsof browsersarethereforerecommendedorrequired,withcurrentpluginsandthe abilitytousecookiesturnedon.Unfortunately,thisalsomakesabrowsermore vulnerabletopopupadvertisingandadwarefromotherWebsites.(Notethat systematictestingwithdifferentbrowserswasnotpartofthisstudy.) StudentsratedBostonTestPrep,Kaplan'sSATOnlinePrepandNumber2.com asgenerallyreliable.Alsogood,butwithminorglitchesoroccasionalserver outages,wereBarron'sTestPrep,ThePrincetonReviewandPeterson'sSAT OnlineCourse.Studentsreportedapatternofbothserverandbrowsererrors withTheOfficialSATOnlineCourse,PrepMeandTestPreparationProgram.

CustomerServiceGenerallyGood
Researchersreviewedeachsitespaymentandorderconfirmationprocessto assesstheproperhandlingofconfidentialcreditcardinformation.Forthreeof the10servicesreviewedTestPreparationProgram,PrepMeandPeterson's SATOnlineCoursetestersexperiencedeitherimpropercreditcardchargesor didntreceiveservicesthatwerepromised.Intwocases,researcherswereable toresolvetheproblemsbycallingsitesdirectly.Becausetherewasnophoneor emailcontactinformationfortheTestPreparationProgram,researcherswere unabletoobtainarefund.

Page17

RESULTSBYSERVICE
Barron'sTestPrep
Barron'sEducationalSeries,Inc. $10.95for6months http://www.barronstestprep.com Basic,easytouseandaffordable($10.95forsixmonthsofaccess),testers foundBarron'sTestPreptobelackinginfeaturesbuteffectivefortakingself scoringpracticeexams.Barron'strustworthinesswaserodedbytheuseoffive differentpublishernamesonthesite.Barrons,BarronsTestPrep, Barronseduc.com,BarronsEducationalSeries,Inc.andPlug&Play Technologies,Inc.wereallusedatvariouspointsineithertheregistration processorintheinitialpitchscreen,leavingtheconsumertowonder,Willthe realBarron'spleasestandup? Afterastraightforwardsigninprocess,studentswerepresentedwithasummary oftheiractivities,includingpriorSATpracticetestscores.Sitecontentincluded longlistsofdrySAThintsandtechniquesandgradedpracticeexams.Onetester foundthemathsectiontobeharderthantheactualSATtest.Inaddition,some ofthesamequestionswereusedondifferentpracticetests.Onetesterwanted betterhelponsomeofthequestionsandfoundtheessayscoringfeaturetobe "uselessbecauseessayswerereturnedwithacomputergeneratedscore. ThevocabularysectionwasnothingmorethanlistsofhighfrequencySAT words,withstandarddictionarydefinitions.Testersnotedthattheirtrustin Barron'swaserodedbythelowprice("what'sthecatch?"),anunprofessional looktothegraphicsandpresentation,andtheconfusingpublisheridentity.

BostonTestPrep
BostonTestPrep $99for3months http://sat.bostontestprep.com Wellorganizedandeasytouse,BostonTestPrep(BTP)offereda comprehensiveservicewithsomeinnovativeoptions,liketheabilitytointeract withotherstudentsonline.Whiletestersfoundthesitetobedryandlackingin interactivefeatures,theylikedthecleandesign,withthemath,writingand readingsectionsbrokenintoeasilydigestedsubsections.Theyalsolikedhaving theabilitytocustombuildcustomquizzesaroundspecificweaknesses. Noessaygradingfeatureswerenoticed,justgeneralwritingtipsandsample essaysforstudentstocompare.Testersweredisappointedtofindthatoneofthe essaywritingoptionsconsistedofnothingmorethanaPDFfileofablanksheet Page18

ofessaypaper.Onetestersubmittedaquestionusingthe"contactus"form askingifanessaygradingoptionwasavailable,butdidnotgetareply. Toitscredit,BTPisoneoftheonlysitestoaskforsuggestionsonhowto improvethesite,bywayofa"BugSubmissionForm"toreporterrorsusinge mail.TrustinBTPwasdiminishedbothbyunfamiliaritywiththebrandname,and theappearanceofmorethanoneorganizationsnameinthepurchasingprocess. ThiswasnoticedontheMediatech'screditcardstatement,whenacompany called"Acadient,Inc."waslistedinsteadofBostonTestPrep,asthecompany thatprocessedtheregistrationfees.Duringregistration,testers,whohad difficultyfindingaphonenumberandconfirmingthesitesidentity,wererelieved tofindaBoston,Mass.,mailingaddress. BTPwastheonlyonlinetestprepservicetestedtouseapotentiallyconfusing "freetrial"registrationsystem,withanautomaticallyrenewablesubscriptionplan. Iftheuserselectedthemonthlyplan,BTPwouldautomaticallybillthecreditcard for$99,eveniftheservicewasnotused.Researcherswereabletocancelthe serviceafterthetestingperiod,bycallingtechnicalsupport. Thefirstattempttoregisterresultedinacrypticerrormessage,requiringa successfulcalltotechnicalsupport,whichblamedtheproblemonan"upgraded server." AdvertisingappearedonthehomepageforadditionalBTPproducts,butnot insidethetestprepareas.Whiletherewasastatedprivacypolicy,wecouldnot findawaytooptoutofemailcorrespondence.Forexample,onSeptember1, 2005,bothtestersreceivedthefollowingemailpromotion: "DoyouhaveanMP3player?(ifnot,hereisanewreasontogetone.) BostonTestPrephaspartneredwithAudible.comtobringyouBTPTO GO:SATPrepTalk.Partmotivation,partpreparation,SATPrepTalkgives youthetoolsandtechniquestoconquerthenewSAT.SATPrepTalkisa greatcomplimenttoouronlineSATprepprogram.Whenyoucan'tbe online,putyourheadphonesonandkeeppreping.DownloadourSAT PrepTalkMP3today."

Kaplan'sSATOnlinePrep
Kaplan,Inc. $399.00for6months http://www.kaptest.com PartofTheWashingtonPostCompany,Kaplan'sSATOnlineTestPreparation providedawelldesignedbutexpensiveservicethatwaseasytouseandfullof stateofthearttutorialfeatures.SATOnlinePrepwasjustoneofavarietyof servicesforsaleatkaptest.com.

Page19

Aftersomeinitialtroublefindingtheloginpage,studentswereabletosigninand takea"courseintroduction"sectionofthesite.Thefeatureswereclearly explainedinbothtextandaudio,whichstudentsliked.Afteradiagnosticexam, Kaplan'spresentedapersonalstudyplanbasedontheusersstatedanticipated SATtestdate.Forexample,onestudentwasencouragedtobeginwithher weakersubjectareaandproceedtoherstrongersubjects,mathandwriting. Theserviceprovidedcomprehensiveprogresstracking,usingrollovergraphics andcolorcodedgraphs.Studentslikedtheextrahandholdingwhenthey neededit,andKaplan'sprovidedit,bywayofremediallessons,immediate feedback,andstepbystepassistance.ThesefeaturesmadeKaplan'sthemost likedofanyoftheservicesusedbyourtesters. Kaplan'sSATOnlineTestPrepwasrunbyaknowncompany,withfunctionale mailandtelephonecontactnumbers.Refundpoliciesweredefined,andKaplan's useda"higherscoreormoneybackguarantee"thatpromisedadditionaltest prephelpatnocost,ifahigherscorewasnotprovided.Thisguaranteewasnot testedaspartofthisstudy. However,ofalltheonlinetestprepservices,Kaplan'susedatroublingtechnique toharvestandsharestudentinformation.Tryingtoavoidgivingpermissionto releasepersonalinformationonaKaplan'sregistrationformislikeagameof whackamole,duetospringloadedoptinfieldsthatrepopulatethemselvesas adefaultcondition,unlessavigilantstudentnoticesandreunchecksthem.For example:

Kaplan'sRegistration,Screen1:Notethatthethree"optin"boxesarenot checked,andoneofthemoreobscurefields("HowdidyouhearaboutKaplan's TestPrepandAdmissions")hasbeenoverlooked,whichiseasytodo,because itlooksfull. Page20

Kaplan'sregistration,Screen2:Thecheckoutformwasrejected,witharequest tofillinthemissingfield(toparrow).Kaplan'salsopopulates,orcheckstwoof thethreeoptinboxes,sothattheyhavetobemanuallyuncheckedeachtime theformisviewed.BecausethesearelocatedawayfromtheCONTINUEbox,a studentmayoverlooktheboxes.

Number2.com
XapCorp. Freenotimelimit http://www.number2.com Thiswelldesigned,freeservicecombineddepthofcontent,recordkeeping,and individualizedfeatureswithSATtestpractice.Number2matchesupwithsites costing$500,andinsomecasesdoesthingsbetter.So,what'sthecatch? Number2.comwascreatedbyXap,Corp.,acompanythatsellssoftwareand servicestocollegeadmissionspersonnel.Xapisnostrangertothecollege admissionsbusiness,andaccesstotheeyesandearsofthousandsofsoonto becollegestudentsobviouslyisofkeyinteresttoXap'sclients.Whileitis

Page21

possibletooptoutofemailsolicitations,thisisnotthedefaultcondition.We couldfindnoevidenceofdirectsellingtostudents.Inaddition,Number2.com wastheonlyservicetoincludebanneradvertising(forabank)throughoutthe testpreparationportionoftheservice. Becauseofalackofneedforcreditcardandsubscriptionverificationsteps,the registrationprocessforNumber2.comwastheeasiestofanyothersite,requiring onlyaname,addressandvalidemailaccount.Itwaspossibletoregisterusing fictionalinformation,althoughavalidemailaddresswasrequiredtoaccessthe logininformation. Number2.comgeneratedweeklyemailmessagestoremindstudentsoftheir impendingdatewiththeSAT,asfollows: "It'sbeenoverfiveweekssinceyouloggedintoyouraccountat Number2.comforyourSATprepcourse.Remember,practicingatregular intervals(atleastonceaweek)isthebestwaytopreparefortheSAT. SinceyouhavenotvisitedNumber2.comforsolong,we'regoingto assumenowthatyoudon'twantanymorereminders.Wewon'tsendyou anymoretutorialemails,unlessyouloginagain,inwhichcasetheywill automaticallyrestart.Ifyouwanttocontinuereceivingemails,don'tworry. Youcanrestarttheemailsatanytimebyjustloggingintoyouraccount. Andwe'llkeepyouraccountactiveforatleastayearfromsignup.Ifyou haveforgottenyourpassword,youcangetanewpasswordbygoingto thelostpasswordpageat<linktosignin.>Oryoucangostraightinby followingthislink<linktostudentaccount>Foryourconvenience, followingthislinkwillautomaticallylogyouin(buttoprotectyoursecurity, itwillonlyworkonce).Withmostemailreadersyoucanjustclickonthe link,butforsomeyouneedtocutandpastethelinkintothewebbrowser. Goodluckwithyourtestprep! Theservicemaintainedastudentpagewithbookmarksforwheretheuserleft off,aswellashints,testtipsanddailyquestions.Number2.comusedemailto reachstudents,sendingoutadailytipandaprogressreport,andresearchers encounterednoevidenceofusingemailtotemptstudentswithupgrades,a chancetowinalaptop,surveysorROTCscholarships,aswithothersitestested. Testersreportedthatusingtheservicewasstraightforward,withtips,quizzes andplentifulhints.StudentsreportedthatNumber2.comwasespeciallyhelpful withthemathsectionoftheSATbyprovidinginstantfeedbackforquestions, alongwithhintsifneeded. Thesiteexplainedthatrevenueisgeneratedfromsponsorshipsandlicensing, butdoesnotdisclosesourcesorprovidedetails.

Page22

Peterson'sSATOnlineCourse
TheThomsonCorporation $169.95for4months http://www.petersons.com ThistestprepservicecostsabouthalfofKaplan'sandThePrincetonReviewand promisedcustomizedlessons,practicetestsandexpertstrategies,including essayhelpfrom"aHarvardeducatededitor."Bothtesterswerehappywiththeir Peterson'stestprepexperience,recommendingittoothers. Afterastraightforwardsigninandregistrationprocess,theprogramstartedwith adiagnostictest.Basedonthetestresults,eachwasassigneda"Customized LearningPath,"withlessonsbasedonweaknesses.Testerswereabletochoose totakelessonsandtestsasoftenasneededineacharea,findingthesiteflexible andeasytoread,withhelpfulgraphics. Theessaygradingsectionwasnotasgoodaspromised.Therewasno coaching,otherthangeneraltips.Aftersubmittingtheiressays,testers immediatelyreceivedanautogeneratedscorefrom1to6,oneachofthefive sectionsofacomputergeneratedscoringrubric,alongwithamessagestating thattheiressaywouldbescoredbyaHarvardeducatededitorafteronetotwo days. Neithertesterreceivedaresponse.Belowisascreengrabdocumenting theprocessforsubmittinganessay,alongwithasamplequestion.

Despitebeingasolidandusefultestprepservice,severalfactorsdecreased studenttrustinPetersons.Itwasdifficulttopindowntheexactnameofthe company(Petersons,ThomsonPetersonsandEssayEdgewereeachusedat variouspoints),andquestionsabouttheessaygradingpolicywerenot answered.Thesitewashighlycommercialized,sellingcollegeconsultingand helpfindingscholarshipsthroughbanneradvertisingandemailcorrespondence. Duringregistration,testerswereabletosuccessfullyoptoutofreceiving promotionalmessages.

PrepMe
PrepMeCorporation $499.99for4months Page23

http://www.prepme.com PrepMecombinedonlineadvice,aboxofbooksworthabout$38shippeddirectly fromAmazon.com,andphone/emailaccesstoalivetutorwho,accordingto promotionalmaterials,hadalready"aced"theSAT. TheonlineregistrationprocessforPrepMewasprimitiveatbest.Afterfillingin therequiredinformation,includingavalidcreditcardnumber,anerrormessage appeared:"Transactionprocessed.Howeverwehaveencounteredaproblem. Clicktoretryposttomerchant."Aftertworetries,testerstriedathirdtimeusinga differentcomputerbrowsercombination,resultinginthesameerror.Wethen contactedthetelephonenumbergivenastechnicalsupport,whichturnedoutto beamobilephone.Weweretoldthatourcreditcardhadbeenbilled$1,499.97. Theproblem,accordingtothecompany,was,"wejustswitchedserversto Yahoo." Testadministratorswererelievedtoreachhelp,andwereableto successfullygettworegistrationsrefunded. Afterthecumbersomeonlineregistrationprocess,bothtestersreceiveda welcomeemailfromtheirlivetutors,readytohelp.Thetesterswereinterested intheonlinecomponentofPrepMe,whichconsistedofaseriesofslidesof poorlywrittenhints,tipsandsuggestedstrategiesfor"acing"theSAT.Therewas nowaytopracticewhatwaslearnedfromtheslides,andnowritingexercises.In addition,grammarandspellingmistakeswerenotedthroughoutthetestprep portionoftheservice,whichloweredstudenttrustintheprogramasascholarly resource.Finally,bothstudentsnotedfeelinguncomfortablesharing correspondencewithastrangeronline,eventhoughthispersonwasahired tutor. Becauseoftheseissues,bothstudentseventuallyaskedtobeputonanother program,onewithinfourhoursandtheotherafterfivedays.Testadministrators weresuccessfulinreceivingapartialrefundforoneofthestudents($499.97 minus$38,thecostoftheAmazon.combooks). PrepMeprovidedalegalbusinessname,physicaladdressandbackground informationonline,andgaveeveryappearanceofbeinganestablished company.However,informationaboutthesiteslengthofservicewasnot obvious,andtherewasnodefinedprivacypolicy. Theserviceoffereda100pointincreaseguarantee,orthestudentwouldbe allowedtotakethecourseagainatnocost,forasmanyasthreetimes.The guaranteeonlyappliedtoastudentwhoscoredbelow2260. BothstudentsfoundtheonlinecomponentofPrepMetobecrudelydone,"like readingareallybadbookonthecomputerscreen."BothPrepMetutorsmade diligenteffortstocontactthetestersbyemail.Asawaytoconnectwithareal

Page24

tutor,PrepMemayhavebeeneffective,butasanSATonlinetestprepresource, itdidnotmeettestersneeds.

SATSecrets
MorrisonMediaLLC $49.95 http://www.secretsstudyguide.com/sat/ Ourtwotestersdroppedoutofthis$49.95service,asitactuallywasmoreofan elaborateorderform fora145pagebookpreviouslypublishedbyMorrison Mediato"teachthetest."Thisbookisavailableineitherelectronic(pdforWord document)orprintform.Therewerenointeractivefeaturesonthesite. Unfortunately,theonlywaytolearnthetruenatureoftheprogramwasto registerandpaythe$49.95. Aspartoftheorder,wereceivedaspecialofferintheformofareporttitled"How toOvercomeTestTakingAnxiety"and"ContactInformationforMajorColleges andUniversities." Thepublisher,MorrisonMedia,soldavarietyofotherpublications,suchas "WhatyourtestscorecantellyouaboutyourIQ"and"ScholarshipSecrets." Whilethesalespitchofthesitewasmisleading,thecompanyprovidessufficient backgroundandworkingcontactinformation,andansweredthetechsupport line,offeringa"100%cashrefundwithinoneyearifnotsatisfied.Researchers testedthismoneybackguaranteeinJanuary2006,sixmonthsaftertesting,and promptlyreceivedafullrefundfromMorrisonMediawithin10days.

TestPreparationProgramOnlineTestPrep
$29.95for3months http://www.onlinetestprep.com Poorlydesigned,fulloferrors,andhardtouse,bothtestersdroppedoutofthis testprepservice,oneafteroneday,andtheotherafterfivedays.Thesitewas functionalenoughtotakea$29.95registrationfee,twice.Thetestprep experiencewasriddenwitherrors,andnotinteractive. Theprogramprogressandpracticetestscontainedfrequentspellingand grammaticalerrors,includingwordslike"whore"insteadof"where."Content includednoninteractivepagesoftechniques,tipsandstrategiesthatstudents weretoldtomemorizebeforegoingontothepracticetests. Thisservicelosttrustquickly,with20nonfunctionallinksonthehomepage,and noinformationaboutthepublisher.Notechnicalsupportnumberoraddresswas

Page25

listed,andpaymentwashandledbyPayPal,athirdpartyservice.Unfortunately, oneoftheregisteredstudentscouldnotsignbackintotheserviceandtherewas norecourseduetothesiteslackoftechnicalsupport. UsingNetworkSolutionsWHOISfunctiontolearnmoreaboutwhothesiteis registeredto,wediscoveredthelistingNicoleSmithinSanAntonio,Texas,with anInternetserviceproviderlocatedoverseas(ZoomBangkok.com).Email requestsforarefundorforhelpsigninginwerenotanswered.

TheOfficialSATOnlineCourse
TheCollegeBoard $69.95for4months http://store.collegeboard.com Promotedas"createdbythesameexpertswhocreatedtheSAT,"TheOfficial SATOnlineCoursewasoneofhundredsofitemssoldatTheCollegeBoard's onlinestore,alongwithbooks,reports,andteachingmaterialsthatdealwiththe SATReasoningTest.ThefactthatthisservicewaspublishedbyTheCollege Boardadetailhighlightedinmarketingeffortshelpedboostitsperceived trustworthinessamongtesters.Onestudentsaid,"Consideringthattheymake theSATandIregisteredwiththemalreadytotaketheactualSAT,Itrustthem." Theservicewaseasytouse,providingplentyoftesttakingpractice,especially formath,butwaslackingininteractivefeatures.Theservicewasalsobuggy, withmoreserverdowntimethantheotherservices.Inonecase,asectionofthe testappearedtwice.Anotherpagewasmissinganessay,makingitimpossible forthestudenttoanswerasetofquestions.Testerscomplainedthatthe presentationwasdry("likeabookonthecomputer"),withtipsand"lessons" consistingofnothingmorethannoninteractiveslideshows. Theorderingprocesswasstraightforward,buttheclaim"studentsreceive detailedpersonalizedfeedbackonallpracticetestsandquizzes"didnotsquare withtheactualstudentexperience,accordingtotesters. Researchersnotedthepromotionof"official"booksandtestprepservices,listed alongsidefreetestprepresources,withnoindicationwhichwerefree,andwhich wereforsale.ItemspromotedincludedprintbooksontheSAT,andaseriesof subjectspecificworkbooks.Testerswereabletoeffectivelyoptoutofreceiving amonthlynewsletterandbeingcontactedbycolleges.

ThePrincetonReview
ThePrincetonReview,Inc. $399.00for4months http://www.review.com

Page26

ThePrincetonReview'sOnlineSATtestprepprogramwasavailableinthree formats:SATExpressOnline($99),SATOnline($399)andSATLiveOnline ($599).Eachprovidedidenticalonlinepresentations,withdifferentlengthsof accesstime,varyingsetsofprintedmaterialsshippedbymail.TheSAT LiveOnlineservice,whichwedidnottest,includedrealtime,onlineclass sessionswithaliveinstructor.Researcherstestedthe$99andthe$399services andfoundbothtobewelldesignedandhighlycommercialized.ThePrinceton Reviewofferedthebestessaygradingserviceofanyotherservicetested, accordingtostudents.Otherstrongsitecharacteristicsincludedinteractive features,studentrecordkeepingtoolsanddiagnosticfeatures. WebWatchfoundThePrincetonReviewsregistrationprocesstobetime consuming.Moreover,inonecase,thesitestronglycompelledastudentinto optinginsothatbooksthatwerepartofthe$399servicewouldbemailedtohis home:"Youhaverequestedtestprepinformationbuthavenotgrantedus permissiontosendittoyou.PleasechecktheoptinboxregardingPrinceton ReviewTestPrepproductsandservices."Afterwards,thestudentreceiveda steadystreamofpromotionaloffers,includingthese: "Upgradeyourcomputer.Upgradeyourlife.Completethisformtoreceive moreinformationfromourpartnerinstitutionsaboutdistancelearning andallthatitcouldofferyouandyoullautomaticallybeenteredtowin anInspiron6000laptopcomputerfromDell.Maybeitsyourtimeto upgrade!" "AirForceReserveOfficerTrainingCorps(ROTC)Scholarshipprogram forhighschoolstudentsisagreatwaytocoverthecostsofschoolsoyou cangetthemostoutofyourcollegeexperience.AirForceROTC hasscholarshipsthatcancoverupto100%ofyourcollegetuition....Plus, ascholarshipfromAirForceROTCofferssomethingtraditional scholarshipsdon'taguaranteedjobimmediatelyaftergraduationwhere youcanmakeadifference." Note:Thislatteremailpromotionaskedforthestudent'sSocialSecurity Numberonthefirstscreen. StudentswereaskedfortheirpriorSATscores,highschoolgradesandfuture goals.Theservicethenusedthisinformationtoprescribeacompletestudyplan, usingfrequentemailnoticestoremindstudentsoftheiractivitylevelorsuggest additionalservices.Testersnotedthemathquestionswereharderthanthoseon theactualSAT,andsomeofthemultimediafeaturesseemedofftopic.Butthe overallpresentationwasdeemedusefulforholdinginterest.Inaddition,students likedthetesttakingadviceandnotedthemathtipssectionwasespecially helpful. Liketheotherservicestested,ThePrincetonReviewwasnotwithouttechnical glitches.Studentsexperiencedasignificantamountofserverdowntime.One

Page27

day(August9)inparticular,anentiremorningwaslostduetoanunexpected serverproblem.Callstotechnicalsupportwerealwaysansweredpromptly, thoughaccurateinformationregardingthestatusoftheserverwasnotprovided. ThePrincetonReview's$399testprepofferingwastheservicetestedtoprovide live,humanfeedbackelectronicallyononeofthesubmittedessays.Feedback includedpersonalcommentsandrecommendations,asfollows. Hi<studentname>, Yourresultsarein!Youearnedascoreof12onyour essay.Hereswhatourgradershadtosayaboutyouressayandhowto improveittomaximizeyourscoreontestday. Content:Interestingintroductoryparagraph!Yourargumentiswell reasonednicelydone.Yougointoagoodlevelofdetailwithyour examples.Yourexamplesareappropriateforthetopic. Other:Thistypewrittenessaywasverylong.Pleaseawareofthefactthat youwillnotbeabletowritethismuchontherealessay,which,asyou know,mustbehandwritten.Youdonthavetogivethreeexamples.Its finetogowithoneortwoexamples,aslongastheyarewelldeveloped andyoustillreachtwopages.Yourconclusionismoreimportantthana thirdexample.Youressaydisplayedagreathistoricalawarenessanda talentforwriting. Overall,studentsratedThePrincetonReviewhighintermsoftrust.However, testersdidnotknowthatThePrincetonReviewwaslocatedinNewYorkCity andhadnoaffiliationwithPrincetonUniversity.Whilethisfactwasdisclosedon thebooksandprintmaterialssentfromThePrincetonReview,researcherswere notabletofindthisclearlydisclosedonlineeitheronthehomepageorinthe "WhoWeAre"sectionoftheWebsite,perhapsexplainingwhytesterswere confused. Whiletherewerenoadvertisementsinthepasswordsectionsoftheactualtest prepservice,thestudent'ssigninscreenfeaturedadvertisementsforadditional testpreptools,mediatv.com,banks,loans,andcolleges,aswellasfrequent exposuretoadvertisementsforserviceupgrades. Aswiththeothertestprepsitesreviewed,ThePrincetonReviewwasvagueat bestindescribinghowitcollectedstudentinformation,whoitsbusinesspartners were,andwhatinformationwasshared.ThePrincetonReviewguaranteewas vague:Yourscoreisguaranteedtoimprovebyatleast200pointscombined,or wewillworkwithyouagainforfree.

Page28

TIPSFORSTUDENTSANDFAMILIES
TheSATisdesignedtobechallenging,anddespitemarketingclaimsmadein thepromotionalmaterialsofmanyonlineSATtestprepsites,thereareno secretsortricksthatcansubstituteforyearsofelementary,middleandhigh schooleducation.Justasyoucan'tgrowaplantfasterbywateringitmore,you can'tgetahigherSATscorewithlastminutecramming. Whenitcomestimetothetakethetest,validonlinetestprepservicesareoneof severalwaysstudentscanfamiliarizethemselveswiththeSATformat,andthis familiaritycanreduceanxiety,accordingtoKenHartman,theformerDirectorof AdmissionsandGuidanceServicesforTheCollegeBoard.Othertestprep options,generallypricedmuchlowerthanmanyonlinetestprepservices, includebooks,CDROMs,orhighschoolcourses. Basedonthisreport,WebWatchmakesthesefollowingrecommendationsto studentsandparents: *Costlieroptionsarenotalwaysbest.Considerfreeoraffordableservices,such asNumber2.comandothers.Takeadvantageoffreetestprepresources availableatTheCollegeBoard(www.collegeboard.com). *Makesureyourbrowser,computerandInternetconnectionareuptodateprior toenrollinginonlineprograms.OlderbrowsersandslowInternetconnections canresultinsluggishperformanceorerrorsinthewayinformationisdisplayed onthescreen.Inaddition,beawarefiltersandpopupblockingsoftwaremay needtobedisabledinordertouseacommercialtestprepservice. *Beforepurchasing,contacttheservicebyphonetoconfirmwhoisrunningit, andthatitwillbepossibletoreceivearefundifyoucannotsuccessfullylogin. Alsoduringthiscall,askthecustomerservicerepresentativeaboutthesites privacypolicyandwhetherastudentsprivateinformationwillbesharedwith companiesandadvertisers. *Don'tpurchaseservicesthatwon'tbeused.Wefoundthatoncethemandatory testingperiodwasover,studentsuseoftheservicesquicklydiminished.Unless yourstudentisselfmotivatedorcanfollowascheduleestablishedpriortothe SATtest,onlinetestprepmaynotbeaviableoption. *Bevigilantandpresentduringtheregistrationprocess,tomakesureyourchild doesnotaccidentallyauthorizethereleaseofpersonalinformation,unlessyou wanthimorherto.

Page29

*Knowthatinformationsenttoyourchildregardingcollegescholarships,student loans,oradditionaltestprepservicesmostlikelyisadvertising,withbusiness motives.

Page30

RECOMMENDATIONSFORPUBLISHERS
ConsumerReportsWebWatchrecommendsthatonlinetestprepservice companiestakethesestepstoimprovetheirproducts: Clearlylabeladvertisingandestablishmorefirmboundariesbetween editorial,testrelatedcontentandpromotionsfortestmaterials. Makeiteasierforuserstosigninandbeginusingtheserviceimmediately afterregisteringandpayingfortheservice. Clearlyidentifyyourbusinessnameandlocation.Provideaworking phonenumberand/oremailaddresssostudentsandparentscanfollow upifnecessary. Makeiteasyforuserstoreporterrors. Assureyourservicefunctionsonavarietyofbrowsers.Ifitdoesnt,state soclearly,upfront,beforestudentspayforservicestheywontbeableto use. Bemorediligentincheckingyoursiteforaccuracy,from correcting spellingandgrammarerrorstotestingthevalidityandeffectivenessof sampletests. KeepinmindpeoplepurchasingonlineSATtestprepservicesare studentsand,therefore,legalminors.Asaresult,careshouldbetakento guardstudentinformationandadheretostatedorbestpracticeprivacy policies.. Clearlydisclosebusinessrelationshipsandotheraffiliations.

Page31

Appendix1:OnlineSATTestPreparationEvaluation Instrument
TitleofService: Browser&OSUsedforReview: NameofReviewer: TesterCode: DateofReview: NumberofHoursUsingtheService: Startandenddatethattheservicewasevaluated:Start___End___ KEY A=Always(1point)SE=SomeExtent(.5point)N=Never0pointsNA=Not Applicable(notcalculated) PART1:DESIGN I.EaseofUse Theserviceiseasytoinstall,registerfor,anduseoveranextendedtime. ASENNA 1.____________Theregistrationprocessissimple 2.____________Startingasessioniseasy 3.____________Endingasessioniseasy 4.____________Itispossibletogetinandoutofmenusandactivitiesatany point 5.____________Theexperienceisresponsiveandeasytocontrol 6.____________Instructionsareclearandeasytofollow 7.____________Aclear,consistentmenusystemisused NOTES: II.EducationalValidity Whatdoyouwalkawayfromtheexperiencewith,thatyoudidn'thavewhenyou firstcametotheservice?Theserviceprovidesaneffectivetestpreparation service. 1.____________QuestionsetsmatchtheactualSATtest 2.____________Theserviceconsidersthestrengthsandweaknessesofthe student 3.____________Theservicecreatesanindividualprofile,includingpriortest scores 4.____________Theserviceemploysavariedpresentationwithadegreeof novelty 5.____________TestTakingSkills:includesstrategies,techniques&facts aboutthetest,beforetestday,stresstechniques,lastminute advice

Page32

6.____________CriticalReading:ThepracticeexamssimulatetheactualSAT (pacingincriticalreading,comprehension,shortpassages, longpassages,pairedpassages,wordbranches) 7.____________Writing:Examplesaregiventojudgethatsimulatetheactual SAT(pacing,style,commonerrors) 8.____________Math:ThepracticeexamssimulatetheactualSAT(gridins, workingbackwards,calculatorstrategy,trigonometry,word problems) 9.____________Questionsarewelldesigned 10.____________Providesvalidtesttakingadviceandtechniques 11.____________QuestionsreflectthenewSATformat NOTES: III.MotivationalTechniques Thesiteusesstateoftheartgraphicandprogrammingtechniquestoprovidean engaginginteractivemediaexperience. 1.____________Allowsthestudenttovarytheorderofstudying 2.____________Usesaudiofeedback 3.____________Usesdraganddropfeatures(e.g.,movepartsofsentences around) 4.____________Usesrolloverfeatures(e.g.,rolloverwordstoseethem highlighted) 5.____________Usesflashandshockwavefeatures NOTES:

IV.Features Theserviceadaptsandadjusts,usingastudentstimewisely 1.____________Remembersallowbookmarkingofstudentprogress 2.____________IndividualizesAdjustsindifficultyaccordingtostudentlevel 3.____________IsDiagnosticRecommendsthebestpathtosuccessbased onsurveyorpretestinformation. 4.____________ListensAllowsforstudentinputintotheprocess NOTES:

V.TechnicalRobustness Theserviceworksaspromised,atdifferenttimesofthedayandondifferent computers. 1.____________Providessystemrequirementsupfront 2.____________Providesaninitialpass/failsystemcheck 3.____________Provideslinkstoneededplugins 4.____________Hasreasonablesystemrequirements 5.____________Worksfromdifferentcomputers (e.g.,home/laptop/school/library)

Page33

6.____________Workswithmorethanonebrowser 7.____________Doesnothaveserverdowntime 8.____________Can"limp"withanoldersystem,andstillfunction NOTES:

PART2:TRUST VI.Identity 1._________Provideslegalnameofthebusiness 2._________Providesphonecontact 3._________Providesemailcontact 4._________Provideslocationofbusiness(geographicaladdress) 5.____________Aconsistentcompanynameisusedthroughoutthe transaction NOTES: VII.Disclosurepriortopurchase. 1.____________Featuresareclearlydescribed 2._________Thetermorlengthofserviceisprovided 3._________Noticeofadditionalcostsisprovided 4._________Terms,conditionandmethodsofpaymentareprovided 5._________Restrictions,limitationsorconditionsofpurchasearedefined 6.____________Otherlimitsaredefined,e.g.,geography,languageorparent approval 7.____________Refundpolicyinformationisprovided 8.____________Guaranteesaredefined 9.____________Sponsorsoraffiliationsaredisclosed,andanyothermotive otherthantestprepisidentified NOTES:

VII.Paymentandorderconfirmation 1.____________Cantheconsumerconfirmtheitem/itemsselectedbefore finalizingthepurchase? 2.____________Areconsumersprovidedwitheasytouse,securepayment mechanismsandinformationonthelevelofsecuritysuch mechanismsafford? 3.____________Doesadisputeresolutionprocessexists? NOTES:

VIII.Privacy Howdoestheservicehandleinformationthatcouldbeconsideredconfidential? Thisincludescreditcardinformation,physicaladdress,phone,email,passwords.

Page34

Informationrelatedtothetestprocessincludestestpreferences,testscores, performancedataandessayquestions. 1._________Isitpossibletorequestthatyouremailinformationnotbe distributed? 2.____________Isthereaprivacypolicyinplacebeforethepurchase, describinghowpersonalinformation(suchasscoreson testsoressaycontent)willbehandled? 3._________Isthedefaultconditiononethatdoesnotdistributeorshare personalinformation?(Thestudentmustindicateinsome way(e.g.,checkabox)torequestadditionalinformation andservicesduringtheregistrationprocess) 4.____________Doestheserviceusestudentcorrespondencetomarketits ownproducts? 5.____________Doestheservicereleasestudentdatatooutsidefirms?Is thereevidencethattheemailaddresshasbeengivento thirdpartygroups?) NOTES:

IX.TechnicalSupport? Doestheserviceprovidetechnicalsupport? 1.____________Bytelephone(checklistednumberstoseeiftheyare working) 2.____________Byemailcontact(checkforaresponse) Rateyourexperiencewithcustomerservice nothelpful 1 2 veryhelpful 9 10

X.Advertising 1.Wereexternalproductsadvertisedbehindthesigninscreen?YN (externalproductsareproductsnotproducedbythepublisher) 2.Wereinternalproductsadvertisedbehindthesigninscreen?YN (internalproductsareadditionalbooks,tapesorservicesofferedbytheservice provider)

XI.Trustworthiness Ratethissiteonascaleof1to10onitsgeneraltrustworthiness nottrustworthy 1 2 3 verytrustworthy 8 9 10

Page35

Doyoususpectthatthereisanyothermotiveotherthantestprep?(yes,no)If so,pleasedefine. NOTES:

Page36

Page37

AppendixII:StudentResponses
Note:Aminimumoftwostudentsusedeachservice,independentlyofoneanother,forfivefourhourdays.Insome cases,morethan2studentsusedthesameservice.Studentsweresurveyedononlytheservicestheyused. After20hours,eachstudentwasinterviewed,guidedbythechecklist(Appendix1)withresultsrecordedbothonpaper andonvideotape.Thescoreswerecheckedforvalidityinotherwords,iftwostudentsgavedifferentanswersaboutthe PrincetonReview'suseofadvertisingorhadnotyettriedtheessaygradingfeatureonKaplans,anadultresearchwould explorefurther.Insomecases,asmanyasfourstudentstestedthesameservice. Themeansurveyscoreisreportedonthischart.Ourgoalwastoeliminatevariabilityamongtesteranswers,orto understandwhytherewasvariability,untilweunderstoodeachproduct.
Barron's TestPrep Boston TestPrep Kaplan's SAT Online Prep Number2. com Peterson's OnlineSAT Course PrepMe SAT Secrets Test Preparation Program The Official SAT Online Course The Princeton Review

DESIGN EaseofUse Educational Validity Motivating Features Technical Robustness TRUST Identity Disclosure Payment& Order Confirmation Privacy Technical Support Noticed advertising outsideprep sections?

63 82 70 51 31 81 75 80 50 100 40 Y N

76 93 85 40 75 89 77 80 100 100 40 N N

86 89 95 90 69 88 92 100 94 100 75 Y N

83 100 82 90 57 88 82 70 75 NA 80 Y Y

74 93 82 75 50 72 86 100 83 67 80 Y N

31 64 30 5 0 58 72 50 67 60 80 Y N

43 NA 43 NA NA NA 92 100 100 100 100 Y N

31 64 23 10 25 31 13 20 22 0 13 N N

64 96 93 40 31 59 92 100 100 83 80 Y N

72 79 91 75 38 80 91 100 83 100 80 Y N

Notice advertising insidetheprep sections? Overall trustworthiness

80

90

85

85

85

75

60

10

96

83

Page39

Page40

AppendixIII:AcademicProfileofStudentTesters
SATOnlineTestPreparationStudyParticipantData Student# 1013 1014 1016 1011 1019 1015 1021 1008 1012 1003 1002 1020 1001 1017 1005 1004 1006 1010 1022 1009 Gender Age(yrs) F M F M M M M F F M M M M F F F M M F F 18 18 17.4 17.1 18 17 17.6 17.6 17.7 17.7 17.2 17.6 17.2 17.1 17.6 17.6 16.8 17.3 17.4 17.4 PriorTutoring Y N N Y N N N N Y N N N N N N N N N N N H.S. SATMath 1 2 3 3 3 4 3 2 1 2 4 5 2 3 2 2 3 3 2 2 510 360 420 490 630 480 650 410 690 670 520 690 520 570 470 520 470 580 620 530 540 SAT Reading 530 240 450 440 530 430 700 500 630 500 500 630 550 580 380 400 470 490 590 680 511 SATWriting SATEssay HSGPS TotalSAT Birth Order 2 1 3 3 1 1 1 3 2 1 1 1 1 3

600 240 460 550 590 530 650 470 680 n/a 450 430 640 550 800 370 340 420 480 590 580 521

8 3.8778 3 21,370 8 3.2785 8 3.7122 10 3.2929 8 n/a 10 3.9043 9 3.2484 3.9253 7 3.1734 6 2.9475 9 n/a 8 2.5737 12 2.8265 6 4.0696 6 4.1366 7 2.7969 8 3.3231 10 4.444 7 3.9219 7.5

1640 840 1330 1480 1750 1440 2000 1380 2000 1620 1450 1960 1620 1950 1220 twin 1260 twin 1360 1550 1800 1790 1572

3 1 1 1

Page42

AppendixIV:StudentTesterSATScores,Beforeand AfterStudy,byService
Gender Reading Math BTP BTP Barrons Barrons PR Express PR Express Princeton Review Princeton Review Petersons Petersons College Board College Board College Board Number2 Number2 Number2 Number2 Kaplan's Kaplan's Prepme Percent F M M F M F M F F M F M M M F M F F M M 530 470 500 680 490 400 500 590 500 700 630 630 550 240 450 530 580 380 470 430 510 470 520 530 580 520 670 620 410 650 690 690 520 360 420 630 570 470 510 480 Writing 600 420 430 580 540 340 450 590 470 650 680 640 550 240 460 590 800 370 450 530 1610 1260 1620 1800 1380 2000 2000 1960 1620 840 1330 1750 1950 1220 1430 1440 570 470 680 670 410 670 640 570 320 440 * 640 420 470 400 510 440 500 620 500 630 680 510 440 480 * 560 340 570 580 520 410 510 560 530 650 710 480 260 470 * 710 370 530 510 1600 1320 1690 1850 1440 1950 2030 1560 1020 1390 * 1910 1130 1570 1490 10 60 70 50 60 50 70 60 180 60 * 40 90 140 50 620 38.75 1.61% Total 1640 1360 1430 Reading 530 * 520 Math Writing Total 540 * 530 640 * 440 1710 * 1490 Change 70 * 60

Total Mean Percentchange

*=didnottakesecondexamduetoachangeinplans. =scoresnotyetturnedin.

Mean Variance Observations PearsonCorrelation HypothesizedMean Difference df tStat P(T<=t)onetail tCriticalonetail P(T<=t)twotail tCriticaltwotail

1640 1710 1526 1562.666667 100940 84663.80952 15 15 0.974145509 0.05 14 1.918295991 0.037849819 1.761310115 0.075699638 2.144786681

Page44