北大暑期課程《回歸分析》(Linear-Regression-Analysis)講義PKU5

Cｌａss5:ANOＶA(AnaｌysiｓｏfＶarianｃe)andF-tｅsｔs

Ｉ.WｈaｔisAＮOＶA

ＷhatｉsAＮOVA？ANＯVＡisｔhesｈoｒtｎaｍｅｆoｒtheAnalｙｓｉsofＶarｉance．TｈｅｅｓsencｅofAＮOＶAiｓtodｅｃomｐoｓetｈeｔotａlｖariancｅoｆtｈｅdｅｐeｎｄｅnｔvariaｂleiｎtoｔwoaｄdiｔiｖecｏmponｅｎｔs,onｅfｏｒthｅｓtructurａlpaｒt,aｎdtheotherfｏrtheｓtoｃhａsticparｔ,oｆａrｅｇｒessiｏn.Ｔoｄａywｅarｅgoingｔoｅxamiｎethｅｅaｓiｅstｃaｓe．

IＩ.ANOVA:AnＩnｔrｏductiｏn

Lｅtthｅmｏｄelbe

.

Asｓｕｍｉｎgxisacolumnvｅcｔoｒ(oｆlｅngｔhp)oｆindｅｐｅndeｎｔｖariablｅvａlｕeｓfoｒtheｉth＇observatｉon，

.

Tｈeniｓthｅpｒｅｄｉｃｔｅdｖａlue．

sｕｍｏfsqｕａrｅstotal:

bｅcaｕse．

ＴhｉsisalｗａyｓtruebyOLＳ.

=SSＥ+ＳSR

Impoｒtaｎt:thｅtｏtaｌvariａnceoftheｄepｅnｄeｎｔvarｉａblｅisｄｅcompｏsedｉｎtotwoadditivepａｒts:SSE，whicｈisduｅtoerｒors，andSSR，ｗhicｈｉsduｅtｏｒｅgression.

Ｇeoｍｅｔｒｉｃｉnｔeｒpreｔaｔiｏｎ：[ｂlaｃkboaｒd］

DeｃoｍｐositｉonｏｆＶａrｉａnce

Iｆweｔｒｅａt(yī)Xasaｒaｎdｏｍvａriable,ｗｅcandｅcomposetｏtaｌｖarｉanｃｅtｏｔhｅbｅtｗeｅn-grｏuｐｐｏrtiｏnaｎdthｅwiｔｈiｎ-ｇｒoｕpｐｏrtｉoninａnyｐoｐulatioｎ:

?

Prove:

?

?

（bｙtheassumｐｔｉonthat,fｏｒａｌlpoｓsiｂlek.）

TｈeＡＮＯＶAtａbｌｅｉｓtoeｓtimａt(yī)ethｅｔhｒｅeqｕantｉｔiesoｆequatioｎ(1）ｆrｏmｔｈeｓａｍple.Astｈｅｓamplesizegetｓlａrｇerandｌarｇeｒ，theANOVAtａbｌewｉllapproachthｅeqｕaｔiｏnｃlｏｓerandclｏsｅr.

Inasampｌe,decｏmｐｏｓｉtｉonｏｆｅsｔimatｅdｖarｉancｅisｎoｔstｒictｌytrue．Weｔhｕsneｅdtoｓｅparatelydｅcomposeｓｕmsofsquａresａnｄdｅgreesｏfｆreｅdom.ＩsANOVAamisｎomer？

ＩII．ＡＮＯＶAｉnMatrｉｘ

Iwｉｌｌtrytogｉveａsimplieｄｒepｒｅsenｔａt(yī)ｉoｎofＡNOVＡａsfoｌlｏws：

(ｂecａuｓｅ)

(ｉｎｙoｕrｔextbｏok,ｍonsｔｅrlooｋ)

SSE=ｅ'e

(bｅcause,ａsalwayｓ)

（inyouｒtextbｏｏk，mｏnｓterlook)

IV．ANOVATaｂle

ＳOＵＲＣE

SＳ

DF

ＭＳ

F

with

Regresｓiｏn

SSＲ

DF(R)

MＳＲ

ＭＳR／MＳＥ

DＦ(R)

Error

ＳSＥ

DF(E）

MSＥ

DF(E)

Toｔal

ＳST

DＦ（T)

Lｅtｕｓuｓearealexamplｅ．Ａssumeｔhatweｈavｅaｒegresｓiｏneｓtimateｄｔoｂe

ｙ=-1.７０+０.84０x

ＡNOＶＡＴablｅ

SOＵRCE

SＳ

DF

MＳ

F

ｗｉｔｈ

Ｒeｇｒｅsｓioｎ

6．4４

１

６.44

６.４4／0．19＝３３．８９

1，18

Erｒoｒ

３.４０

18

0.19

Ｔotａl

9．８4

１9

Ｗeknow,,,,．IｆｗekｎoｗthatDＦforＳＳＴ=１9,ｗhaｔｉsn？

ｎ＝20

?=201.7１.7+0．840．84５09．12－２1.7０．８41０0－１25.0

?=6．44

SＳE=SＳT-SＳR＝9.8４－６.４4=3.４0

DＦ(Dｅgreeｓoｆfreedom)：ｄemonｓｔratiｏn.Nｏte:ｄiscountｉｎgｔｈeｉｎtercｅpｔwhencａｌculatinｇＳSＴ．

ＭS=SS/DF

ｐ=０．０00［ａｓksｔudents].Ｗhatdoｅsｔｈep－ｖaｌuｅsay？

Ｖ.F－Ｔests

Ｆ-ｔesｔsaremoregｅｎｅraltｈaｎt－ｔｅｓts,t-ｔｅｓｔsｃanbeseｅｎａsaspeｃialcaseofＦ-ｔests．IfyｏuｈaｖedifficultｙｗithF-tｅsｔｓ,pｌeaseaskyoｕrGSＩstoｒeviewＦ-ｔｅstｓｉｎｔｈelab．F-teｓｔstakestheｆoｒmofａfｒａctiｏnoｆtwｏＭS＇ｓ.

AｎＦｓtatｉｓｔｉchasｔwodeｇrｅesoffｒeｅdｏmａsｓociaｔedwitｈｉt:thｅdegrｅｅoｆｆreeｄｏmｉntheｎumｅraｔor，anｄtheｄｅgｒeeoffreedoｍｉｎｔｈeｄeｎoｍinaｔｏr．

AnFstａｔisticiｓuｓuaｌlyｌargｅrthａn1.ＴｈeｉnｔerpｒetationofanＦｓｔatistｉcsistｈatwhetｈｅrtheexｐlaiｎｅdvarianｃebythｅａｌternaｔivehypoｔｈesisｉsdｕｅtochａnｃe．Inｏthｅrｗｏrds,thenullｈｙpoｔheｓisisthaｔtheexplａinｅdvａriａnceｉｓdueｔochance,orａllthecoeｆfiｃientsaｒezero．

ThｅlargeｒaｎF-ｓｔatisｔiｃ,tｈemorelikelythａt(yī)theｎullｈypothesisiｓｎoｔtrue.Ｔherｅｉsａt(yī)abｌeinthebａckｏｆｙourboｏｋｆromｗhichｙoｕｃａnｆｉndexaｃｔprobaｂｉlｉtｙvalues.

Inoureｘaｍｐlｅ,ｔheFis３4，ｗhiｃｈishｉｇhlｙsigｎiｆicａnt.

VI.R２

R２＝ＳSR/SSＴ

Theｐropoｒtｉoｎｏｆvariａnｃeeｘpｌaｉｎeｄｂythemｏｄｅl.

Inourexａmple,

Ｒ-sq=6５.４%

VII.Wｈathａｐｐｅnsｉfwｅｉncreａsemｏｒｅindｅpeｎｄentvａrｉａbleｓ.

1.SＳTstａyｓｔhｅsaｍe.

2．ＳSRalwaysincreａｓes.

３.SSEalwａｙｓｄecｒeasｅｓ.

4．R2alwaysincreａｓes.

5.ＭSRusｕalｌyinｃrｅaｓｅｓ.

６.ＭSEｕsｕallydeｃreａses.

７.Ｆ-tｅsｔusｕaｌlｙiｎcｒｅaｓｅｓ.

Ｅｘcｅpｔioｎｓｔo5aｎd7:irrelevａntvarｉａblｅsmａynoteｘplaintheｖaｒｉancebuttakeupdeｇreesoｆｆｒeedｏm.Wｅreallyneedｔoｌooｋａt(yī)ｔｈｅresulｔs.

VIII.Ｉｍpｏrtanｔ:GeneｒalＷａｙsｏfHyｐｏtｈｅsisTestｉngwithF-Statｉsticｓ.

AｌｌtｅstsｉｎliｎearrｅgreｓsionｃａｎｂｅｐerformeｄｗithＦ-tesｔstaｔｉsｔｉcs.Tｈｅtrickｉstｏｒun"ｎesｔedmodeｌｓ."

Twomodｅlｓａｒenesteｄifｔheiｎｄependentｖariablesｉnｏnemodｅlaｒeａｓubsｅtｏrliｎeａrcｏmbiｎａt(yī)ionsoｆasｕbsｅt(子集)ofｔhｅｉｎdeｐendeｎtvａrｉablesiｎｔhｅoｔhｅrmｏdeｌ．

Ｔｈaｔistｏsay．IfｍｏdｅlＡｈasiｎdｅpeｎｄenｔｖａriabｌｅｓ(1,,），andmｏdeｌＢhaｓｉndｅpenｄｅntvarｉablｅｓ(1,,,）,AandBaｒenｅsted.Aiscalledｔhｅｒestrictedmodel；Biscallｅｄleｓsrestrｉcｔeｄorｕnrｅｓｔriｃtedｍodel.WｅcａlｌAｒeｓｔｒictｅｄbeｃauseAiｍplｉｅsｔhaｔ.Ｔhisｉsaｒesｔｒｉcｔion.

Anotherexamｐｌe:Cｈａsｉndepenｄｅnｔｖaｒｉable(1,,+),Dhａｓ(１，＋)．

CaｎdAaｒｅnotnesｔed.

CaｎｄＢａｒenesｔeｄ.OnereｓtriｃｔｉoninC：.

CａndＤarenｅｓted.OnerestrictioninD：.

ＤaｎdAarenotｎｅsｔｅd.

DandBarenesｔeｄ：twｏrestriｃｔｉonｉnＤ:；．

Wｅｃanａlｗａyｓｔｅstｈｙpｏｔheseｓimpliｅdｉntｈerｅstｒictｅdmoｄeｌｓ.Ｓteps：runtwoｒeｇｒeｓｓｉonfｏreａｃhhyｐoｔｈesis，onefｏｒtｈｅreｓtrictedmｏdｅlaｎｄｏnｅfoｒｔｈeuｎｒeｓｔriｃtｅdmodｅl.TheSＳTｓｈouｌｄｂeｔｈeｓａmeaｃｒosstheｔｗomodeｌｓ.ＷｈaｔisｄiffereｎtiｓSSＥａndSSR.Tｈatis,ｗｈatiｓdｉｆｆerｅｎｔiｓＲ2．Let

；

Useｔheｆoｌlowｉnｇformulａｓ：

?

ｏｒ

（ｐｒoｏf:uｓeＳＳＴ=ＳSE+SSR）

Ｎｏte,df（SSＥr)－df(SSEu）＝ｄf(SSRｕ）－df(SSRｒ）＝,

ｉsｔhenumｂerofconstraｉnts(notnｕｍｂｅｒoｆｐaｒametｅrs）iｍｐliedbｙｔheｒesｔｒｉcｔedmodel

or

Notｅthａｔ

Ｔｈatiｓ，ｆor1dｆteｓts,youｃaneｉtherdoanF-ｔｅstoraｔ-test.Theyyiｅｌｄthｅsaｍeresulｔ．Aｎoｔhｅrｗaytoｌookatitｉｓｔhａt(yī)thet-tｅstisaspｅcialｃａｓeｏfthｅＦｔｅｓｔ,withｔhｅｎumｅraｔｏrDFｂeiｎg１．

IX.ＡsｓumptionsofF－tesｔs

WｈａｔａｓｓumptｉonsdoｗenｅedtomaｋeanANOVAｔaｂleｗork?

Nｏｔmuｃhanaｓsｕｍption.Aｌｌweneediｓｔheasｓumptionthat(X'X）isnotｓiｎguｌaｒ，ｓｏthaｔｔｈelｅasｔsｑuareestimaｔeｂｅｘiｓtｓ.

Ｔｈeａsｓumptioｎoｆ＝0isnｅeｄediｆyouｗanttheAＮＯVＡｔaｂｌetobｅａｎｕnbiaｓeｄｅsｔimａt(yī)eｏftｈetｒueAＮＯVA（ｅｑuaｔiｏn１)ｉｎｔhepｏpulatiｏn.Reａｓoｎ:wewａｎtｂtobeanｕｎbｉaseｄeｓtiｍaｔorｏf,anｄthecｏvａriａｎcｅｂetweｅnbａndtodiｓapｐeａr.

ForｒeaｓoｎｓIdiｓcusｓedeａrliｅr,tｈeassumpｔiｏnsｏfｈomoscedａstｉcityanｄｎｏｎ-ｓerｉａｌｃoｒｒelatｉｏnareneｃesｓaryfｏrthｅesｔｉｍａt(yī)ioｎoｆ.

Thenormalityassuｍptionthatiｉsdisｔributedinａnoｒmalｄｉsｔribuｔiｏｎisneededfoｒsmａllsaｍｐles.

Ｘ．TheConcｅpｔｏfIncreｍeｎt

Eveｒyｔiｍｅyoｕｐutoｎｅmｏｒeｉndｅpendｅntvａrｉabｌeinｔoｙｏurmodeｌ,youｇeｔａnincｒeaｓein.Wｅsｏmeｔiｍecalledｔhｅiｎｃｒeaｓe＂ｉｎcrｅmｅntaｌ."Whａｔismeanｓｉｓtｈaｔmoｒｅvａriaｎｃeｉｓｅｘplaiｎｅd，oｒSSRｉsｉncrｅaｓeｄ，SＳＥisｒeducｅd.Ｗhatｙｏushoｕldundeｒｓtａndisthatｔheincreｍentaｌａt(yī)trｉbutedtoavarｉablｅiｓalｗaｙsｓｍalｌerｔhantｈeｗhｅnｏｔhervarｉａbleｓａreａbsｅnt.

?ＸI.CｏｎｓｅquｅnｃesoｆＯｍitｔingRelevａnｔIndｅpendenｔVaｒiaｂles

Saytheｔrｕemodｅｌｉsthefollｏwｉnｇ:

.

Buｔfｏｒｓoｍeｒｅaｓonwｅoｎlyｃoｌlｅctoｒcｏnsidｅrdataｏn.Thｅｒefoｒｅ,weomitｉｎthｅregrｅsｓｉｏn.Thaｔis，weｏｍitｉｎourmｏdｅl.Webriefｌydｉｓcusｓｅｄthｉspｒｏblembeｆore.Ｔheshｏrｔｓtｏrｙiｓtｈatweａrelikelｙtｏhａveabｉaｓduｅtotｈｅomisｓｉｏnｏfarｅleｖanｔｖａriableinthｅmｏｄel．Thｉsｉsｓoeｖenthouｇｈourｐriｍａｒyinｔeｒeｓtistｏestimatetheeｆfectofｏｒonｙ.

Wｈｙ?Ｗewiｌlhaveａformaｌprｅseｎtａｔionｏfｔhisｐroｂｌem.

XＩI.ＭｅasuｒeｓofGoodｎess－ｏf－Fit

Thｅrearedｉfferentｗaｙsｔoasｓｅｓsｔhegoodnｅsｓ－oｆ-fitofamodeｌ．

A.Ｒ2

Ｒ2ｉsaｈeurｉsticmeasurefｏrｔheoveｒallｇooｄnｅｓｓ－ｏf-ｆit．Itdｏeｓｎｏthaｖeａnassociatｅdteｓｔsｔａt(yī)iｓtｉｃ.

R２mｅasureｓtｈeproportｉonoｆｔｈevarｉaｎｃeｉntｈedｅpenｄentvaｒiablethａt(yī)ｉｓ“eｘｐlainｅd”byｔhｅmodel：

R２＝

B.ＭｏdelF-teｓt

TｈｅｍodｅlF－tesｔteｓｔsthejｏinｔｈypoｔｈesestｈａt(yī)allthemｏdelcoeｆfｉcieｎtsｅｘcｅｐtfｏｒｔhｅcｏnstanttermａrezerｏ.

DegreｅｓoｆfｒeeｄomsassociａｔｅｄwiththeｍodelF-ｔeｓｔ:

Nｕmeraｔｏr:p－１

Ｄenｏminaｔor：n-p．

C.ｔ-testsｆｏrinｄｉｖidualpａrａｍeｔers

At-tｅsｔfｏranｉｎdividuａｌｐaｒａmeteｒtesｔsｔｈeｈypothesｉｓｔhａt(yī)apａｒtｉcｕlａrcｏｅfficｉeｎｔiseqｕaltoａｐarｔiｃulaｒnｕmｂｅｒ(ｃomｍoｎlyｚero）.

tｋ=(bk－k0）/SEk,wheｒｅＳＥkiｓthｅ(k,k)elementofＭＳE(Ｘ’X）-1,withdeｇrｅeoffreｅｄom=ｎ-p.

D.IncｒeｍentalR2

Reｌaｔivetoarestｒiｃtedmｏｄel，theｇainiｎR２forｔhｅuｎrestricteｄmoｄel:

R2=Ru2-Rｒ2

?E.Ｆ-teｓtsforNesｔedＭｏｄel

IｔistheｍｏstgeneralforｍofF-testsandt-tests．

?

Itisequaltoａt(yī)-testifthｅuｎｒestrictｅdanｄrestrｉｃtedmｏdｅｌsｄifｆｅroｎlｙbyｏneｓｉngleparａmeter．

ItiｓequaltotｈemodeｌF－ｔestｉfwｅｓｅttｈeｒeｓｔｒｉctｅdmodeltotheconstant-onlymoｄel．

[Ａskｓｔudenｔｓ]WhaｔａｒｅSST,SＳＥ,andＳSR,andｔheiraｓｓociatedｄeｇｒeｅｓｏffrｅｅdom,ｆｏrtｈｅconstaｎt－ｏnlyｍｏdeｌ？

NumeｒicａｌExaｍple

Asｏcioｌｏｇicaｌｓtuｄｙｉsinｔｅｒｅsｔｅdinｕｎdeｒsｔandingｔhesｏciａlｄｅtermｉnanｔsofmａt(yī)hemaｔicａlaｃhievｅmｅnｔamoｎgｈighsｃhｏoｌsｔuｄents.Yoｕarｅnoｗaskeｄｔoansｗeｒaｓeriesｏfqueｓｔｉonｓ.Ｔheｄataarereａlbｕｔhaveｂeentａilｏreｄforeduｃａt(yī)ｉｏnａｌpurpｏses.Ｔｈeｔotalnumbｅroｆｏbｓervatｉonｓｉs４0０．Ｔhevariablesaredefineｄａs：

y：mathscorｅ

ｘ1:ｆａt(yī)ｈｅｒ＇seducatiｏn

x２:ｍｏthｅr＇sｅｄucaｔiｏn

ｘ3:fａｍiｌｙ'sｓｏcioecoｎoｍｉcｓｔａｔus

x4：nｕmberｏfｓｉｂlings

x5:ｃlａssrａnk

x6：pａｒeｎtｓ'totａleｄｕcatｉｏn(ｎoｔe:ｘ6

=

ｘ1

＋

x2)

Ｆoｒtｈefoｌlowingregresｓiｏnmodels，ｗeknoｗ:

Table１

ＳＳＴ?ＳＳＲ?SSE?DFR2

(1）yon（1x1x２x3x４)?３486３?4２01

(2)yon(1x6x３x４)?34８６3???396 .10６５

(3)yon(１x6x３x4x5）?3486３?１０426?24437?395?.2９9１

（4)x5ｏn（１ｘ6ｘ3ｘ4)???2６975３ ３96?．0210

1.PｌeasｅfillthｅmiｓsinｇceｌlsiｎTａblｅ1.

2.Testtheｈyｐoｔｈｅｓｉstｈatｔheeｆfeｃtｓoｆfather'seｄｕcａt(yī)ioｎ(x１）aｎdｍother'ｓeｄuｃation(x2)ｏnｍathｓcｏreａrethｅsameafｔeｒcontrolｌｉnｇｆｏrｘ3ａndｘ4.

3.Tesｔｔheｈypotｈeｓisｔｈaｔｘ6，x３ａndx４inMｏdel(２)ａllhａｖeaｚeroｅfｆectoｎy．