阻尼最小二乘法

阻尼最小二乘法(dampedleastsqures，又稱Levenberg-Marquardtalgorithm)theLevenberg-Marquardtalgorithm(LMA)[1]providesanumericalsolutiontotheproblemofminimizingafunction,generallynonlinear,overaspaceofparametersofthefunction.Theseminimizationproblemsariseespeciallyinleastsquarescurvefittingandnonlinearprogramming.TheLMAinterpolatesbetweentheGauss-Newtonalgorithm(GNA就是最小二乘)andthemethodofgradientdescent.TheLMAsmorerobustthantheGNA,whichmeansthatinmanycasesitfindsasolutionevenifitstartsveryfaroffthefinalminimum.Forwell-behavedfunctionsandreasonablestartingparameters,theLMAtendstobeabitslowerthantheGNA.LMAcanalsobeviewedasGauss-Newtonusingatrustregionapproach.However,theLMAfindsonlyalocalminimum,notaglobalminimum.^和最小二乘一樣。這是所有線性反演的通病。TheproblemTheprimaryapplicationoftheLevenber—Marquardtalgorithmisintheleastsquarescurvefittingproblem:givenasetofmempiricaldatumpairsofindependentanddependentvariables,(xi,yi),optimizetheparametersPofthemodelcurvef(x,p)sothatthesumofthesquaresofthedeviationsm=工也-0：/3)]2■i=lbecomesminimal.ThesolutionLikeothernumericminimizationalgorithms,theLevenberg-Marquardtalgorithmisaniterativeprocedure.Tostartaminimization,theuserhastoprovideaninitialguessfortheparametervector,p.Incaseswithonlyoneminimum,anuninformedstandardguesslikepT=(1,1,...,1)

willworkfine;incaseswithmultipleminima,thealgorithmconvergesonlyiftheinitialguessisalreadysomewhatclosetothefinalsolution.Ineachiterationstep,theparametervector,P,isreplacedbyanewestimate,P+estimate,P+5.Todeterminedthefunctions」'??「.二誨—■areapproximatedbytheirlinearizationsfgW+3)ef0,p)+Jidwhereisthegradient(row-vectorinthiscase)offwithrespecttop.Atitsminimum,thesumofsquares,S(p),thegradientofSwithrespectwillbezero.Theabovefirst-orderapproximationwillbezero.givesmstp+睥￡皿—KsB)一如i=iOrinvectornotation,Takingthederivativewithrespecttodandsettingtheresulttozerogives:(JTJ)5=JT[y-f(/3)]whereJistheJacobianmatrixwhoseithrowequalsJ.,andwherefandyarevectorswithithcomponent，'3.andy”respectively.Thisisasetoflinearequationswhichcanbesolvedfor5.上面是傳統(tǒng)的最小二乘法的線性方程。

Levenberg'scontributionistoreplacethisequationbya"dampedversion",(JTJ+AI)J=JT[y-f(/3)]whereIistheidentitymatrix,givingastheincrement,S,totheestimatedparametervector,fi.ofSwithrespecttopequalsvaluesof入，thestepwillbetakenapproximatelyinthedirectionofthegradient.Ifeitherthelengthofthecalculatedstep,6,orthereductionofsumofsquaresfromthelatestparameterfallbelowpredefinedlimits,iterationstopsandthevector,p,isconsideredtobethesolution.The(non-negative)dampingfactor,入，isadjustedateachiteration.IfreductionofSisrapid,asmallervaluecanbeused,bringingthealgorithmclosertotheGauss-Newtonalgorithm,whereasifaniterationgivesinsufficientreductionintheresidual,ofSwithrespecttopequalsvaluesof入，thestepwillbetakenapproximatelyinthedirectionofthegradient.Ifeitherthelengthofthecalculatedstep,6,orthereductionofsumofsquaresfromthelatestparameterfallbelowpredefinedlimits,iterationstopsandthevector,p,isconsideredtobethesolution.vector,p+6,lastparameterLevenbergsalgorithmhasthedisadvantagethatifthevalueofdampingfactor,入，islarge,JTJ+入Iisnotusedatall.Marquardtprovidedtheinsightthatwecanscaleeachcomponentofthegradientaccordingtothecurvaturesothatthereislargermovementalongthedirectionswherethegradientissmaller.Thisavoidsslowconvergenceinthedirectionofsmallgradient.Therefore,Marquardtreplacedtheidentitymatrix,I,withthediagonalmatrixconsistingofthediagonalelementsofJTJ,resultingintheLevenberg-Marquardtalgorithm:(J*+Adiag(jTj))B=JT[y-f(/3)].ChoiceofdampingparameterVariousmore-or-lessheuristicargumentshavebeenputforwardforthebestchoiceforthedampingparameter入.Theoreticalargumentsexistshowingwhysomeofthesechoicesguaranteedlocalconvergenceofthealgorithm;howeverthesechoicescanmaketheglobalconvergenceofthealgorithmsufferfromtheundesirablepropertiesofsteepest-descent,inparticularveryslowconvergenceclosetotheoptimum.Theabsolutevaluesofanychoicedependsonhowwell-scaledtheinitial

problemis.Marquardtrecommendedstartingwithavalue\0andafactorv>1.Initiallysetting入二入0andcomputingtheresidualsumofsquaresS(p)afteronestepfromthestartingpointwiththedampingfactorof入二入0andsecondlywith入0/v.Ifbothoftheseareworsethantheinitialpointthenthedampingisincreasedbysuccessivemultiplicationbyvuntilabetterpointisfoundwithanewdampingfactor。仇0vkforsomek.asdampingfactor.Ifuseofthedampingfactor入/vresultsinareductioninsquaredresidualthenthisistakenasthenewvalueof\(andthenewoptimumlocationistakenasthatobtainedwiththisdampingfactor)andtheprocesscontinues;ifusing入/vresultedinaworseresidual,butusing入resultedinabetterresidualthen入isleftunchangedandthenewoptimumistakenasthevalueobtainedwith入asdampingfactor.Exampleacos(bX)+bsin(aX)usinginGNUOctaveastheInthisexamplewetrytofitthefunctiony=theLevenberg-Marquardtalgorithmimplementedleasqrfunction.The3graphsFig1,2,3showprogressivelybetterfittingfortheparametersa=100,b=102usedintheinitialcurve.OnlywhentheparametersinFig