遷移學(xué)習(xí)算法研究課件

【5A文】遷移學(xué)習(xí)算法研究【5A文】遷移學(xué)習(xí)算法研究TrainingDataClassifierUnseenData(…,long,T)good!Whatif…2023/10/62傳統(tǒng)監(jiān)督機(jī)器學(xué)習(xí)(1/2)[fromProf.QiangYang]TrainingClassifierUnseenData(傳統(tǒng)監(jiān)督機(jī)器學(xué)習(xí)(2/2)2023/10/63傳統(tǒng)監(jiān)督學(xué)習(xí)同源、獨(dú)立同分布兩個基本假設(shè)標(biāo)注足夠多的訓(xùn)練樣本在實(shí)際應(yīng)用中通常不能滿足！訓(xùn)練集測試集分類器訓(xùn)練集測試集分類器傳統(tǒng)監(jiān)督機(jī)器學(xué)習(xí)(2/2)2023/8/33傳統(tǒng)監(jiān)督學(xué)習(xí)同源遷移學(xué)習(xí)2023/10/64實(shí)際應(yīng)用學(xué)習(xí)場景HP新聞Lenovo新聞不同源、分布不一致人工標(biāo)記訓(xùn)練樣本，費(fèi)時耗力遷移學(xué)習(xí)運(yùn)用已有的知識對不同但相關(guān)領(lǐng)域問題進(jìn)行求解的一種新的機(jī)器學(xué)習(xí)方法放寬了傳統(tǒng)機(jī)器學(xué)習(xí)的兩個基本假設(shè)遷移學(xué)習(xí)2023/8/34實(shí)際應(yīng)用學(xué)習(xí)場景HP新聞Leno遷移學(xué)習(xí)場景(1/4)2023/10/65遷移學(xué)習(xí)場景無處不在遷移知識遷移知識圖像分類HP新聞Lenovo新聞新聞網(wǎng)頁分類遷移學(xué)習(xí)場景(1/4)2023/8/35遷移學(xué)習(xí)場景無處不在遷移學(xué)習(xí)場景(2/4)異構(gòu)特征空間2023/10/66Theappleisthepomaceousfruitoftheappletree,speciesMalusdomesticaintherosefamilyRosaceae...BananaisthecommonnameforatypeoffruitandalsotheherbaceousplantsofthegenusMusawhichproducethiscommonlyeatenfruit...Training:TextFuture:ImagesApplesBananas[fromProf.QiangYang]XinJin,FuzhenZhuang,SinnoJialinPan,ChangyingDu,PingLuo,QingHe:HeterogeneousMulti-taskSemanticFeatureLearningforClassification.CIKM2015:1847-1850.遷移學(xué)習(xí)場景(2/4)異構(gòu)特征空間2023/8/36TheTestTestTrainingTrainingClassifierClassifier72.65%DVDElectronicsElectronics84.60%ElectronicsDrop!遷移學(xué)習(xí)場景(3/4)2023/10/67[fromProf.QiangYang]TestTestTrain遷移學(xué)習(xí)場景(4/4)2023/10/68DVDElectronicsBookKitchenClothesVideogameFruitHotelTeaImpractical![fromProf.QiangYang]遷移學(xué)習(xí)場景(4/4)2023/8/38DVDElectroOutlineConceptLearningforTransferLearningConceptLearningbasedonNon-negativeMatrixTri-factorizationforTransferLearningConceptLearningbasedonProbabilisticLatentSemanticAnalysisforTransferLearningTransferLearningusingAuto-encodersTransferLearningfromMultipleSourceswithAutoencoderRegularizationSupervisedRepresentationLearning:TransferLearningwithDeepAuto-encoders2023/10/69OutlineConceptLearningforTrConceptLearningbasedonNon-negativeMatrixTri-factorizationforTransferLearning2023/10/6ConceptLearningforTransferLearning10ConceptLearningbasedonNon-Introduction2023/10/6ConceptLearningforTransferLearning11Manytraditionallearningtechniquesworkwellonlyundertheassumption:Trainingandtestdatafollowthesamedistribution

Training(labeled)ClassifierTest(unlabeled)FromdifferentcompaniesEnterpriseNewsClassification:includingtheclasses“ProductAnnouncement”,“Businessscandal”,“Acquisition”,……Productannouncement:HP'sjust-releasedLaserJetProP1100printerandtheLaserJetProM1130andM1210multifunctionprinters,price…performance

...AnnouncementforLenovoThinkPad

ThinkCentre–price$150offLenovoK300desktopusingcouponcode...LenovoThinkPad

ThinkCentre–price$200offLenovoIdeaPadU450plaptopusing....theirperformanceHPnewsLenovonewsDifferentdistributionFail!Introduction2023/8/3ConceptLeMotivation(1/3)2023/10/6ConceptLearningforTransferLearning12ExampleAnalysis

Productannouncement:HP'sjust-releasedLaserJetProP1100printerandtheLaserJetProM1130andM1210multifunctionprinters,price…performance

...AnnouncementforLenovoThinkPad

ThinkCentre–price$150offLenovoK300desktopusingcouponcode...LenovoThinkPad

ThinkCentre–price$200offLenovoIdeaPadU450plaptopusing....theirperformanceHPnewsLenovonewsProductwordconceptLaserJet,printer,price,performanceThinkPad,ThinkCentre,price,performanceRelatedProductannouncementdocumentclass:Sharesomecommonwords:announcement,price,performance…indicateMotivation(1/3)2023/8/3ConcepMotivation(2/3)2023/10/6ConceptLearningforTransferLearning13ExampleAnalysis:

HPLaserJet,printer,price,performanceetal.LenovoThinkpad,Thinkcentre,price,performanceetal.Thewordsexpressingthesamewordconceptaredomain-dependent

ProductProductannouncementwordconceptindicatesTheassociationbetweenwordconceptsanddocumentclassesisdomain-independent

Motivation(2/3)2023/8/3ConcepMotivation(3/3)2023/10/6ConceptLearningforTransferLearning14Furtherobservations:Differentdomainsmayusesamekeywordstoexpressthesameconcept(denotedasidenticalconcept)Differentdomainsmayalsousedifferentkeywordstoexpressthesameconcept(denotedasalikeconcept)Differentdomainsmayalsohavetheirowndistinctconcepts(denotedasdistinctconcept)TheidenticalandalikeconceptsareusedasthesharedconceptsforknowledgetransferWetrytomodelthesethreekindsofconceptssimultaneouslyfortransferlearningtextclassificationMotivation(3/3)2023/8/3ConcepPreliminaryKnowledge2023/10/6ConceptLearningforTransferLearning15Basicformulaofmatrixtri-factorization:wheretheinputXistheword-documentco-occurrencematrix

denotesconceptinformation,mayvaryindifferentdomainsFdenotesthedocumentclassificationinformation

indeedistheassociationbetweenwordconceptsanddocumentclasses,mayretainstablecrossdomainsGSPreliminaryKnowledge2023/8/3CPreviousmethod-MTrickinSDM2010(1/2)2023/10/6ConceptLearningforTransferLearning16SketchmapofMTrick

SourcedomainXs

FsGsFtGtTargetdomainXtSKnowledgeTransferConsideringthealikeconcepts Previousmethod-MTrickinSDMTrick(2/2)OptimizationproblemforMTrick2023/10/6ConceptLearningforTransferLearning17G0isthesupervisioninformationtheassociationSissharedasbridgetotransferknowledgeDualTransferLearning(Longetal.,SDM2012),consideringidenticalandalikeconceptsMTrick(2/2)OptimizationproblTriplexTransferLearning(TriTL)(1/5)2023/10/6ConceptLearningforTransferLearning18Furtherdividethewordconceptsintothreekinds:

F1,identicalconcepts;F2,alikeconcepts;F3,distinctconceptsInput:ssourcedomainXr(1≤r≤s)withlabelinformation,ttargetdomainXr(s+1≤r≤s+t)WeproposeTriplexTransferLearningframeworkbasedonmatrixtri-factorization(TriTLforshort)

TriplexTransferLearning(TriF1,S1andS2

aresharedasthebridgeforknowledgetransferacrossdomainsThesupervisioninformationisintegratedbyGr(1≤r≤s)insourcedomainsTriTL(2/5)OptimizationProblem

2023/10/6ConceptLearningforTransferLearning19F1,S1andS2aresharedasthTriTL(3/5)Wedevelopanalternativelyiterativealgorithmtoderivethesolutionandtheoreticallyanalyzeitsconvergence 2023/10/6ConceptLearningforTransferLearning20TriTL(3/5)WedevelopanalterTriTL(4/5)Classificationontargetdomains When1≤r≤s,Grcontainsthelabelinformation,soweremainitunchangedduringtheiterationswhenxibelongstoclassj,thenGr(i,j)=1,elseGr(i,j)=0Aftertheiteration,weobtaintheoutputGr(s+1≤r≤s+t),thenwecanperformclassificationaccordingtoGr2023/10/6ConceptLearningforTransferLearning21TriTL(4/5)ClassificationontTriTL(5/5)AnalysisofAlgorithmConvergence Accordingtothemethodologyofconvergenceanalysisinthetwoworks[Leeetal.,NIPS’01]and[Dingetal.,KDD’06],thefollowingtheoremholds.Theorem(Convergence):Aftereachroundofcalculatingtheiterativeformulas,theobjectivefunctionintheoptimizationproblemwillconvergemonotonically.2023/10/6ConceptLearningforTransferLearning22TriTL(5/5)AnalysisofAlgorit2023/10/6ConceptLearningforTransferLearning23rec.autosrec.motorcyclesrec.baseballrec.hockeysci.cryptsic.electronicssci.medsci.spacecomp.graphicscomp.sys.ibm.pc.hardwarecomp.sys.mac.hardwarecomp.windows.xtalk.politics.misctalk.politics.gunstalk.politics.mideasttalk.religion.miscrecscicomptalkDataPreparation(1/3)20Newsgroups Fourtopcategories,eachtopcategorycontainsfoursub-categories SentimentClassification,fourdomains:books,dvd,electronics,kitchenRandomlyselecttwodomainsassources,andtherestastargets,then6problemscanbeconstructed

2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning24rec.autosrec.motorcyclesrec.baseballrec.hockeysci.cryptsic.electronicssci.medsci.spacerec+sci-baseballcrypySourcedomainautosspaceTargetdomainFortheclassificationproblemwithonesourcedomainandonetargetdomain,wecanconstruct144()

problemsDataPreparation(2/3)Constructclassificationtasks(TraditionalTL)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning25Constructnewtransferlearningproblemsrec.autosrec.motorcyclesrec.baseballrec.hockeysci.cryptsic.electronicssci.medsci.spacerec+sci-baseballcrypyautosspacecomp.graphicscomp.sys.ibm.pc.hardwarecomp.sys.mac.hardwarecomp.windows.xtalk.politics.misctalk.politics.gunstalk.politics.mideasttalk.religion.misccomptalkautosgraphicsMoredistinctconceptsmayexist!DataPreparation(3/3)SourcedomainTargetdomain2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning26ComparedAlgorithmsTraditionallearningAlgorithmsSupervisedLearning:LogisticRegression(LR)[Davidetal.,00]SupportVectorMachine(SVM)[Joachims,ICML’99]Semi-supervisedLearning:TSVM[Joachims,ICML’99]TransferlearningMethods:CoCC[Daietal.,KDD’07],DTL[Longetal.,SDM’12]Classificationaccuracyisusedastheevaluationmeasure2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning27ExperimentalResults(1/3)SorttheproblemswiththeaccuracyofLRDegreeoftransferdifficultyeasierGenerally,thelowerofaccuracyofLRcanindicatethehardertotransfer,whilethehigheronesindicatetheeasiertotransferharder2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning28ExperimentalResults(2/3)ComparisonsamongTriTL,DTL,MTrick,CoCC,TSVM,SVMandLRondatasetrecvs.sci(144problems)TriTLcanperformwelleventheaccuracyofLRislowerthan65%2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning29ExperimentalResults(3/3)Resultsonnewtransferlearningproblems,weonlyselecttheproblems,whoseaccuraciesofLRarebetween(50%,55%](Onlyslightlybetterthanrandomclassification,thustheymightbemuchmoredifficult).Weobtain65problemsTriTLalsooutperformsallthebaselines2023/8/3ConceptLearningforTConclusions2023/10/6ConceptLearningforTransferLearning30Explicitlydefinethreekindsofwordconcepts,i.e.,identicalconcept,alikeconceptanddistinctconceptProposeageneraltransferlearningframeworkbasedonnonnegativematrixtri-factorization,whichsimultaneouslymodelthethreekindsofconcepts(TriTL)Extensiveexperimentsshowtheeffectivenessoftheproposedapproach,especiallywhenthedistinctconceptsmayexistConclusions2023/8/3ConceptLeaConceptLearningbasedonProbabilisticLatentSemanticAnalysisforTransferLearning2023/10/6ConceptLearningforTransferLearning31ConceptLearningbasedonProb2023/10/6ConceptLearningforTransferLearning32MotivationProductannouncement:HP'sjust-releasedLaserJetProP1100printerandtheLaserJetProM1130andM1210multifunctionprinters,price…performance

...AnnouncementforLenovoThinkPad

ThinkCentre–price$150offLenovoK300desktopusingcouponcode...LenovoThinkPad

ThinkCentre–price$200offLenovoIdeaPadU450plaptopusing....theirperformanceHPnewsLenovonewsProductwordconceptLaserJet,printer,price,performanceThinkPad,ThinkCentre,price,performanceRelatedProductannouncementdocumentclass:Sharesomecommonwords:announcement,price,performance…indicateRetrospecttheexample

2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning33SomenotationsddocumentydocumentclasszwordconceptSomedefinitionse.g.,p(price|Product),p(LaserJet|Product,)wwordrdomaine.g,p(Product|Productannouncement)PreliminaryKnowledge(1/3)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning34PreliminaryKnowledge(2/3)ProductLaserJet,printer,announcement,price,ThinkPad,ThinkCentre,announcement,priceProductannouncementp(w|z,r1)p(w|z,r2)p(z|y)p(w|z,r1)≠p(w|z,r2)E.g.,p(LaserJet|Product,HP)≠p(LaserJet|Product,Lenovo)p(z|y,r1)=p(z|y,r2)E.g.,p(Product|Productannoucement,HP)=p(Product|Productannoucement,Lenovo)Alikeconcept2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning35DualPLSA

(D-PLSA)Jointprobabilityoverallvariablesp(w,d)=p(w|z)p(z|y)p(d|y)p(y)GivendatadomainX,theproblemofmaximumloglikelihoodislogp(X;θ)=logΣz

p(Z,X;θ)

θ

includesalltheparametersp(w|z),p(z|y),p(d|y),p(y).Z

denotesallthelatentvariablesPreliminaryKnowledge(3/3)TheproposedtransferlearningalgorithmbasedonD-PLSA,denotedasHIDC2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning36Identicalconceptp(w|za)p(za|y)AlikeconceptTheextensionandintensionaredomainindependentp(w|zb,r)p(zb|y)HIDC(1/3)Theextensionisdomaindependent,whiletheintensionisdomainindependent2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning37Distinctconceptp(w|zc,r)p(zc|y,r)ThejointprobabilitiesofthesethreegraphicalmodelsHIDC(2/3)Theextensionandintensionarebothdomaindependent2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning38Givens+t

datadomainsX={X1,…,Xs,Xs+1,…,Xs+t},withoutlossofgenerality,thefirstsdomainsaresourcedomains,andthelefttdomainsaretargetdomainsConsiderthethreekindsofconcepts:TheLog

likelihoodfunctionislogp(X;θ)=logΣz

p(Z,X;θ)

θ

includesallparametersp(w|za),p(w|zb,r),p(w|zc,r),p(za|y),p(zb|y),p(zc|y,r),p(d|y,r),p(y|r),p(r).HIDC(3/3)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning39UsetheEMalgorithmtoderivethesolutionsEStep:ModelSolution(1/4)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning40M

Step：ModelSolution(2/4)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning41Semi-supervisedEMalgorithm:whenrisfromsourcedomains,thelabeledinformationp(d|y,r)isknownandp(y|r)

canbeinferedp(d|y,r)=1/ny,r,ifdbelongsyindomainr,ny,risthenumberofdocumentsinclassyindomainr,else

p(d|y,c)=0p(y|r)=ny,r/nr

,nr

isthenumberofdocumentsindomainr

whenrisfromsourcedomains,p(d|y,r)andp(y|r)keepunchangedduringtheiterations,whichsupervisetheoptimizingprocessModelSolution(3/4)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning42ClassificationfortargetdomainsAfterweobtainthefinalsolutionsofp(w|za),p(w|zb,r),p(w|zc,r),p(za|y),p(zb|y),p(zc|y,r),p(d|y,r),p(y|r),p(r)Wecancomputetheconditionalprobabilities:

ThenthefinalpredictionisDuringtheiterations,alldomainssharep(w|za),p(za|y),p(zb|y),

whichactasthebridgeforknowledgetransferModelSolution(4/4)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning43BaselinesComparedAlgorithmsSupervisedLearning:LogisticRegression(LG)[Davidetal.,00]SupportVectorMachine(SVM)[Joachims,ICML’99]Semi-supervisedLearning:TSVM[Joachims,ICML’99]TransferLearning:CoCC[Daietal.,KDD’07]CD-PLSA[Zhuangetal.,CIKM’10]DTL[Longetal.,SDM’12]OurMethodsHIDCMeasure:classificationaccuracy2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning44Resultsonnewtransferlearningproblems,weselecttheproblems,whoseaccuraciesofLRarehigherthan50%,then334problemsareobtainedExperimentalResults(1/5)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning45Resultsonnewtransferlearningproblems,weselecttheproblems,whoseaccuraciesofLRarehigherthan50%,then334problemsareobtainedExperimentalResults(2/5)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning46ExperimentalResults(3/5)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning47Sourcedomain:S

(rec.autos,

sci.space)，Targetdomain:T(rec.sport.hockey,talk.politics.mideast)STSTDistinctconceptSTAlikeconceptExperimentalResults(4/5)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning48ExperimentalResults(5/5)Indeed,theproposedprobabilisticmethodHIDCisalsobetterthanTriTLThismayduetothereasonthatthereismoreclearerprobabilisticexplanationofHIDCp1(z,y)=p2(z,y)orp1(z|y)=p2(z|y)whichisbetter?p(z|y)p(y)2023/8/3ConceptLearningforT2023/10/6ConceptLearningforTransferLearning49[1]FuzhenZhuang,PingLuo,HuiXiong,QingHe,YuhongXiong,ZhongzhiShi:ExploitingAssociationsbetweenWordClustersandDocumentClassesforCross-DomainTextCategorization.SDM2010,pp.13-24.[2]FuzhenZhuang,PingLuo,ZhiyongShen,QingHe,YuhongXiong,ZhongzhiShi,HuiXiong:CollaborativeDual-PLSA:miningdistinctionandcommonalityacrossmultipledomainsfortextclassification.CIKM2010,pp.359-368.[3]FuzhenZhuang,PingLuo,ZhiyongShen,QingHe,YuhongXiong,ZhongzhiShi,HuiXiong:MiningDistinctionandCommonalityacrossMultipleDomainsUsingGenerativeModelforTextClassification.IEEETrans.Knowl.DataEng.24(11):2025-2039(2012).[3]FuzhenZhuang,PingLuo,ChangyingDu,QingHe,ZhongzhiShi:Triplextransferlearning:exploitingbothsharedanddistinctconceptsfortextclassification.WSDM2013,pp.425-434.[4]FuzhenZhuang,PingLuo,PeifengYin,QingHe,ZhongzhiShi.:ConceptLearningforCross-domainTextClassification:aGeneralProbabilisticFramework.IJCAI2013,pp.1960-1966.References2023/8/3ConceptLearningforTOutlineConceptLearningforTransferLearningConceptLearningbasedonNon-negativeMatrixTri-factorizationforTransferLearningConceptLearningbasedonProbabilisticLatentSemanticAnalysisforTransferLearningTransferLearningusingAuto-encodersTransferLearningfromMultipleSourceswithAutoencoderRegularizationSupervisedRepresentationLearning:TransferLearningwithDeepAuto-encoders2023/10/650OutlineConceptLearningforTrTransferLearningfromMultipleSourceswithAutoencoderRegularization2023/10/6TransferLearningUsingAuto-encoders51TransferLearningfromMultipl2023/10/652Motivation(1/2)TransferlearningbasedonoriginalfeaturespacemayfailtoachievehighperformanceonTargetdomaindataWeconsidertheautoencodertechniquetocollaborativelyfindanewrepresentationofbothsourceandtargetdomaindataElectronicsVideoGames

Compact;easytooperate;verygoodpicture,excited

aboutthequality;lookssharp!Averygood

game!Itisactionpacked

andfullofexcitement.Iamverymuchhooked

onthisgame.52TransferLearningUsingAuto-encoders2023/8/352Motivation(1/2)TransPreviousmethodsoftentransferfromonesourcedomaintoonetargetdomainWeconsidertheconsensusregularizedframeworkforlearningfrommultiplesourcedomainsDVDBookKitchenElectronicsWeproposeatransferlearningframeworkofconsensusregularizationautoencoderstolearnfrommultiplesourcesMotivation(2/2)2023/10/6TransferLearningUsingAuto-encoders53PreviousmethodsoftentransfeAutoencoderNeuralNetworkMinimizingthereconstructionerrortoderivethesolution:whereh,garenonlinearactivationfunction,e.g.,Sigmoidfunction,forencodinganddecoding2023/10/6TransferLearningUsingAuto-encoders54AutoencoderNeuralNetworkMConsensusMeasure-(1/3)Example:three-classclassificationproblem,threeclassifierspredictinstancesf1f2f3f1f2f3x1111x2333x3222x4231x5313x61232023/10/6TransferLearningUsingAuto-encoders55ConstraintSource1:D1Source2:D2Source3:D3ConsensusMeasure-(1/3)ExamConsensusMeasure-(2/3)Example:three-classclassificationproblem,predictiononinstancex2023/10/6TransferLearningUsingAuto-encoders56Minimalentropy,MaximalConsensusMaximalentropy,MinimalConsensusEntropybasedConsensusMeasure(Luoetal.,CIKM’08)θiistheparametervectorofclassifieri,CistheclasslabelsetConsensusMeasure-(2/3)ExamConsensusMeasure-(3/3)Forsimplicity,theconsensusmeasureforbinaryclassificationcanberewrittenasInthiswork,weimposetheconsensusregularizationtoautoencoders,andtrytoimprovethelearningperformancefrommultiplesourcedomainssincetheireffectsonmakingthepredictionconsensusaresimilar.2023/10/6TransferLearningUsingAuto-encoders57ConsensusMeasure-(3/3)ForSomeNotationsSourcedomainsGivenrsourcedomains:,i.e.,

,.ThefirstcorrespondingdatamatrixisTargetdomainThecorrespondingdatamatrixisThegoalistotrainaclassifier

ftomakeprecisepredictionson.2023/10/6TransferLearningUsingAuto-encoders58SomeNotationsSourcedomaiFrameworkofCRAThedatafromallsourceandtargetdomainssharethesameencodinganddecodingweightsTheclassifierstrainedfromthenewrepresentationareregularizedtopredictthesameresultsontargetdomaindata2023/10/6TransferLearningUsingAuto-encoders59FrameworkofCRAThedatafrOptimizationProblemofCRATheoptimizationproblem:ReconstructionError2023/10/6TransferLearningUsingAuto-encoders60OptimizationProblemofCRAOptimizationProblemofCRATheoptimizationproblem:ConsensusRegularization2023/10/6TransferLearningUsingAuto-encoders61OptimizationProblemofCRAOptimizationProblemofCRATheoptimizationproblem:ThetotallossofsourceclassifiersoverthecorrespondingsourcedomaindatawiththehiddenrepresentationWeighdecayterm2023/10/6TransferLearningUsingAuto-encoders62OptimizationProblemofCRATheSolutionofCRAWeusethegradientdescentmethodtoderivethesolutionofallparameters?isthelearningrate.ThetimecomplexityisO(rnmk)Theoutput:theencodinganddecodingparameters,andsourceclassifierswithlatentrepresentation.2023/10/6TransferLearningUsingAuto-encoders63TheSolutionofCRAWeusetheTargetClassifierConstructionTwoScheme:Trainthesourceclassifiersbasedonandcombinethemas,whereCombineallthesourcedomaindataasZSandtrainaunifiedclassifierusinganysupervisedlearningalgorithms,e.g.,SVM,LogisticRegression(LR).ThetwoaccuraciesaredenotedasCRAvandCRAu,respectively2023/10/6TransferLearningUsingAuto-encoders64TargetClassifierConstructionDataSets-(1/2)ImageData(fromLuoetal.,CIKM08)(Someexamples)AB

A1A2A3A4B1B2B3B4Threesources:A1B1A2B2A3B3Targetdomain:A4B4Totally,96()3-sourcevs1-targetdomain(3vs1)probleminstancescanbeconstructedfortheexperimentalevaluation2023/10/6TransferLearningUsingAuto-encoders65DataSets-(1/2)ImageData(DataSets-(2/2)SentimentClassification(fromBlitzeretal.,ACL07)Four3-sourcevs1-targetdomainclassificationproblemsareconstructedDVDBookKitchenElectronicsTheaccuracyontargetdomaindataisusedastheevaluationmeasureBothSVMandLRareusedtotrainclassifiersonthenewrepresentation2023/10/6TransferLearningUsingAuto-encoders66DataSets-(2/2)SentimentClassAllComparedAlgorithmsBaselinesSupervisedlearningonoriginalfeatures:SVM[Joachims,ICML’99],LogisticRegression(LR)[Davidetal.,00]Embeddingmethodbasedonautoencoders(EAER)[Yuetal.,ECML’13]MarginalizedStackedDenoisingAutoencoders

(mSDA)[Chenetal.,ICML’12]TransferComponentAnalysis(TCA)[Panetal.,TNN’11]Transferlearningfrommultiplesources(CCR3)(Luoetal.,CIKM’08)Ourmethod:CRAvandCRAuForthemethodswhichcannothandlemultiplesources,wetraintheclassifiersfromeachsourcedomainandmergeddataofallsources(r+1accuracies).Finally,maximal,meanandminimalvaluesarereported.2023/10/6TransferLearningUsingAuto-encoders67AllComparedAlgorithmsBaselinTransferLearningwithMultipleSourcesviaConsensusRegularizationAutoencodersFuzhenZhuang,XiaohuCheng,SinnoJialinPan,WenchaoYu,QingHe,andZhongzhiShi68ExperimentalResults-(1/2)Resultson96imageclassificationproblemsTransferLearningwithMultiplTransferLearningwithMultipleSourcesviaConsensusRegularizationAutoencodersFuzhenZhuang,XiaohuCheng,SinnoJialinPan,WenchaoYu,QingHe,andZhongzhiShi69ExperimentalResults-(2/2)Resultson4sentimentclassificationproblemsTransferLearningwithMultiplConclusionsThewellknownrepresentationlearningtechniqueautoencoderisconsidered,andweformalizetheautoencodersandconsensusregularizationintoaunifiedoptimizationframeworkExtensivecomparisonexperimentsonimageandsentimentdataareconductedtoshowtheeffectivenessoftheproposealgorithm2023/10/6TransferLearningUsingAuto-encoders70ConclusionsThewellknownrSupervisedRepresentationLearning:TransferLearningwithDeepAuto-encoders2023/10/6TransferLearningUsingAuto-encoders71SupervisedRepresentationLearAutoencoderisanunsupervisedfeaturelearningalgorithm,whichcannoteffectivelymakeuseofthelabelinformationLimitationofBasicAutoencoderContributionofThisWorkWeextendAutoencodertomulti-layerstructure,andincorporatethelabelasonelayerMotivation2023/10/6TransferLearningUsingAuto-encoders72Autoencoderisanunsupervised源領(lǐng)域和目標(biāo)領(lǐng)域共享編碼和