deep learning for natural language processing深度學習用于自然語言處理

DeepLearningforNaturalLanguageProcessingJonathanMugan,PhDNLPCommunityDayJune4,2015OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedTheimportanceoffindingdumbmistakesTheimportanceoffindingdumbmistakesOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedDeeplearningenablessub-symbolicprocessingSymbolicsystemscanbebrittle.Iboughtacar.<i><bought><a><car><.>Youhavetoremembertorepresent“purchased”and“automobile.”Whatabout“truck”?Howdoyouencodethemeaningoftheentiresentence?DeeplearningbeginswithalittlefunctionItallstartswithahumblelinearfunctioncalledaperceptron.weight1?input1weight2?input2weight3?input3sum?Perceptron:Ifsum>threshold:output1Else:output0Example:Theinputscanbeyourdata.Question:ShouldIbuythiscar?0.2?gasmilage0.3?horepower0.5?numcupholderssum?Perceptron:Ifsum>threshold:buyElse:walkTheselittlefunctionsarechainedtogetherDeeplearningcomesfromchainingabunchoftheselittlefunctionstogether.Chainedtogether,theyarecalledneurons.whereTocreateaneuron,weaddanonlinearitytotheperceptrontogetextrarepresentationalpowerwhenwechainthemtogether.Ournonlinearperceptronis

sometimescalledasigmoid.

PlotofasigmoidSingleartificialneuronOutput,orinputto

nextneuronweight1?input1weight2?input2weight3?input3Three-layeredneuralnetworkAbunchofneuronschainedtogetheriscalledaneuralnetwork.Layer2:hiddenlayer.Called

thisbecauseitisneitherinput

noroutput.Layer3:output.E.g.,cat

ornotacat;buythecaror

walk.Layer1:inputdata.Can

bepixelvaluesorthenumber

ofcupholders.Thisnetworkhasthreelayers.(Someedgeslighter

toavoidclutter.)[16.2,17.3,?52.3,11.1]TrainingwithsupervisedlearningSupervisedLearning:Youshowthenetworkabunchofthingswithalabelssayingwhattheyare,andyouwantthenetworktolearntoclassifyfuturethingswithoutlabels.Example:herearesomepicturesofcats.Tellmewhichoftheseotherpicturesareofcats.Totrainthenetwork,wanttofindtheweightsthatcorrectlyclassifyallofthetrainingexamples.Youhopeitwillworkonthetestingexamples.DonewithanalgorithmcalledBackpropagation[Rumelhartetal.,1986].[16.2,17.3,?52.3,11.1]TrainingwithsupervisedlearningSupervisedLearning:Youshowthenetworkabunchofthingswithalabelssayingwhattheyare,andyouwantthenetworktolearntoclassifyfuturethingswithoutlabels.

[16.2,17.3,?52.3,11.1]Learningislearningthe

parametervalues.WhyGoogle’sdeeplearningtoolboxiscalledTensorFlow.DeeplearningisaddingmorelayersThereisnoexactdefinitionof

whatconstitutes“deeplearning.”

Thenumberofweights(parameters)isgenerallylarge.Somenetworkshavemillionsofparametersthatarelearned.(Someedgesomitted

toavoidclutter.)[16.2,17.3,?52.3,11.1]RecallourstandardarchitectureLayer2:hiddenlayer.Called

thisbecauseitisneitherinput

noroutput.Layer3:output.E.g.,cat

ornotacat;buythecaror

walk.Layer1:inputdata.Can

bepixelvaluesorthenumber

ofcupholders.Isthisacat?[16.2,17.3,?52.3,11.1]NeuralnetswithmultipleoutputsOkay,butwhatkindofcatisit?

Introduceanewnode

calledasoftmax.Probabilitya

housecatProbabilitya

lionProbabilitya

pantherProbabilitya

bobcatJustnormalizetheoutputoverthesumoftheotheroutputs(usingtheexponential).Givesaprobability.[16.2,17.3,?52.3,11.1]Learningwordvectors

Fromthesentence,“Themanranfast.”

Probability

of“fast”Probability

of“slow”Probability

of“taco”Probability

of“bobcat”Learnsavectorforeachwordbasedonthe“meaning”inthesentenceby

tryingtopredictthenextword[Bengioetal.,2003].Thesenumbersupdatedalongwiththeweightsandethevectorrepresentationsofthewords.Comparingvectorandsymbolicrepresentations

Vectorshaveasimilarityscore.Atacoisnotaburritobutsimilar.Symbolscanbethesameornot.AtacoisjustasdifferentfromaburritoasaToyota.Vectorshaveinternalstructure[Mikolovetal.,2013].Italy–Rome=France–ParisKing–Queen=Man–WomanSymbolshavenostructure.Symbolsarearbitrarilyassigned.Meaningrelativetoothersymbols.Vectorsaregroundedinexperience.Meaningrelativetopredictions.Abilitytolearnrepresentationsmakesagentslessbrittle.OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedEncodingsentencemeaningintoavectorh0The“Thepatientfell.”Encodingsentencemeaningintoavectorh0Theh1patient“Thepatientfell.”Encodingsentencemeaningintoavectorh0Theh1patienth2fell“Thepatientfell.”EncodingsentencemeaningintoavectorLikeahiddenMarkovmodel,butdoesn’tmaketheMarkovassumptionandbenefitsfromavectorrepresentation.h0Theh1patienth2fellh3.“Thepatientfell.”DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.

Elh3DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.

Elh3h4DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.

Elh3pacienteh4DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.

Elh3pacienteh4cayóh5.h5[Choetal.,2014]Itkeepsgeneratinguntilitgeneratesastopsymbol.GeneratingimagecaptionsConvolutionalneuralnetworkAnh0angryh1sisterh2.h3[KarpathyandFei-Fei,2015][Vinyalsetal.,2015]Imagecaptionexamples[KarpathyandFei-Fei,2015]See:Attention[Bahdanauetal.,2014]Elh3pacienteh4cayóh5.h5h0Theh1patienth2fellh3.RNNsandStructureLearningThesearesometimescalledseq2seqmodels.Inadditiontomachinetranslationandgeneratingcaptionsforimages,canbeusedtolearnjustaboutanykindofstructureyou’dwant,aslongasyouhavelotsoftrainingdata.OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedDeeplearningandquestionansweringRNNsanswerquestions.WhatisthetranslationofthisphrasetoFrench?Whatisthenextword?Attentionisusefulforquestionanswering.Thiscanbegeneralizedtowhichfactsthelearnershouldpayattentiontowhenansweringquestions.DeeplearningandquestionansweringBobwenthome.Timwenttothejunkyard.Bobpickedupthejar.Bobwenttotown.Whereisthejar?A:townMemoryNetworks[Westonetal.,2014]Updatesmemoryvectorsbasedonaquestionandfindsthebestonetogivetheoutput.Theofficeisnorthoftheyard.Thebathisnorthoftheoffice.Theyardiswestofthekitchen.Howdoyougofromtheofficetothekitchen?A:south,eastNeuralReasoner[Pengetal.,2015]Encodesthequestionandfactsinmanylayers,andthefinallayerisputthroughafunctionthatgivestheanswer.OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedLimitationsofdeeplearningTheencodedmeaningisgroundedwithrespecttootherwords.Thereisnolinkagetothephysicalworld."ICubLugan01Reaching".LicensedunderCCBY-SA3.0viaWikipedia-TheiCubLimitationsofdeeplearningBobwenthome.Timwenttothejunkyard.Bobpickedupthejar.Bobwenttotown.Whereisthejar?A:townDeeplearninghasnounderstandingofwhatitmeansforthejartobeintown.Forexamplethatitcan’talsobeatthejunkyard.OrthatitmaybeinBob’scar,orstillinhishands.Theencodedmeaningisgroundedwithrespecttootherwords.Thereisnolinkagetothephysicalworld.LimitationsofdeeplearningImagineadudestandingonatable.Howwouldacomputerknowthatifyoumovethetableyoualsomovethedude?Likewise,howcouldacomputerknowthatitonlyrainsoutside?Or,asMarvinMinskyasks,howcouldacomputerlearnthatyoucanpullaboxwithastringbutnotpushit?LimitationsofdeeplearningImagineadudestandingonatable.Howwouldacomputerknowthatifyoumovethetableyoualsomovethedude?Likewise,howcouldacomputerknowthatitonlyrainsoutside?Or,asMarvinMinskyasks,howcouldacomputerlearnthatyoucanpullaboxwithastringbutnotpushit?Nooneknowshowtoexplainallofthesesituationstoacompute