參考文稿標(biāo)記化

?Text?EveryNLPtaskneedstodotextSegmen-ng/tokenizingwordsinrunningNormalizingwordSegmen-ngsentencesinrunningHowmanyFragments,filledSeuss’scatinthehatisdi?erentfromotherLemma:samestem,partofspeech,roughwordcatandcats=sameWordform:thefullinflectedsurfacecatandcats=di?erentHowmanytheylaybackontheSanFranciscograssandlookedatthestarsandType:anelementoftheToken:aninstanceofthattypeinrunningHow15tokens(or13types(or12)(orHowmanyN=numberofV=vocabulary=setof|V|isthesizeofthe

ChurchandGale(1990):|V|>Tokens=Types=2.42031113SimpleTokeniza4onin(InspiredbyKenChurch’sUNIXforGivenatextfile,outputthewordtokensandtheirtr-scSortinSortinalphabeticalMergeandcountMergeandcounteach194572195...

2561563....Thefirststep:tr-sc’A-Za-z’’\n’<shakes.txt|Thesecondstep:tr-sc’A-Za-z’’\n’<shakes.txt|sort|WhatWhathappenedMoreMergingupperandlowertr‘A-Z’‘a(chǎn)-z’<shakes.txt|tr–sc‘A-Za-z’‘\n’|sort|uniqSor-ngthetr‘A-Z’‘a(chǎn)-z’<shakes.txt|tr–sc‘A-Za-z’‘\n’|sort|uniq–c|sort–niadIssuesinFinland’s FinlandFinlandsFinland’swhat’re,I’m,isn’t Whatare,Iam,is HewlettPackard stateoftheart lower-caselowercaselowercaseSan onetokenorm.p.h., Tokeniza4on:languageL'ensembleonetokenorL?L’?LeWantl’ensembletomatchwithunGermannouncompoundsarenot‘lifeinsurancecompanyGermaninforma-onretrievalneedscompoundDan

Tokeniza4on:languageChineseandJapanesenospacesbetween莎拉波娃現(xiàn)在居住在美國(guó)東南 的佛羅里 lives FurthercomplicatedinJapanese,withmul-plealphabetsDates/amountsinmul-pleWordTokeniza4oninAlsocalledWordChinesewordsarecomposedofCharactersaregenerally1syllableand1Averagewordis2.4charactersStandardbaselinesegmenta-onMaximumMatching(alsocalledWordSegmenta4onGivenawordlistofChinese,andaStartapointeratthebeginningoftheFindthelongestwordindic-onarythatmatchesthestringstar-ngatpointerMovethepointeroverthewordinGotoDan

thecatinthehatthetabledowntherethetabledownDoesn’tgenerallyworkinButworksastoni