10-2-訓(xùn)練詞向量實(shí)踐

深度學(xué)習(xí)與自然語言處理Python自然語言處理第十章課前回顧詞向量word2vec詞向量模型CBOW和Skip-gram模型訓(xùn)練詞向量實(shí)踐02代碼實(shí)現(xiàn)訓(xùn)練詞向量實(shí)踐

下載word2vec：①Google（/dav/word2vec）②C++11（/jdeng/word2vec）③Java（/NLPchina/Word2VEC_java）④Python（/pypi/gensim）訓(xùn)練詞向量實(shí)踐

基本系統(tǒng)配置：①WindowsOriginal：cygwin②C++11：VS2013Linux/MacOS③任意版本word2vec

中文語料庫：維基百科中文語料（地址：/zhwiki/latest/zhwiki-latests-articles.xml.bz2）搜狗SouGouT訓(xùn)練詞向量實(shí)踐1、安裝Python3.6，安裝庫：gensim、jieba、hanziconv2、下載數(shù)據(jù)下載Word2vec.git下載中文維基百科數(shù)據(jù)下載jieba字典訓(xùn)練詞向量實(shí)踐3、載入相關(guān)庫#把一些警告的訊息暫時(shí)關(guān)掉importwarningswarnings.filterwarnings('ignore')#Utilities相關(guān)庫importosimportnumpyasnpimportmmapfromtqdmimporttqdm#可視化相關(guān)庫importjiebafromgensim.corporaimportWikiCorpusfromgensim.modelsimportword2vecfromhanziconvimportHanziConvimportmatplotlib.pyplotasplt訓(xùn)練詞向量實(shí)踐4、參數(shù)設(shè)置#文檔的根目錄路徑ROOT_DIR=os.getcwd()#訓(xùn)練/驗(yàn)證用的文檔路徑DATA_PATH=os.path.join(ROOT_DIR,"data")#模型目錄MODEL_PATH=os.path.join(ROOT_DIR,"model")#設(shè)定jieba自定義字典路徑JIEBA_DICTFILE_PATH=os.path.join(MODEL_PATH,"extra_dict","dict.txt.big")#設(shè)定字典jieba.set_dictionary(JIEBA_DICTFILE_PATH)訓(xùn)練詞向量實(shí)踐5、獲取corpus語料#將wiki數(shù)據(jù)集下載后進(jìn)行提取，且將xml轉(zhuǎn)換成plaintxtwiki_articles_xml_file=os.path.join(DATA_PATH,"zhwiki-20180220s-articles.xml.bz2")wiki_articles_txt_file=os.path.join(DATA_PATH,"zhwiki_plaintext.txt")#使用gensim.WikiCorpus來讀取wikiXML中的corpuswiki_corpus=WikiCorpus(wiki_articles_xml_file,dictionary={})#迭代提取出來的詞匯withopen(wiki_articles_txt_file,'w',encoding='utf-8')asoutput:text_count=0fortextinwiki_corpus.get_texts():#把詞匯寫進(jìn)文件中備用

output.write(''.join(text)+'\n')text_count+=1iftext_count%10000==0:print("目前已處理%d篇文章"%text_count)print("簡繁轉(zhuǎn)換已完畢,總共處理了%d篇文章!"%text_count)訓(xùn)練詞向量實(shí)踐6、進(jìn)行中文分詞與stop-word移除（1）#一個(gè)取得一個(gè)數(shù)的函數(shù)式defget_num_lines(fi文件行l(wèi)e_path):fp=open(file_path,"r+")buf=mmap.mmap(fp.fileno(),0)lines=0whilebuf.readline():lines+=1returnlines#進(jìn)行簡體轉(zhuǎn)繁體wiki_articles_zh_tw_file=os.path.join(DATA_PATH,"zhwiki_zh_tw.txt")wiki_articles_zh_tw=open(wiki_articles_zh_tw_file,"w",encoding="utf-8")#迭代轉(zhuǎn)換成plaintext的wiki文件,并透過HanziConv來進(jìn)行簡繁轉(zhuǎn)換withopen(wiki_articles_txt_file,"r",encoding="utf-8")aswiki_articles_txt:forlineintqdm(wiki_articles_txt,total=get_num_lines(wiki_articles_txt_file)):wiki_articles_zh_tw.write(HanziConv.toTraditional(line))print("成功簡繁轉(zhuǎn)換!")wiki_articles_zh_tw.close()#進(jìn)行中文分詞并同步停用詞過濾stops_word_file=os.path.join(ROOT_DIR,"stopwords.txt")訓(xùn)練詞向量實(shí)踐6、進(jìn)行中文分詞與stop-word移除（2）stopwordset=set()#讀取stopword詞典，并保存到stopwordset中withopen("stopwords.txt","r",encoding="utf-8")asstopwords:forstopwordinstopwords:stopwordset.add(stopword.strip('\n'))#保存分詞后的結(jié)果wiki_articles_segmented_file=os.path.join(DATA_PATH,"zhwiki_segmented.txt")wiki_articles_segmented=open(wiki_articles_segmented_file,"w",encoding="utf-8")#迭代轉(zhuǎn)換成繁體的wiki文檔,并透過jieba來進(jìn)行分詞withopen(wiki_articles_zh_tw_file,"r",encoding="utf-8")asCorpus:forsentenceintqdm(Corpus,total=get_num_lines(wiki_articles_zh_tw_file)):#forsentenceinCorpus:sentence=sentence.strip("\n")pos=jieba.cut(sentence,cut_all=False)forterminpos:iftermnotinstopwordset:wiki_articles_segmented.write(term+"")print("jieba分詞完畢，并已完成過濾詞工序!")wiki_articles_zh_tw_file.close()訓(xùn)練詞向量實(shí)踐7、訓(xùn)練向量詞fromgensim.modelsimportword2vec#可參考/gensim/models/word2vec.html更多運(yùn)用print("word2vec模型訓(xùn)練中...")#加載文件sentence=word2vec.Text8Corpus(wiki_articles_segmented_file)#設(shè)置參數(shù)和訓(xùn)練模型(Train)model=word2vec.Word2Vec(sentence,size=300,window=10,min_count=5,workers=4,sg=1)#保存模型word2vec_model_file=os.path.join(MODEL_PATH,"zhwiki_word2vec.model")model.wv.save_word2vec_format(word2vec_model_file,binary=True)#model.wv.save_word2vec_format("wiki300.model.bin",binary=True)print("Word2vec模型已存儲(chǔ)完畢")訓(xùn)練詞向量實(shí)踐8、詞向量實(shí)驗(yàn)fromgensim.models.keyedvectorsimportKeyedVectorsword_vectors=KeyedVectors.load_word2vec_format(word2vec_model_file,binary=True)print("詞匯相似詞前5排序")query_list=['校長']res=word_vectors.most_similar(query_list[0],topn=5)foriteminres:print(item[0]+","+str(item[1]))print("計(jì)算2個(gè)詞匯間的Cosine相似度")query_list=['爸爸','媽媽']res=word_vectors.similarity(query_list[0],query_list[1])print(res)query_list=['爸爸','老公','媽媽']print("%s之于%s，如%s之于"%(query_list[0],que