2022大數(shù)據(jù)技術(shù)ELK講義

大數(shù)據(jù)技術(shù)之ELKELK、ElasticSearch、名詞解釋之前版本的結(jié)構(gòu)Es6.0之后：一個索引中只有一個typedocument用戶存儲在es中的數(shù)據(jù)文檔元數(shù)據(jù)_index:文檔所在索引名稱_type:文檔所在類型名稱_id:文檔唯一id_uid:組合id，由_type和_id組成（6.x后，_type不再起作用，同_id）_source:文檔的原始Json數(shù)據(jù)，包括每個字段的內(nèi)容_all:將所有字段內(nèi)容整合起來，默認禁用（用于對所有字段內(nèi)容檢索）Index由具有相同字段的文檔列表組成，用于定義字段名和字段值，一個集群或elasticsearch由多個索引組成，例如可以按照日期生成多個索引，方便數(shù)據(jù)搜索具有相同特征文檔的集合（ES6之后一個索引中只能定義一個type）Field具有相同特性數(shù)據(jù)名稱類型名稱組成字符串text、keyword（不分詞）數(shù)值型longintegershortbytedoublefloathalf_floatscaled_float(長度短)布爾boolean日期Date二進制binary范圍類型long_rangedouble_rangedate_range(做數(shù)據(jù)范圍查詢)坐標(biāo)附近的人節(jié)點Node一個Elasticsearch的運行實例，時集群構(gòu)成的基本單元cluster由一個或多個節(jié)點組成，對外提供服務(wù)、Elasticsearch的APIRESTES需要HttpMethod、URI）（postman）Linux請求：[root@localcurl-XPOST'2:9200/atguig/doc'-i-H"Content-Type:application/json"-d'{"name":"haha","age":"10"}'相應(yīng)：HTTP/1.1201CreatedLocation:/atguig/doc/KF_t32QBxRaDZXTftAxgcontent-type:application/json;charset=UTF-8content-length:172{"_index":"atguig","_type":"doc","_id":"KF_t32QBxRaDZXTftAxg","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}KibanaDevCerebroES語法：ip:post/_cat/[args](?v|?format=json&pretty)（?v表示顯示字段說明,?format=json&pretty表示顯示成json格式）1、查看所有索引GET_cat/indices?v2、查看es集群狀態(tài)GET_cat/health?v3、集群節(jié)點健康查看GET_cat/nodes?v4、列出倒敘索引GET_cat/segment?v語法：GET_cluster/(args][?v|?format=json&pretty)（?v表示顯示字段說明,?format=json&pretty表示顯示成json格式）1、添加語法：PUT index2、查看索引信息語法：GET index3、刪除索引語法：DELETE index4、查看索引狀態(tài)語法：HEADindex名稱語法：GET index稱1、添加和修改語法：(PUT|POST) index稱/type稱/[id]?不加id自生成id2語法：DELETE index稱/type稱/[id]4查看語法：GET index稱/type稱/[id]、正排索引和倒排索引記錄文檔Id到文檔內(nèi)容、單詞的關(guān)聯(lián)關(guān)系docidcontent1尚硅谷是最好的培訓(xùn)機構(gòu)2php是世界上最好的語言3尚硅谷是如何誕生的記錄單詞到文檔id的關(guān)聯(lián)關(guān)系，包含：單詞詞典（TermDicTionary）：記錄所有文檔的單詞，一般比較大倒排索引（PostingList）：記錄單詞倒排列表的關(guān)聯(lián)信息例如：尚硅谷1、TermDictionary尚硅谷2、PostingListDocIdTFPositionOffset110<0,2>310<0,2>DocId：文檔id，文檔的原始信息TF：單詞頻率，記錄該詞再文檔中出現(xiàn)的次數(shù)，用于后續(xù)相關(guān)性算分Position：位置，記錄Field分詞后，單詞所在的位置，從0開始Offset：偏移量，記錄單詞在文檔中開始和結(jié)束位置，用于高亮顯示等3、內(nèi)存結(jié)構(gòu)B+Tree每個文檔字段都有自己的倒排索引、分詞分詞是指將文本轉(zhuǎn)換成一系列單詞（termortoken）的過程，也可以叫做文本分析，在es里面稱為AnalysisCharacterFilter對原始文本進行處理例：去除html標(biāo)簽、特殊字符等Tokenizer將原始文本進行分詞例：培訓(xùn)機構(gòu)-->培訓(xùn)，機構(gòu)TokenFilters分詞后的關(guān)鍵字進行加工例：轉(zhuǎn)小寫、刪除語氣詞、近義詞和同義詞等API1、直接指定測試（指定分詞器）Request:POST_analyzePOST_analyze{"analyzer":"standard","text":"hello1111"}Response:}}#分詞#開始偏移#結(jié)束偏移#單詞類型#位置"tokens":[{"hello","start_offset":"end_offset":5,"type":"<ALPHANUM>","position":0},{"token":"world","start_offset":6,"end_offset":11,"type":"<NUM>","position":1}]{2、針對索引的字段進行分詞測試（利用該字段的分詞器）Request：POSTatguigu/_analyzePOSTatguigu/_analyze{"field":"name","text":"helloworld"}{"tokens":[{"tokens":[{"hello","start_offset":"end_offset":5,"type":"<ALPHANUM>","position":0},{"token":"world","start_offset":6,"token":"world","start_offset":6,"end_offset":11,"type":"<ALPHANUM>","position":1}]}3、自定義分詞器Request:POST_analyzePOST_analyze{"tokenizer":"standard","filter":["lowercase"],"text":"HelloWORLD"}{"tokens":[{"tokens":[{"hello","start_offset":"end_offset":5,"type":"<ALPHANUM>","position":0},{"token":"world","start_offset":6,"end_offset":11,"type":"<ALPHANUM>","position":1}]}Elasticsearch分詞器（Analyzer）特點Standard（es默認）支持多語言，按詞切分并做小寫處理Simple按照非字母切分，小寫處理Whitespace按照空格來切分Stop去除語氣助詞，如the、an、的、這等Keyword不分詞Pattern正則分詞，默認\w+,即非字詞符號做分割符Language常見語言的分詞器（30+）分詞器名稱介紹特點地址IK實現(xiàn)中英文單詞切分自定義詞庫/medcl/elasticsearch-analysis-ikJiebapython支持分詞和詞性標(biāo)注支持繁體、自定義、并行分詞/sing1ee/elasticsearch-jieba-pluginHanlp由一系列模型于算法組成的java工具包普及自然語言處理在生產(chǎn)環(huán)境中的應(yīng)用/hankcs/HanLPTHULAC清華大學(xué)中文詞法分析工具包具有中文分詞和詞性標(biāo)注功能/microbun/elasticsearch-thulac-pluginCharacterFilters在進行Tokenizer之前對原始文本進行處理，如增加、刪除或替換字符等HTMLStrip去除html標(biāo)簽和轉(zhuǎn)換html實體Mapping字符串替換操作PatternReplace正則匹配替換POST_analyze{"tokenizer":"keyword","char_filter":["html_strip"],"text":"<div><h1>B⁺Trees</h1></div>"}positionPOST_analyze{"tokenizer":"keyword","char_filter":["html_strip"],"text":"<div><h1>B⁺Trees</h1></div>"}{"tokens":[{"tokens":[{"token":"""B+Trees""","start_offset":0,"start_offset":0,"end_offset":38,"type":"word","position":0}]}Filter對輸出的單詞（term）進行增加、刪除、修改等操作Lowercase將所有term轉(zhuǎn)換為小寫stop刪除stopwordsNGram和EdgeNGram連詞分割Synonym添加近義詞的termPOST_analyze{POST_analyze{"tokenizer":"standard","text":"aHelloWorld","filter":["stop","lowercase",{"type":"ngram","min_gram":3,"max_gram":4}]}{"tokens":[{"tokens":[{"hel","start_offset":"end_offset":7,"type":"<ALPHANUM>","position":1},{"token":"hell","start_offset":2,"end_offset":7,"type":"<ALPHANUM>","position":1},{"ell","start_offset":"end_offset":7,"type":"<ALPHANUM>","position":1},{"ello","start_offset":"end_offset":7,"type":"<ALPHANUM>","position":1},{"llo","start_offset":"end_offset":7,"type":"<ALPHANUM>","position":1},{"token":"wor","start_offset":8,"end_offset":13,"type":"<ALPHANUM>","position":2},{"token":"worl","start_offset":8,"end_offset":13,"type":"<ALPHANUM>","position":2},{"orl","start_offset":"end_offset":13,"type":"<ALPHANUM>","type":"<ALPHANUM>","position":2},{"token":"orld","start_offset":8,"end_offset":13,"type":"<ALPHANUM>","position":2},{"token":"rld","start_offset":8,"end_offset":13,"type":"<ALPHANUM>","position":2}]}PUTmy_analyzer{"settings":{"analysis":{PUTmy_analyzer{"settings":{"analysis":{"analyzer":{"my":{"tokenizer":"punctuation","type":"custom","char_filter":["emoticons"],"filter":["lowercase","english_stop"]}},"tokenizer":{"punctuation":{"type":"pattern","pattern":"[.,!?]"}},"char_filter":{"emoticons":{"type":"mapping","mappings":["mappings":[":)=>_happy_",":(=>_sad_"]}},"filter":{"english_stop":{"type":"stop","stopwords":"_english_"}}}}}測試：POSTmy_analyzer/_analyzePOSTmy_analyzer/_analyze{"analyzer":"my","text":"l'ma:)person,andyou?"}{{"tokens":[{"token":"l'ma_happy_person","start_offset":0,"end_offset":15,"type":"word","position":0},{"token":"andyou","start_offset":16,"end_offset":23,"type":"word","position":1}]}1、索引時分詞：創(chuàng)建或更新文檔時，會對相應(yīng)得文檔進行分詞(指定字段分詞)PUTmy_testPUTmy_test{“mappings”:{“title”:{“analyzer”:”whitespace”}}}}}2、查詢時分詞：查詢時會對查詢語句進行分詞POSTmy_test/_searchPOSTmy_test/_search{“query”:{“match”:{“message”:{“query”:”hello”,“analyzer”:”standard”}}}}##查詢指定分詞器PUTmy_test{“mappings”:{“title”:{“analyzer”:”whitespace”,}}}}}type、Mapping作用：定義數(shù)據(jù)庫中的表的結(jié)構(gòu)的定義，通過mapping來控制索引存儲數(shù)據(jù)的設(shè)置Index（FieldName）documentIdposition、打分等mapping不進行配置時，自動創(chuàng)建的mapping請求：GET/atguigu/_mapping響應(yīng)：##字段類型，字符串默認類型#子字段屬性設(shè)置#分詞類型（不分詞）#索引名稱#mapping設(shè)置#type名稱"atguigu":{"mappings":{"student":{"properties":{"clazz":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}{}}}}}}}}mapping請求：#索引名稱PUTmy_index{"mappings":{"doc":{"dynamic":false,"properties":{"title":{"type":"text"},"name":{"type":"keyword"},"age":{"type":"integer"}}}}}響應(yīng)：{{"acknowledged":true,"shards_acknowledged":true,"index":"my_index"}DynamicMappinges依靠json文檔字段類型來實現(xiàn)自動識別字段類型，支持的類型JSON類型es類型null忽略booleanboolean浮點類型float整數(shù)longobjectobjectarray由第一個非null值的類型決定string匹配為日期則設(shè)為data類型（默認開啟）floatlong（）注意：mapping中的字段類型一旦設(shè)定后，禁止修改原因：Lucene實現(xiàn)的倒排索引生成后不允許修改(提高效率)如果要修改字段的類型，需要從新建立索引，然后做reindex操作dynamictrue（）Falsestrict）cope_to作用:將字段的值賦值到目標(biāo)字段，實現(xiàn)類似_all的作用例如：1、創(chuàng)建mapping，包含copy_to字段PUTmy_indexPUTmy_index{"mappings":{"doc":{"properties":{"frist_name":{"type":"text","copy_to":"full_name"},"last_name":{"type":"text","copy_to":"full_name"},"full_name":{"type":"text"}}}}}2、創(chuàng)建文檔PUTmy_index/doc/1PUTmy_index/doc/1{"frist_name":"John","last_name":"Smith"}3、查詢文檔GETmy_index/_searchGETmy_index/_search{"query":{"match":{"full_name":{"query":"JohnSmith","operator":"and"}}}}Index屬性Index屬性，控制當(dāng)前字段是否索引，默認為true，即記錄索引，false不記錄，即不可以搜索，比如：手機號、身份證號等敏感信息，不希望被檢索例如：1、創(chuàng)建mappingPUTmy_indexPUTmy_index{"mappings":{"doc":{"properties":{"cookie":{"type":"text","index":false}}}}}2、創(chuàng)建文檔PUTmy_index/doc/1PUTmy_index/doc/1{"cookie":"123","name":"home""name":"home"}3、查詢GETmy_index/_searchGETmy_index/_search{"query":{"match":{"cookie":"123"}}}#報錯GETmy_index/_search{"query":{"match":{"name":"home"}}}#有結(jié)果index_option41、docs只記錄docid2、freqs記錄docid和termfrequencies3、positions記錄docid、termfrequencies和termposition4、docid、termfrequencies、termpositioncharacterpositionsdocs記錄的內(nèi)容越多暫用的空間越大PUTmy_indexPUTmy_index{“doc”:{“properties”:{“cookie”:{“type”:”text”,“index_option”:”offsets”}}}}null_valuenull、數(shù)據(jù)類型字符串型：text、keyword數(shù)值型：long、integer、short、byte、double、float、half_float、scaled_float日期類型：date布爾類型：boolean二進制類型：binary范圍類型：integer_range、float_range、long_range、double_range、date_range對象類型：object嵌套類型：nestedobjectgeo_point(點)、geo_shape(形狀)記錄IP地址ip實現(xiàn)自動補全completion記錄分詞數(shù)：token_count記錄字符串hash值母乳murmur3multi-fieldspinyin即可1、創(chuàng)建mappingPUTmy_index1PUTmy_index1{"mappings":{"doc":{"properties":{"username":{"type":"text","fields":{"pinyin":{"type":"text"}}}}}}}}2、創(chuàng)建文檔PUTmy_index1/doc/1PUTmy_index1/doc/1{"username":"hahaheihei"}3、查詢GETmy_index1/_searchGETmy_index1/_search{"query":{"match":{"username.pinyin":"haha"}}}DynamicMappinges可以自動識別文檔字段類型，從而降低用戶使用成本PUT/test_index/doc/1PUT/test_index/doc/1{"username":"alfred","age":1}{{"test_index":{"mappings":{"doc":{"properties":{"age":{"type":"long"},"username":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}}age自動識別為long類型，username識別為text類型PUTtest_index/doc/1PUTtest_index/doc/1{"username":"samualz","age":14,"birth":"1991-12-15","year":18,"tags":["boy","fashion"],"money":"100.1"}{{"test_index":{"mappings":{"doc":{"properties":{"age":{"type":"long"},"birth":{"type":"date"},"money":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tags":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},},"username":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"year":{"type":"long"}}}}}}日期的自動識別可以自行配置日期格式，以滿足各種需求1、自定義日期識別格式PUTmy_indexPUTmy_index{"mappings":{"doc":{"dynamic_date_formats":["yyyy-MM-dd","yyyy/MM/dd"]}}}2、關(guān)閉日期自動識別PUTmy_indexPUTmy_index{"mappings":{"doc":{"date_detection":false}}}字符串是數(shù)字時，默認不會自動識別為整形，因為字符串中出現(xiàn)數(shù)字時完全合理的Numeric_datection可以開啟字符串中數(shù)字的自動識別PUTmy_indexPUTmy_index{"mappings":{"mappings":{"doc":{"numeric_datection":true}}}Dynamic允許根據(jù)es自動識別的數(shù)據(jù)類型、字段名等來自動設(shè)定字段類型-所有字符串類型都設(shè)定為-message開頭的字段都設(shè)定為text-所有以long_開頭的字段都設(shè)定為long類型-所有自動匹配為double類型的都設(shè)定為float類型，以節(jié)省空間1、匹配規(guī)則match_mapping_type：匹配es自動識別的字段類型，如boolean，long，string等match,unmatch：匹配字段名path_match,path_unmatch：匹配對象內(nèi)部字段2、例子#mapping#數(shù)組，可指定多個模板PUTtest_index{"mappings":{"doc":{"dynamic_templates":[{"strings":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}]}}}messagetextPUTtest_indexPUTtest_index{"mappings":{"doc":{"dynamic_templates":[{{"message_as_text":{"match_mapping_type":"string","match":"message*","mapping":{"type":"text"}}}]}}}double類型設(shè)定為float，節(jié)省空間PUTtest_indexPUTtest_index{"mappings":{"doc":{"dynamic_templates":[{"message_as_text":{"match_mapping_type":"double","mapping":{"type":"float"}}}]}}}用于在新建索引時自動應(yīng)用預(yù)先設(shè)定的配置，簡化索引創(chuàng)建的操作步驟1、可以設(shè)定索引的配置和mapping2、可以有多個模板，根據(jù)order設(shè)置，order大的覆蓋小的配置3、mapping1、寫入一條文檔到es的臨時索引中，獲取es自動生成的mapping2、修改步驟1得到的mapping，自定義相關(guān)配置3、使用步驟2中的mapping創(chuàng)建實際所需的索引#order#orderorder小的模板PUT_template/test_template{"index_patterns":["te*","bar*"],"order":0,"settings":{"number_of_shards":1},"mappings":{"doc":{"_source":{"enabled":false},"properties":{"name":{"type":"keyword"}}}}}有時索引創(chuàng)建出問題，首先查看模板、SearchAPI(URI)GET/_search #GET/my_index/_search #查指索文GET#多引詢GET/my_*/_searchURI（）GET/my_index/_search?q=user:alfred #GET/my_index/_search?q=alfred&df=user&sort=age:asc&from=4&size=10&timeout=1sq:指定查詢的語句，例如q=aa或q=user:aadf:q中不指定字段默認查詢的字段，如果不指定，es會查詢所有字段Sort：排序，asc升序，desc降序timeout：指定超時時間，默認不超時from，size：用于分頁termterm相當(dāng)于單詞查詢，phrase相當(dāng)于詞語查詢term：Alfredway等效于alfredorwayphrase：”Alfredway”詞語查詢，要求先后順序泛查詢Alfred等效于在所有字段去匹配該term(不指定字段查詢)name:alfredGroup（）（quickORbrown）ANDfox：通過括號指定匹配的優(yōu)先級status:(activeORpending)title:(fulltextsearch)：把關(guān)鍵詞當(dāng)成一個整體1、批量創(chuàng)建文檔POSTtest_search_index/doc/_bulkPOSTtest_search_index/doc/_bulk{"index":{}}{"username":"alfredway","job":"javaengineer","age":18,"birth":"1991-12-15","isMarried":false}{"index":{"_id":2}}{"username":"alfred","job":"javaseniorengineerandjavaspecialist","age":28,"birth":"1980-05-07","isMarried":true}{"index":{"_id":3}}{"username":"lee","job":"javaandrubyengineer","age":22,"birth":"1985-08-07","isMarried":false}}{"index":{}}{"username":"leejunior"job":"rubyengineer","age":23,"birth":"1986-08-07","isMarried":false}2、泛查詢GETtest_search_index/_search?q=alfredGETtest_search_index/_search?q=alfred3、查詢語句執(zhí)行計劃查看GETtest_search_index/_search?q=alfredGETtest_search_index/_search?q=alfred{"profile":true}4、term查詢GETGETtest_search_index/_search?q=username:alfred#alfredOR5、phrase查詢GETtest_search_index/_search?q=username:"alfredway"GETtest_search_index/_search?q=username:"alfredway"6、group查詢GETtest_search_index/_search?q=username:(alfredway)GETtest_search_index/_search?q=username:(alfredway)7、布爾操作符（1）AND(&&),OR(||),NOT(!)例如：name:(tomNOTlee)#表示name字段中可以包含tom但一定不包含lee（2）+、-分別對應(yīng)must和must_not例如：name:(tom+lee-alfred)#表示name字段中，一定包含lee，一定不包含alfred，可以包含tom注意：+在url中會被解析成空格，要使用encode后的結(jié)果才可以，為%2BGETtest_search_index/_search?q=username:(alfred%2Bway)GETtest_search_index/_search?q=username:(alfred%2Bway)1、區(qū)間：閉區(qū)間：[]，開區(qū)間:{}age:[1TO10]#1<=age<=10age:[1TO10}#1<=age<10age:[1] #1<=ageage:[*10]2、算術(shù)符號寫法age:>=1age:(>=1&&<=10)或者age:(+>=1+<=10)?:1個字符*:0name:t*m?/*name:/[mb]oat/fuzzyname:roam~1匹配與roam差1個character的詞，比如foam、roams等proximitysearch“foxquick”~5以term為單位進行差異比較，比如”quickfox”“quickbrownfox”、SearchAPI(RequestBodySearch)MatchQuery對字段作全文檢索，最基本和常用的查詢類型GETtest_search_index/_searchGETtest_search_index/_search{"profile":true, #"query":{"match":{"username":"alfredway"}}}通過operator參數(shù)可以控制單詞間的匹配關(guān)系，可選項為or和andGETtest_search_index/_searchGETtest_search_index/_search{"query":{"query":{"match":{"username":{"query":"alfredway","operator":"and"}}}}GETtest_search_index/_search{"query":{"match":{"username":{"query":"alfredway","minimum_should_match":1GETtest_search_index/_search{"query":{"match":{"username":{"query":"alfredway","minimum_should_match":1}}}}、相關(guān)性算分相關(guān)性算分是指文檔與查詢語句間的相關(guān)度，英文為relevance本質(zhì)就是搜索結(jié)果返回文檔的排序問題TermFrequency(TF)詞頻單詞在該文檔中出現(xiàn)的次數(shù)。詞頻越高，相關(guān)度越高DocumentFrequency(DF)文檔頻率單詞出現(xiàn)的文檔數(shù)InverseDocumentFrequency(IDF)逆向文檔頻率Field-lengthNormELK課件1、ELK簡介、ELK是什么ElasticsearchrestfulLogstash（。KibanaKibanaLogstash和ElasticSearch提供、ELK應(yīng)用問題1、API不一樣，我們?nèi)绾稳フ希?-》dubbo定義統(tǒng)一的api規(guī)范2、各子系統(tǒng)之間會產(chǎn)生操作痕跡（用戶行為軌跡）---》日志3、各個子系統(tǒng)都會生成各自的日志---日志整合--》logstash4、AOP埋點，異步日志輸出1通過第三方進行移動話費充值日志輸出：每次調(diào)用都會打印異步日志分布式負載均衡：很多太機器都可以充值（動態(tài)的去選擇一臺目前比較空閑的機器去執(zhí)行這個任務(wù)）問題：A：兄弟，幫忙查一下今天手機號碼138001380000充值日志記錄（是否成功）B：稍等5分鐘后A：怎么樣了B：稍等，還剩下3臺機器沒查完結(jié)論：如果能把所有的日志整理在一起，就不會出現(xiàn)一臺一臺去查的問題解決方案：1、可不可以把日志放在數(shù)據(jù)庫中。數(shù)據(jù)量太大，且日志沒有規(guī)范日志格式，數(shù)據(jù)庫方案不太建議，且壓力過大2、采用大數(shù)據(jù)日志處理方案成本太高，且分布式環(huán)境每個系統(tǒng)的日志規(guī)則不一樣。具體業(yè)務(wù)實踐日志收集：Logstash日志存儲：ElasticSearch日志展示：Kibana針對對臺服務(wù)器日志不統(tǒng)一的問題，提供多種檢索規(guī)則，方便可視化展示分布式帶來的問題：多節(jié)點、負載均衡、日志分散、運維成本高（需要人為跟蹤）、集中式日志管理系統(tǒng)當(dāng)前主流的一些集中日志管理系統(tǒng)1、簡單的：Rsyslog2、商業(yè)化：Splunk3、開源的：Scribe（FaceBook），Chukwa（Apache）4、ELK最廣泛的（ElasticStack）(java語言編寫)、ELKElasticSearchJava實時的分布式搜索和分析引擎，他可以用于全文檢索，結(jié)構(gòu)化搜索以及分析，lucene。SolrLogstashJRuby具有實時渠道能力的數(shù)據(jù)收集引擎，包含輸入、過濾、輸出模塊，一般在過濾模塊中做日志格式化的解析工作KibanaJavaScriptElasticSerachElasticSerach、日志日志：記錄程序的運行軌跡---級別：ERROR、INFO、DEBUG、WARN目的：方便定位和查找信息，記錄除去業(yè)務(wù)外的附加的信息，鏈路Filebeat簡介SSHFilebeatLogstash或Elasticsearch時，F(xiàn)ilebeatLogstashFilebeatFilebeat無論在任何環(huán)境中，隨時都潛伏著應(yīng)用程序中斷的風(fēng)險。Filebeat能夠讀取并轉(zhuǎn)發(fā)日志行，如果出現(xiàn)中斷，還會在一切恢復(fù)正常后，從中斷前停止的位置繼續(xù)開始。2、準(zhǔn)備工作、安裝Centos7建議內(nèi)存2G以上、基本配置IPvi/etc/sysconfig/network-scripts/ifcfg-eno33servicenetworkrestart[root@localhost~]#adduserelk1[root@localhost~]#passwdelk1[root@localhost~]#whereissudoers[root@localhost~]#ls-l/etc/sudoers[root@localhost~]#chmod-vu+w/etc/sudoers[root@localhost~]#vi/etc/sudoers##Allowroottorunanycommandsanywherroot ALL=(ALL) ALLlinuxidc ALL=(ALL) ALL #[root@localhost~]#chmod-vu-w[root@localhost~]#suelk13、ElasticSerach、環(huán)境安裝[root@localhostjdk1.8]#tar-zxvfjdk-8u171-linux-x64.tar.gz[root@localhostjdk1.8.0_171]#vi/etc/profile在文件最后添加exportJAVA_HOME=/home/elk1/jdk1.8/jdk1.8.0_171exportJRE_HOME=$JAVA_HOME/jreexportJAVA_HOME=/home/elk1/jdk1.8/jdk1.8.0_171exportJRE_HOME=$JAVA_HOME/jreexportCLASSPATH=.:$JAVA_HOME/LIB:$JRE_HOME/LIB:$CLASSPATHexportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH[root@localhostjdk1.8.0_171]#source/etc/profile[root@localhostjdk1.8.0_171]#java-versionjavaversion"1.8.0_171"Java(TM)SERuntimeEnvironment(build1.8.0_171-b11)JavaHotSpot(TM)64-BitServerVM(build25.171-b11,mixedmode)、ElasticSerach單機安裝[root@localhostelasticserach]#tar-zxvfelasticsearch-6.3.1.tar.gz[root@localhostelasticserach]#cdelasticsearch-6.3.1/bin[root@localhostbin]#./elasticsearch[root@localhostbin]#suelk1[elk1@localhostbin]$./elasticsearch[root@localhostbin]#chown-Relk1:elk1/home/elk1/elasticsearch[elk1@localhostbin]$./elasticsearch[elk1@localhostconfig]$vijvm.options[elk1@localhostbin]$./elasticsearch[root@localhostjdk1.8.0_171]#curl:9200#后臺啟動[elk1@localhostbin]$./elasticsearch-d#關(guān)閉程序[elk1@localhostbin]$ps-ef|grepelastic[elk1@localhostbin]$kill10097#設(shè)置瀏覽器訪問[root@localhostbin]systemctlstopfirewalld[root@localhostbin]viconfig/elasticsearch.yml安裝問題：[1][2]解決方案[root@localhostbin]#vi/etc/security/limits.conf[3]解決方案[root@localhostbin]#vi/etc/sysctl.conf[root@localhostbin]#sysctl-p、ElasticSerach集群安裝修改配置文件vim/:aubin-cluster#:aubin-cluster#必須相同#集群名稱（不能重復(fù)）:els1（必須不同）#節(jié)點名稱，僅僅是描述名稱，用于在日志中區(qū)分（自定義）#指定了該節(jié)點可能成為master節(jié)點，還可以是數(shù)據(jù)節(jié)點node.master:truenode.data:truepath.data:/var/lib/elasticsearch#數(shù)據(jù)的默認存放路徑（自定義）path.logs:/var/log/elasticsearch#日志的默認存放路徑network.host:#當(dāng)前節(jié)點的IP地址http.port:9200#對外提供服務(wù)的端口transport.tcp.port:9300#9300為集群服務(wù)的端口discovery.zen.ping.unicast.hosts:["1","2","3"]#集群個節(jié)點IP地址，也可以使用域名，需要各節(jié)點能夠解析discovery.zen.minimum_master_nodes:2#為了避免腦裂，集群節(jié)點數(shù)最少為半數(shù)+1注意：清空data和logs數(shù)據(jù)2:9200/_cat/nodes?v、安裝head插件headwget/mobz/elasticsearch-head/archive/elasticsearch-head-master.zip也可以用git下載，前提yuminstallgitunzipelasticsearch-head-master.zipnode.jswget/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gztar-zxvfnode-v9.9.0-linux-x64.tar.gznode.jssource/etc/profile測試node-vnpm-vgrunt（grunt是一個很方便的構(gòu)建工具，可以進行打包壓縮、測試、執(zhí)行等等的工作）elasticsearch-head-masternpminstall-ggrunt-clinpminstall(npminstall-gcnpm--registry=)Elasticsearch編輯elasticsearch-6.3.1/config/elasticsearch.yml,加入以下內(nèi)容：http.cors.enabled:truehttp.cors.allow-origin:"*"修改Gruntfile.js（’，’）打開elasticsearch-head-master/Gruntfile.js，找到下面connect屬性，新增hostname:’*’:connect:{server:{options:{hostname:'*',port:9100,base:'.',keepalive:true}}}啟動elasticsearch-head進入elasticsearch-head目錄，執(zhí)行命令：gruntserver后臺啟動elasticsearch-headnohupgruntserver&exitheadps-aux|grepkill、ElasticSerachAPIelasticsearchrestapicurl-X<RESTVerb><Node>:<Port>/<Index>/<Type>/<ID>檢查escurlIP:9200http://IP:9200/_cat/health?vhttp://IP:9200/_cat/nodes?vhttp://IP:9200/_cat/indices?vcurl-XPUT'IP:9200/XX?pretty'curl-XPUT'IP:9200/XX/external/2?pretty'-d'{"gwyy":"John"}'curl-XPOST'IP:9200/XX/external/1/_update?pretty'-d'{"doc":{"name":"Jaf"}}'curl-XDELETE'IP:9200/_index?pretty'、配置詳情ES##集群的名字，以此作為是否同一集群的判斷條件:elasticsearch#節(jié)點名字，以此作為集群中不同節(jié)點的區(qū)分條件:node-1#設(shè)置當(dāng)前節(jié)點既可以為主節(jié)點也可以為數(shù)據(jù)節(jié)點node.master:truenode.data:true#索引分片個數(shù)，默認為5片#index.number_of_shards:5#索引副本個數(shù)，默認為1個副本#index.number_of_replicas:1#數(shù)據(jù)存儲目錄（多個路徑用逗號分隔）discovery.zen.ping.unicast.hosts:["4","5"]discovery.zen.minimum_master_nodes:2#數(shù)據(jù)目錄path.data:/home/elk1/elasticserach/data#日志目錄path.logs:/home/elk1/elasticserach/logs#修改一下ES的監(jiān)聽地址，這樣別的機器才可以訪問network.host:3#設(shè)置節(jié)點間交互的tcp端口（集群）,默認是9300transport.tcp.port:9300#監(jiān)聽端口（默認的就好）http.port:9200#增加新的參數(shù)，這樣head插件才可以訪問eshttp.cors.enabled:truehttp.cors.allow-origin:"*"Jvm.optionsJVM的相關(guān)配置Lperties日志相關(guān)配置、Elasticserach模式DevelopmentProduction以transport的地址是否綁定在localhost為標(biāo)準(zhǔn)（實際地址）即：elasticserach.yml文件中的network.host配置Development模式下啟動時會以warningProduction模式下在啟動時會以error、elasticserach操作Document:Index:（）表）Field:字段）QueryDSL:（sql）CRUD請求：POST/atguigu/student/1{"name":"zhangsan","clazz":"0115bigdata","description":"wearefamily"}返回：{"_index":"atguigu","_type":"student","_id":"1","_version":1,"result":"_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1}請求：GETatguigu/student/1返回：{"_index":"atguigu","_type":"student","_id":"1","_version":1,"found":true,"_source":{"name":"zhangsan","clazz":"0115bigdata","description":"wearefamily"}}請求：POST/atguigu/student/1/_update{"doc":{"description":"helloworld"}}返回：{"_index":"atguigu","_type":"student","_id":"1","_version":2,"result":"_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1,"_primary_term":1}請求：DELETEatguigu/student/1查詢結(jié)果：{"_index":"atguigu","_type":"student","_id":"1","found":false}ElasticserachQueryQueryStringGET/atguigu/student/_sea'rch?q=關(guān)鍵字返回：{"took":8,"timed_out":"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"atguigu","_type":"student","_id":"1","_score":0.2876821,"_source":{"name":"zhangsan","clazz":"0115bigdata","description":"wearefamily"}}]}}QueryDSLGETatguigu/student/_search{"query":{"term":{"name":{"value":"zhangsan"}}}}4、Logstash、安裝logstash[root@localhostlogstash]#tar-zxvflogstash-6.3.1.tar.gz[root@localhostlogstash-6.3.1]#cdconfig[root@localhostconfig]#vilog4j_to_es.confinput{file{input{file{path=>[""]type=>""start_position=>"beginning"}}output{stdout{codec=>rubydebug}}[root@localhostlogstash-6.3.1]#./bin/logstash-fconfig/log4j_to_es.conf、輸入、輸出、過濾輸入input{file{path=>”/tomcat/logs/abc.log”}}輸出output{stdout{codec=>rubydebug}}Grok1、基于正則表達式提供了豐富可重用的模式（pattern）2、基于此可以將非結(jié)構(gòu)化數(shù)據(jù)作結(jié)構(gòu)化處理Date將字符串類型的時間字段轉(zhuǎn)換為時間戳類型，方便后續(xù)數(shù)據(jù)處理進行增加、修改、刪除、替換等字段相關(guān)處理、logstash格式化nginx日志內(nèi)容nginx_logstash.conf文件input{input{stdin{}}filter{grok{match=>{"message"=>-\[%{HTTPDATE:time}\]"%{WORD:request_action} HTTP/%{NUMBER:http_version}"%{NUMBER:response}%{NUMBER:bytes}"%{DATA:referrer}""%{DATA:agent}"'}}date{match=>["time","dd/MMM/YYYY:HH:mm:ssZ"]locale=>en}geoip{source=>"remote_ip"target=>"geoip"}useragent{source=>"agent"target=>"user_agent"}}output{stdout{codec=>rubydebug}}Logstashnginxhead-n2/home/elk1/nginx_logs|./logstash-f../config/nginx_logstash.conf結(jié)果{{"user_name"=>"-","referrer"=>"-","@timestamp"=>2015-05-17T08:05:32.000Z,"request"=>"/downloads/product_1","time"=>"17/May/2015:08:05:32+0000","geoip"=>{"country_code3"=>"NL","longitude"=>4.8995,"continent_code"=>"EU","latitude"=>52.3824,"timezone"=>"Europe/Amsterdam","country_code2"=>"NL","ip"=>"","country_name"=>"Netherlands","location"=>{"lat"=>52.3824,"lon"=>4.8995}},"@version"=>"1","http_version"=>"1.1","remote_ip"=>"","message" => " - - [17/May/2015:08:05:32 +0000] \"GET/downloads/product_1 HTTP/1.1\" 304 0 \"-\" \"Debian (0.8.16~exp12ubuntu10.21)\"","bytes"=>"user_agent"=>{"minor"=>"3","os"=>"Debian","os"=>"Debian","name"=>"DebianAPT-HTTP","os_name"=>"Debian","build"=>"","major"=>"1","device"=>"Other"},"agent"=>"DebianAPT-HTTP/1.3(0.8.16~exp12ubuntu10.21)","host"=>"localhost.localdomain","response"=>"304","request_action"=>"GET"}5、Kibana、Kibana安裝[root@localhostkibana]#tar-zxvfkibana-6.3.1-linux-x86_64.tar.gz[root@localhostkibana]#cdkibana-6.3.1-linux-x86_64/config[root@localhostconfig]#vikibana.yml[root@localhostbin]#./kibana、kibana配置configKibana.ymlServer.host/server.port:訪問的端口號和地址(地址設(shè)置后才能被外網(wǎng)訪問)Elasticsearch.url:訪問elasticserach的地址、kibana功能簡介Discover:數(shù)據(jù)搜索查看Visualize:圖標(biāo)制作Dashboard:儀表盤制作Timeline:時序數(shù)據(jù)的高級可視化分析DevTools:開發(fā)者工具Management:kibana相關(guān)配置6、Filebeat和packetbeat、FilebeatFilebeathttps://www.elastic.co/cn/downloads/beats/filebeat查看系統(tǒng)位數(shù)：getconfLONG_BIT、（）DNS、Http、Redis、Mysql等elasticserachpacketbeat編輯es.yml文件erfaces.device:ens33#erfaces.device:ens33#網(wǎng)卡tocols.http:ports:[9200]#es端口send_request:true#抓取請求信息include_body_for:["application/json","x-www-form-urlencoded"]#包含內(nèi)容output.console:pretty:true#控制臺輸出packetbeatsudo./packetbeat-e-ces.yml-strict.perms=false7、Nginxnginx#安裝依賴環(huán)境yuminstallgcc-c++yuminstallpcre-develyuminstallzlibzlib-develyuminstallopensslopenssl-deve#//一鍵安裝上面四個依賴#yum-yinstallgcczlibzlib-develpcre-developensslopenssl-devel#解壓tar-xvfnginx-1.13.7.tar.gz#進入nginx目錄cd/usr/local/nginx #./configure#執(zhí)行make命令make//執(zhí)行makeinstall命令makeinstall//nginx/sbin/nginx//停止命令nginx/sbin/nginx-sstop或者:nginx-squit//重啟命令nginx-sreload8、數(shù)據(jù)可視化演示實戰(zhàn)、實戰(zhàn)說明需求：收集Elasticserach集群的查詢語句分析查詢語句的常用語句、響應(yīng)時長等方案數(shù)據(jù)收集：Packetbeat+logstash數(shù)據(jù)分析：Kibana+Elasticsearch、前期準(zhǔn)備ProductionCluster()3:92002、Kibana5:5601MonitoringCluster()1、Elasticsearch6:82002、Kibana6:8601Logstash\packetbeat、實戰(zhàn)./elasticsearch修改kibana配置./kibana #同上logstashinput{input{beats{port=>5044}}filter{if"search"in[request]{grok{match=>{"request"=>".*\n\{(?<query_body>.*)"}}grok{match=>{"path"=>"\/(?<index>.*)\/_search"}}if[index]{}else{mutate{add_field =>{"index"=>"All"}}}mutate{}}if"search"in[request]{elasticsearch{hosts=>":9200"}}output{#stdout{codec=>rubydebug}}mutate{remove_field=>["[http][response][body]"]}###}=>{"query_body"=>"{%{query_body}"}}啟動./bin/logstash-fconfig/log4j_to_es.conf附錄：防火墻配置1、firewalld的基本使用startfirewalld關(guān)閉：firewalld查看狀態(tài)：systemctlstatusfirewalld開機禁用：systemctldisablefirewalld開機啟用：systemctlenablefirewalldCentOS7service和chkconfig的功firewalld.servicefirewalld.servicerestartfirewalld.service顯示一個服務(wù)的狀態(tài)：systemctlstatusfirewalld.service在開機時啟用一個服務(wù)：systemctlenablefirewalld.servicedisablefirewalld.serviceis-enabledfirewalld.servicelist-unit-files|grepenabledsystemctlfailedfirewalld-cmdfirewall-cmdversion查看幫助：firewall-cmd--helpfirewall-cmd查看所有打開的端口：firewall-cmd--zone=public--list-ports更新防火墻規(guī)則：firewall-cmd--reload查看區(qū)域信息:firewall-cmd--get-active-zones查看指定接口所屬區(qū)域：firewall-cmd--get-zone-of-interface=eth0拒絕所有包：firewall-cmd--panic-on取消拒絕狀態(tài)：firewall-cmd--panic-off查看是否拒絕：firewall-cmd--query-panic添加firewall-cmd--zone=publicadd-port=80/tcppermanent （--permanent）重新載入firewall-cmd--reload查看firewall-cmd--zone=public--query-port=80/tcp刪除firewall-cmd--zone=public--remove-port=80/tcp--permanentNginx安裝手冊nginx安裝環(huán)境nginx是C語言開發(fā)，建議在linux上運行，本教程使用Centos6.5作為安裝環(huán)境。gccnginxgccgcc環(huán)境，gcc：yuminstallgcc-c++PCREPCRE(PerlCompatibleRegularExpressions)Perlperlpcrelinuxpcre庫。yuminstall-ypcrepcre-devel注：pcre-devel是使用pcre開發(fā)的一個二次開發(fā)庫。nginx