ClouderaManager大數(shù)據(jù)平臺(tái)部署指南

1、部署指南 Cloudera Manager CDH官方共給出了 3 中安裝方式。第一種方法必須要求所有機(jī)器都能連網(wǎng)，由于 各種網(wǎng)絡(luò)超時(shí)錯(cuò)誤，基本沒法用。第二種方法使用CM的在線yum源，在線下載很多包，同樣是超時(shí)嚴(yán)重。 第三種方法是全離線方式， 實(shí)際情況是也只能用這個(gè)。1、Cloudera Manager 的部署條件? CDH 不支持在容器中部署。? 支持Oracle JDK7,目前可使用的最新版本為1.7u80，及以上版本也支持使用oracle JDK8,同一個(gè)CDH集群中的所有節(jié)點(diǎn)使用的JDK版本必須一致。? 使用 Python2.4 及以上，不支持 Python3? 僅支持IPv4，且I

2、Pv6必須要disabled? 基于tar壓縮包的離線安裝方式，會(huì)在中停止支持該方式。? 支持selinux，但如果不熟悉使用方法，建議關(guān)閉。? 管理端口為 7180，此外還有很多基礎(chǔ)應(yīng)用使用的端口，可以運(yùn)行iptables，需要確認(rèn)有恰當(dāng)?shù)氖跈?quán)控制，例如至少要對(duì)同網(wǎng)段的訪問放開全部端口。2、Cloudera Manager 部署的幾個(gè)階段和可選擇的方式Phase 1: Install JDKInstall the JDK required by Cloudcra Manager Server, Management Sei vice, and CDH.Phase 2: Set upDa La

3、 bases Install configure, and start the databases that are required by the Cloudero Manager Ssji vet, Cloudera ddriagerrinnl Service, Hnd That are opTionfil for some CDH services.Cioudcra ins tollation PhasesPhaseThere are two options: Use the Ooudera Manager Instailer to install a supported version

4、 of the Oracle JDK in 丿3匸/二uennd on all hosts in the dustor. Uso th。connmarid lino to martially install supported versions of ths Oracle JCK and set t*ie j=l7a hqmi environment variable to the iretail directory on all hostsThere are two options: Use the ClouderaInstaller lo install configure, and st

5、art anei 11 bedded PoslgrssSOL database.* Use comnand-line package installation tools like yumto irstall, configure, and install the databasePhase 去 InslallCloudefd MaiMgerServerInstall and start Cloudera Manager Server on onm host.Phase 牟：Install Cloudcra ManagerInstall and siarl. the Cloudera Mana

6、ger Agent on dll hosts.Path APath BPathCUse the ClouderaManayttr Inslaller to install its packages and the server RequirBS Intfirnet access and Eudo privileges on the hostUse Linux package install commands (like yunj to install Ckxidera Maragar Server Update database properties.Use service commands

7、to start Cloudera Manager Server.Linux commands to unpack tarballs and service commands to start the server.Use the Cloudero Manager Instalktion wizard to install the AgtsiiLs on dll hosts.There are two options:* Use Linux package install commanda (like 7) to install Cloudera Managti Agents on all h

8、oses.Use Linux commands to unpack tarballs and ber vice coinrnands to start if e aq«nis on all hosts. Use the Cloudera Manager Installation wizard to install the Agents on all hosts.Ptiase 5: InsiallCDH and Managed Service softwareInll, configure, And start CDH and managed services on all host?

9、.Use the Cloudera Manager Installation wizard to install CDH and nthsr msnagerf services.Thefe are two options: Use the Cloudera Maridijer trblalldUori wizard to install CDH And other managed services Use I inux package install commands Qilc舊to install CDH and other managed services on ell hostsUse

10、Linux commands to unpack tarballs and service commands to Rtart CDH詞nti m罰旳開 services on all hosts.PiMMS C: Ciedle, Configuie arid Start CDH 目 ndSei vicesConfiqure and sian CDH and niaridijed services.Usu the Cluudef dManager Installatiuii wizard to install CDH and olliei services, assign roles io a

11、nd configure lhe cluster. Manv configurations are automated.Use the Clouderd IVandyer Insidllatiui 1 wizard ic install CDH and olhef 'liaridged services.日Esiqn roles To hosts, ar kJ confiyuie the cluster. Many configurations are automated.Use the Cluddera Manager Installatiun wizard to install C

12、DH and ulher rnandqed services, assiqn roles 10 hosts, and configuie the duster. Many confiQurations are automated.You can also use tfie Cloudera Manager API to manage a cluster, which ran he useful for scripting preconfigured deplo/msnts.注：因?yàn)镃loudera Manager的官方y(tǒng)um源位于國(guó)外，受網(wǎng)速影響而無(wú)法正常使 用。所以上述部署方式中，一般是使用

13、 PATH C或者在預(yù)先下載好相關(guān) RPM包 的條件下參照PATH B方式。3、主機(jī)節(jié)點(diǎn)信息和基礎(chǔ)配置（1）主機(jī)信息本次部署使用以下主機(jī)，前兩個(gè)作為 NameNod使用。后四個(gè)主機(jī)，分別額 外掛載一塊1TB的磁盤到本地的/dfs/dn目錄下。注1:用于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)磁盤不要配置 RAID,直接裸盤即可。如果有RAID 控制器，則直接對(duì)每塊盤設(shè)置一個(gè)單獨(dú)的RAID0仍然當(dāng)作單盤使用。注2：在有第二塊數(shù)據(jù)盤的情況下，繼續(xù)掛載到/dfs/dn2，更多的數(shù)據(jù)盤則 按命名規(guī)則和掛盤規(guī)則擴(kuò)展即可。注3:對(duì)于生產(chǎn)環(huán)境而言，NameNod所在的節(jié)點(diǎn)一般不要同時(shí)做數(shù)據(jù)節(jié)點(diǎn)使 用。而且NameNod所在主機(jī)的磁

14、盤需要配置適當(dāng)?shù)?RAID保護(hù)級(jí)別，以提高可靠 性。注 4：對(duì)于 DataNode 所在的主機(jī)節(jié)點(diǎn)， 建議系統(tǒng)盤和數(shù)據(jù)盤完全隔離開， 然 后系統(tǒng)盤使用兩磁盤配置為RAID1以提高可靠性。雖然數(shù)據(jù)節(jié)點(diǎn)是冗余的，掛一 個(gè)也不影響CDH平臺(tái)運(yùn)行，但修起來(lái)還是很麻煩啊。cluster-70cluster-71cluster-72cluster-73cluster-74cluster-75操作系統(tǒng)全部使用 CentOS 6.9 Minimal 安裝。將全部主機(jī)的主機(jī)名設(shè)置好， 并將以上映射關(guān)系寫到每個(gè)主機(jī)的 /etc/hosts 文件中。（2）免密登錄在 cluster-70 節(jié)點(diǎn)上制作 ssh 密鑰：

15、 ssh-keygen將該密鑰分發(fā)至其它各節(jié)點(diǎn)，以實(shí)現(xiàn)免密鑰登錄，如： ssh-copy-id cluster-71（ 3） selinux 和 iptables所有節(jié)點(diǎn)配置為關(guān)閉 selinux 。所有節(jié)點(diǎn)的 iptables 設(shè)置為對(duì)同網(wǎng)段地址放開全部端口的訪問權(quán)限。（4）系統(tǒng)性能參數(shù)調(diào)整中即可。cat << EOF >> /etc/sysctl.confvm.swappiness = 0 = 1 = 1 = 1024 65000EOFsysctl -pecho never > /sys/kernel/mm/transparent_hugepage/defra

16、g echo never > /sys/kernel/mm/transparent_hugepage/enabled cat << EOF >> /etc/rc.localecho never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled EOF4、安裝 JDK下載并分發(fā) Oracle JDK 安裝包至所有節(jié)點(diǎn)，執(zhí)行安裝和配置命令。rpm -ivh jdk-7u80-linux-x64.rpm 直

17、接刷下面腳本完成配置： cat << EOF >> CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOMEexport PATHexport CLASSPATHEOFsource /etc/profile以上JDK配置，需要在每個(gè)節(jié)點(diǎn)上都處理5、安裝和配置數(shù)據(jù)庫(kù)主要有以下業(yè)務(wù)會(huì)使用到數(shù)據(jù)庫(kù)服務(wù)：Cloudera Manager ，存放各種服務(wù)、配置數(shù)據(jù)，空間大約100MB ，很重要，應(yīng)經(jīng)常備份 ；Oozie Server - Contains Oozie workflow,

18、coordinator, and bundle data. Can grow very large.Sqoop Server - Contains entities such as the connector, driver, links and jobs. Relatively small.Activity Monitor - Contains information about past activities. In large clusters, this database can grow large. Configuring an Activity Monitor database

19、is only necessary if a MapReduce service is deployed.Reports Manager - Tracks disk utilization and processing activities over time. Medium-sized.Hive Metastore Server - Contains Hive metadata. Relatively small.Hue Server - Contains user account information, job submissions, and Hive queries. Relativ

20、ely small.Sentry Server - Contains authorization metadata. Relatively small.Cloudera Navigator Audit Server - Contains auditing information. In large clusters, this database can grow large.Cloudera Navigator Metadata Server - Contains authorization, policies, and audit report metadata. Relatively sm

21、all.（ 1）因?yàn)闇?zhǔn)備使用 cluster-70 作為 NameNodeMaster ，所以把 mysql 數(shù)據(jù)庫(kù)安 裝在 cluster-71 上面 ，以分散下風(fēng)險(xiǎn)。登錄 cluster-71 ，然后使用任一種自己熟悉的方式把 mysql 數(shù)據(jù)庫(kù)安裝好即 可： yum install mysql-server mysql chkconfig mysqld on service mysqld start 完成數(shù)據(jù)庫(kù)初始化： /usr/bin/mysql_secure_installation（2）優(yōu)化數(shù)據(jù)庫(kù)配置 service mysqld stop 將舊的 InnoDB log 文件移走：

22、/var/lib/mysql/ib_logfile0 and /var/lib/mysql/ib_logfile1 out of /var/lib/mysql/參照下面調(diào)整f參數(shù)，以更適合CDHR務(wù)使用： mysqld datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql# Disabling symbolic-links is recommended to prevent assorted security riskssymbolic-links=0 transaction-isolation = READ-CO

23、MMITTED key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 #binlog_format = mixed read_buffer_size = 2Mread_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M

24、 # InnoDB settings innodb_file_per_table = 1innodb_flush_log_at_trx_commitinnodb_log_buffer_size = 64M innodb_buffer_pool_size = 2Ginnodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512Mmysqld_safe log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid sql

25、_mode=STRICT_ALL_TABLES啟動(dòng)mysql，如果有報(bào)錯(cuò)，可根據(jù)日志信息排查3）創(chuàng)建需要使用的幾個(gè)庫(kù)I RoleDatabaseUserPassword1Activity Monitoramonamonamon_pa$<wordRepcrts Managerrmanrmanrman_passwordHive Metastore Servermetastorehivehive_pasewordSentry Serversentrysentrysen try_pa sswordCloudera Navigaior Audit Servernavnavnav.passwordC

26、loudera Naviyato Metadata Serverravmsnavmsna v ms_pa55wordcreate database database DEFAULT CHARACTER SET utf8;grant all o n database.* TO 'user''%' IDENTIFIED BY 'password'注：在部署中發(fā)現(xiàn)hive , oozie使用的庫(kù)需要使用latin字符集，否則在啟動(dòng)服 務(wù)時(shí)日志中會(huì)報(bào)錯(cuò)創(chuàng)建索引時(shí)超長(zhǎng)度限制。（4）安裝程序驅(qū)動(dòng)包 mysql jdbc driver總計(jì)有以下角色所在的主機(jī)需

27、要安裝這個(gè)驅(qū)動(dòng)包。 如果你把以下角色都部署 在一個(gè)主機(jī)節(jié)點(diǎn)上，那就只需要安裝一個(gè)節(jié)點(diǎn)的驅(qū)動(dòng)，否則就需要各節(jié)點(diǎn)都部署 一下。Cloudera Manager Server host,ActivityMonitor. Reports Manager, HiveMetastore Server, Hue Server, Sentry Server, Cloudera Navigator Audit Server, and Cloudera Navigator Metadata Server從以下網(wǎng)址下載需要使用的驅(qū)動(dòng)包:tar -zxvfmysql-c onn ector-java-mkdir -p

28、 /usr/share/java/cp mysql-connector-java- /usr/share/java/mysql-connector-java.jar6配置NTP時(shí)間同步服務(wù)在所有節(jié)點(diǎn)上安裝和啟用。如果是內(nèi)網(wǎng)環(huán)境，也需要另想辦法保證所有節(jié)點(diǎn) 的系統(tǒng)時(shí)間嚴(yán)格同步。yum -y install ntpservice ntpd restartchkconfig ntpd on注：請(qǐng)選擇一個(gè)節(jié)點(diǎn)做為 ntp server 直接和外部時(shí)間服務(wù)器同步，其它所 有節(jié)點(diǎn)均配置 /etc/ntp.conf ，將同步的 server 信息指向前面這個(gè)節(jié)點(diǎn)即可。7、安裝 Cloudera Manage

29、r Server 軟件我們選擇把 Cloudera Manager Server 部署在 cluster-70 節(jié)點(diǎn)上。（1）下載和分發(fā)安裝資源cloudera manager 下載地址：/cm/5/cdh parcels 下載地址：/parcels/wget -c /cm/5/cloudera-manager-el6-wget -c /parcels/5.12/CDH-wget -c/parcels/5.12/CDH-wget -c/parcels/5.12/manifest.json變更為sha,否則上傳到系統(tǒng)中后，CDH平臺(tái)反而會(huì)說找不到sha文件。將 cloudera-manager-

30、el6- 文件放到所有節(jié)點(diǎn)的 /opt/cloudera-manager 目錄下。mkdir /opt/cloudera-managercd /opttar zxvf cloudera-manager-el6-（2）創(chuàng)建 CM Server 系統(tǒng)用戶賬號(hào)useradd -system -home=/opt/cloudera-manager/cm- -no-create-home -shell=/bin/false -comment "Cloudera SCM User" cloudera-scm（3）創(chuàng)建 CM Server 的本地?cái)?shù)據(jù)存儲(chǔ)目錄登錄 cluster-70 ：

31、mkdir -p /var/lib/cloudera-scm-serverchown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server（ 4）配置所有節(jié)點(diǎn)上的 Cloudera Manager Agentssed -i 's/server_host=localhost/server_host=cluster-70/' /opt/cloudera-manager/cm-（5）創(chuàng)建和初始化 CM Server 數(shù)據(jù)庫(kù)我們的 mysql 數(shù)據(jù)庫(kù)是安裝在 cluster-71 上面，登錄 cluster-71部署一個(gè)JDBC驅(qū)

32、動(dòng)包：cp mysql-connector-java- /opt/cloudera-manager/cm-初始化腳本語(yǔ)法如下：scm_prepare_database.sh database-type options database-name username password/opt/cloudera-manager/cm- mysql -hlocalhost -uroot -pHstest2017 scm scm scm屏幕輸出信息：JAVA_HOME=/usr/java/Verifying that we can write to /opt/cloudera-manager/cm-Cr

33、eating SCM configuration file in /opt/cloudera-manager/cm-Executing: /usr/java/ -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle- connector-java.jar:/opt/cloudera-manager/cm-/opt/cloudera-manager/cm- main DbCommandExecutor INFO Successfully connected to database.All done, your SCM

34、 database is configured correctly!登錄mysql，增加一條對(duì)CM Server主機(jī)的訪問授權(quán)：grant all on scm.* TO 'scm''cluster-70' IDENTIFIED BY 'scm'登錄 cluster-70 節(jié)點(diǎn)配置 CM Server 的數(shù)據(jù)庫(kù)配置文件：/opt/cloudera-manager/cm-需要根據(jù)實(shí)際情況更新以下參數(shù)。6）創(chuàng)建 CDH Parcel 資源目錄在 cluster-70 節(jié)點(diǎn)上：mkdir -p /opt/cloudera/parcel-repo將以下已

35、經(jīng)準(zhǔn)備好的 parcel 資源文件放入該目錄：? CDH-? CDH-? manifest.jsonchown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo在其它所有節(jié)點(diǎn)上：mkdir -p /opt/cloudera/parcelschown cloudera-scm:cloudera-scm /opt/cloudera/parcels（ 7）啟動(dòng) cloudera manager server and agents直接使用 root 運(yùn)行啟動(dòng)腳本即可，腳本中已經(jīng)默認(rèn)設(shè)定為會(huì)切換到 cloudera-scm 用戶運(yùn)行進(jìn)程。在所有

36、節(jié)點(diǎn)上，先調(diào)整下文件屬主權(quán)限：chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager在 cluster-70 節(jié)點(diǎn)上啟動(dòng) CM Server ：/opt/cloudera-manager/cm-start在所有節(jié)點(diǎn)上啟動(dòng) CM agent：/opt/cloudera-manager/cm-start注：在啟動(dòng)上面的服務(wù)時(shí)，注意檢查進(jìn)程是否運(yùn)行，是否因報(bào)錯(cuò)退出。可以 檢查 ././log/下面的日志文件，分析錯(cuò)誤原因。8、登錄Cloudera Manager管理控制臺(tái)并配置 CDH服務(wù)訪問以下地址：.70:7180/cmf初始管理賬號(hào)信息

37、為： admin/admin(1)登錄后按提示操作，先是選擇 license 許可，如果沒采購(gòu)企業(yè)許可， 就選免費(fèi)版，繼續(xù)。(2) 選擇要加入CDH集群的主機(jī)，因?yàn)槲覀円呀?jīng)配置好 6臺(tái)主機(jī)的訪問關(guān) 系了，這 6 個(gè)主機(jī)會(huì)被自動(dòng)識(shí)別出來(lái)，選中然后下一步。(3) 安裝CDH服務(wù)的方式，選擇使用Parcels，因?yàn)橹耙呀?jīng)把離線安裝資 源放到系統(tǒng)默認(rèn)的路徑下了，這里會(huì)自動(dòng)識(shí)別到，點(diǎn)擊下一步將開始在CDH集群 各節(jié)點(diǎn)中分發(fā)所有需要使用的應(yīng)用的 parcels 安裝資源。( 4 )經(jīng)過一個(gè)主機(jī)正確性檢查的步驟，點(diǎn)擊完成按鈕。如果系統(tǒng)提示有參 數(shù)配置上的優(yōu)化建議，遵照處理后重新做檢驗(yàn)檢查即可。(5) 選擇一種集群服務(wù)組合，如果不確定哪個(gè)組合更合適，選擇“全部服 務(wù)”也可以。(6) 進(jìn)入集群角色分配環(huán)節(jié)，這一步比較關(guān)鍵，注意要把NameNod分配到 我們?cè)O(shè)計(jì)的cluster-70 節(jié)點(diǎn)，SecondaryNameNod分配到cluster-71 節(jié)點(diǎn)上。 注意調(diào)整HDFS的 DataNode角色，自定義分配到cluster-72 到cluster-75 這4 個(gè)節(jié)點(diǎn)上。 注意有多個(gè)服務(wù)的角色是還沒有分配的， 請(qǐng)將這些角色統(tǒng)一都分配到 cluster-71 節(jié)點(diǎn)上即可，這樣做是因?yàn)楹?jiǎn)化數(shù)據(jù)節(jié)點(diǎn)的角色和職責(zé)，便于維護(hù)，且 cluster-70 節(jié)點(diǎn)已經(jīng)被默認(rèn)分配