centos 6.8下 CDH5.13.3安装与配置（详细部署）

centos 6.8下 CDH5.13.3安装与配置（详细部署）

一、安装前准备

1.1、下载地址

wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz

http://archive.cloudera.com/cdh6/parcels/5.13.3/

wget http://archive.cloudera.com/cdh6/parcels/5.13.3/manifest.json

wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1

1.2、防火墙关闭

/etc/init.d/iptables stop

chkconfig iptables off

1.3、设置selinux

vi /etc/selinux/config

1.4、设置时间同步ntp(每台机器)

yum -y install ntpdate

crontab -e

*/5 * * * * ntpdate 0.rhel.pool.ntp.org

保存退出

vi /etc/hosts（hadoop01增加）

注意：Cloudera-Manager台需要增加127.0.0.1 hadoop01

127.0.0.1 hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

vi /etc/hosts（其他两台hadoo02、hadoop03）

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

二、依赖包安装（每台机器）

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi openssh-clients

三、jdk安装（每台机器）

tar zxvf jdk-8u111-linux-x64.gz

mv jdk1.8.0_111 /usr/local/jdk

配置环境变量

vi /etc/profile

##jdk

export JAVA_HOME=/usr/local/jdk

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

让环境变量生产

source /etc/profile

四、ssh免密码登录配置

ssh-keygen -t rsa

ssh-copy-id 192.168.1.201

ssh-copy-id 192.168.1.202

ssh-copy-id 192.168.1.202

五、安装mysql(在haoop01上安装)

5.1、安装依赖包

yum -y install libaio*

yum -y install libnuma*

5.2、创建mysql用户

useradd mysql

5.3、创建目录

mkdir -p /data/app

mkdir /data/app/mysql3306

mkdir /data/app/undolog

5.4、解压文件

tar zxvf mysql-5.6.41-linux-glibc2.12-x86_64.tar.gz

mv mysql-5.6.41-linux-glibc2.12-x86_64 /data/app/mysql5.6

mkdir /data/app/mysql5.6/logs

5.5、配置my.cnf

vi /data/app/mysql5.6/my.cnf

[client]

socket=/data/app/mysql5.6/mysqld.sock

port=3306

[mysql]

prompt=\\u@\\d \\r:\\m:\\s>

no-auto-rehash

default-character-set=utf8

[mysqld_safe]

log-error=/data/app/mysql3306/mysqld.error

[mysqld]

socket=/data/app/mysql5.6/mysqld.sock

pid-file=/data/app/mysql5.6/mysqld.pid

basedir=/data/app/mysql5.6

datadir=/data/app/mysql3306

port=3306

server-id=1201

character-set-server=utf8

skip-external-locking

#skip-name-resolve

max_connections=1000

max_connect_errors=1000

max_allowed_packet=128M

table_definition_cache=500

table_open_cache=500

sort_buffer_size = 1M

tmp_table_size = 64M

read_buffer_size = 1M

read_rnd_buffer_size = 1M

myisam_sort_buffer_size = 64M

thread_cache_size = 8

query_cache_type=0

query_cache_size=0

lower_case_table_names = 1

log_bin_trust_function_creators = 1

#################slow log####################

slow-query_log=1

slow-query_log_file=/data/app/mysql5.6/logs/mysql.slow

long_query_time=2

####################binlog######################

log-bin=mysql-bin

binlog-format=ROW

expire_logs_days=10

log-slave-updates=1

sync_binlog=0

auto_increment_increment=2

auto_increment_offset=1

###

###

###

innodb_undo_directory=/data/app/undolog

innodb_undo_logs = 128

innodb_undo_tablespaces = 3

################INNODB################

transaction-isolation=READ-COMMITTED

innodb_buffer_pool_size=2G

innodb_flush_log_at_trx_commit=2

innodb_strict_mode=1

innodb_flush_method=O_DIRECT

innodb_file_format=Barracuda

innodb_log_files_in_group=3

innodb_file_per_table=1

innodb_io_capacity=500

innodb_support_xa=1

[mysqldump]

quick

max_allowed_packet=128M

[mysqlhotcopy]

interactive_timeout = 28400

5.6、增加权限

chown -R mysql:mysql /data/app

5.7、配置环境变量

vi /etc/profile

##mysql

export MYSQL_HOME=/data/app/mysql5.6

export MY_BASEDIR_VERSION=/data/app/mysql5.6

export PATH=/data/app/mysql5.6/bin:$PATH

export LD_LIBRARY_PATH=/data/app/mysql5.6/lib

环境生效

source /etc/profile

5.8、初始化数据库

/data/app/mysql5.6/scripts/mysql_install_db --user=mysql --basedir=/data/app/mysql5.6 --datadir=/data/app/mysql3306

5.9、配置启动文件

cp /data/app/mysql5.6/support-files/mysql.server /etc/init.d/mysqld

vi /etc/init.d/mysqld

basedir=/data/app/mysql5.6

datadir=/data/app/mysql3306

保存退出

chmod +x /etc/init.d/mysqld

chkconfig mysqld on

启动

/etc/init.d/mysqld start

5.10、初始化密码

/data/app/mysql5.6/bin/mysqladmin -u root password 'yb2018#08'

5.11、设置远程权限

# mysql -uroot -pyb2018#08

Warning: Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 2

Server version: 5.6.41-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

root@(none) 10:28:44>grant all on *.* to 'scm'@'localhost' identified by 'temp' with grant option;

Query OK, 0 rows affected (0.00 sec)

root@(none) 10:28:58>grant all on *.* to 'cdh'@'%' identified by 'cdh#2018' with grant option;

Query OK, 0 rows affected (0.00 sec)

root@(none) 10:29:08>create database hive;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:16>create database ooz;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:27>create database hue;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:36>flush privileges;

Query OK, 0 rows affected (0.00 sec)

六、安装Cloudera-Manager

6.1、解压包到指定目录，所有服务器都要

mkdir -p /data/cloudera-manager

tar zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz -C /data/cloudera-manager

6.2创建cloudera-scm用户(所有节点)

useradd -r -d /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm

id cloudera-scm

6.3、配置从节点cloudera-manger-agent指向主节点服务器（hadoop02,hadoop03配置）

vi /data/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini

将server_host改为CMS所在的主机名即hadoop01

6.4、主节点中创建parcel-repo仓库目录（这个只在主节点hadoop01上需要）

mkdir -p /data/cloudera/parcel-repo

chown cloudera-scm:cloudera-scm /data/cloudera/parcel-repo

cp CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 manifest.json /data/cloudera/parcel-repo/

cd /data/cloudera/parcel-repo/

mv CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha

注意：其中CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 后缀要把1去掉

6.5、所有节点创建parcels目录（所有节点）

mkdir -p /data/cloudera/parcels

chown cloudera-scm:cloudera-scm /data/cloudera/parcels

解释：Clouder-Manager将CDHs从主节点的/data/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/data/cloudera/parcels目录中

6.6、初始脚本配置数据库scm_prepare_database.sh(在主节点上)

/data/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql -uscm -ptemp scm scm temp

如果报错，操作

mkdir /usr/share/java

cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

上图说明成功

成功把记住把#127.0.0.1 localhost hadoop01 注释掉，不然后面就会有问题

vi /etc/hosts

#127.0.0.1 hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

6.7、启动主节点cloudera-scm-server(hadoop01操作)

cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server

chmod +x /etc/init.d/cloudera-scm-server

启动

/etc/init.d/cloudera-scm-server start

如果报File not found: /usr/sbin/cmf-server错误

vi /etc/init.d/cloudera-scm-server

找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}

修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

启动

/etc/init.d/cloudera-scm-server start

6.8、启动cloudera-scm-agent所有节点(hadoop02,hadoop03操作)

mkdir /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-agent

cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent

chmod +x /etc/init.d/cloudera-scm-agent

/etc/init.d/cloudera-scm-agent start

vi /etc/init.d/cloudera-scm-agent

找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}

修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

启动

/etc/init.d/cloudera-scm-agent start

6.9、登录用户admin 密码admin

http://192.168.1.201:7180

继续

选择免费，继续

继续

选择当前管理主机选择hadoop02,hadoop03,继续

点击更多选项配置

修改好后，保存修改

重启后才后出下如下图

/etc/init.d/cloudera-scm-server restart

重启好后才能如上图继续，因为默认/opt下安装的

继续

每台执行以下

echo 10 >/proc/sys/vm/swappiness

echo never >/sys/kernel/mm/transparent_hugepage/defrag

echo never >/sys/kernel/mm/transparent_hugepage/enabled

以图根据自己需求选择不一样点继续

以上Datanode修改两台

把连接主机名，数据库名称，用户名，密码输入进去，点击测试连接都Succesful成功后，点击继续

根据自己修改点击继续

继续

安装完成