Auto-deploy Hadoop cluster with HDP


With the heat of Hadoop, the deployment and monitor becoming our system engineers’ focus.
More solution coming out now. such as Serengeti, but it aims for the Vsphere.
Bigtop, is a project under Apache. not bad, will give it a try.
HDP, Hortonworks Data Platform has been release recently.

Will try mention as much details doing the installation as I can.

Preparation:
1. OS, Although HDP offically support CentOS 5.x/ RHEL 5.x. we still found some issue when doing the installation.
such as missing nodes when doing the deployment(can be fixed by update php5.1 to php5.3 but will bring more issue).
so we skip CentOS 5.x and use CentOS 6.x. it’s cool and have less issue ;) , recommend install the OS with minimal packages.
# tips, please update the release file.
sed -i.bak ‘s/6.2/5.8/g’ /etc/redhat-release

2. Disable Selinux and Iptables.
/etc/sysconfig/selinux SELINUX=disabled
chkconfig iptables off
reboot

3. password less auth
ssh-keygen
ssh-copy-id target_host (don’t forget the deployment host itself.)

4. DNS, add the new nodes to DNS with PTR record. or you can add them to /etc/hosts.

Start Installation:

1. Enable Repo
# ruby repo
rpm -Uvh http://passenger.stealthymonkeys.com/rhel/6/passenger-release.noarch.rpm
# HDP
rpm -Uvh http://public-repo-1.hortonworks.com/HDP-1.0.0.12/repos/centos5/hdp-release-1.0.0.12-1.el5.noarch.rpm
# EPEL
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-7.noarch.rpm

2. Install HMC (HDP core package)
Note: when you start the HMC. it ask you download JDK. recommend download by yourself from Oracle site.

[root@hmhdp01 ~]# yum install -y hmc php-process
[root@hmhdp01 ~]# service hmc start
Do you agree to Oracle’s Java License at
/usr/share/hmc/licenses/ORACLE_JDK_LICENSE.txt?(y/n)y
Would you like us to download the JDK binaries for you?(y/n)n
Please download jdk-6u31-linux-x64.bin and jdk-6u31-linux-i586.bin from Oracle to /var/run/hmc/downloads/
Starting HMC Installer [ OK ]
Starting httpd:
Starting HMC
3. Deploy HDP.
3.1
# update puppet for the nagios. remove php-pecl-json. because php5.3 already include it by default.
line 11+ /etc/puppet/master/modules/hdp-nagios/manifests/server/packages.pp
3.2
open http://your_deploy_host/hmc/html/

Then next follow the guide. Good luck.

Snapshots:

  1. #1 by Miguel on 2012-07-28 - 19:56

    did you use a 32bit or 64bit centos?

  2. #2 by edyliu on 2012-08-10 - 22:15

    64bit

(will not be published)