Posts Tagged hadoop

OS Tuning tips for Hadoop Cluster

1.Decrease swappiness.
Reason:
A value from 0 to 100 which controls the degree to which the system swaps. A high value prioritizes system performance, aggressively swapping processes out of physical memory when they are not active. A low value prioritizes interactivity and avoids swapping processes out of physical memory for as long as possible, which decreases response latency. The default value is 60.

Default value: 60
Recommend value: 5
Online Change: Y
Action:
# update online
echo 5 > /proc/sys/vm/swappiness

# update permanently , edit /etc/sysctl.conf and add following line:
vm.swappiness = 5
Read the rest of this entry »

No Comments

Migrate existing hadoop to CDH

Don’t need to sell CDH’s benefits. you should know it before want to migrate 🙂

Very Important, The following has been tested in my lab, all goes fine. can’t grantee if also works for you.
I migrate from Apache Hadoop 2.2 to CDH 5.3 or 5.4 all works.

## Backup namenode
# cd /mnt/hadoop/hdfs/name
# tar -cvf /root/nn_backup_data.tar .

.
./current/fsimage
..
./current/edits
./image/
./image/fsimage

## Install CDH WITHOUT create any service.
Read the rest of this entry »

No Comments

Auto-deploy Hadoop cluster with HDP

With the heat of Hadoop, the deployment and monitor becoming our system engineers’ focus.
More solution coming out now. such as Serengeti, but it aims for the Vsphere.
Bigtop, is a project under Apache. not bad, will give it a try.
HDP, Hortonworks Data Platform has been release recently.

Will try mention as much details doing the installation as I can.
Read the rest of this entry »

2 Comments