Archive for July, 2015

OS Tuning tips for Hadoop Cluster

1.Decrease swappiness.
Reason:
A value from 0 to 100 which controls the degree to which the system swaps. A high value prioritizes system performance, aggressively swapping processes out of physical memory when they are not active. A low value prioritizes interactivity and avoids swapping processes out of physical memory for as long as possible, which decreases response latency. The default value is 60.

Default value: 60
Recommend value: 5
Online Change: Y
Action:
# update online
echo 5 > /proc/sys/vm/swappiness

# update permanently , edit /etc/sysctl.conf and add following line:
vm.swappiness = 5
Read the rest of this entry »

No Comments

Migrate existing hadoop to CDH

Don’t need to sell CDH’s benefits. you should know it before want to migrate 🙂

Very Important, The following has been tested in my lab, all goes fine. can’t grantee if also works for you.
I migrate from Apache Hadoop 2.2 to CDH 5.3 or 5.4 all works.

## Backup namenode
# cd /mnt/hadoop/hdfs/name
# tar -cvf /root/nn_backup_data.tar .

.
./current/fsimage
..
./current/edits
./image/
./image/fsimage

## Install CDH WITHOUT create any service.
Read the rest of this entry »

No Comments