CDH3 Pseudo installation on Ubuntu

1) Do not create username as hadoop as you will have issues in installation.

2) Install Java

Copy the java jdk on to desktop.

$ sudo cp jdk-6u30-linux-x** /usr/local

$ cd /usr/local

$ sudo sh jdk-6u30-linux-x**

3) Install CDH3 package

Go to - http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH3/CDH3u6/CDH3-

Installation-Guide/CDH3-Installation-Guide.html

Click on Installing CDH3 on Ubuntu and Debian Systems

Click on - this link for a Maverick system

Install using GDebi package installer or save it and issue the command below

$ sudo dpkg -i Downloads/cdh3-repository_1.0_all.deb

$ sudo apt-get update

4) Install Hadoop

$ apt-cache search hadoop - Must show all available Hadoop Packages

$ sudo apt-get install hadoop-0.20 hadoop-0.20-native

sudo apt-get install hadoop-0.20-<daemon type> install all Daemons

5) Set Java and Hadoop Home

Using command: gedit ~/.bashrc

# Set Hadoop-related environment variables

export HADOOP_HOME=/usr/lib/hadoop

export PATH=$PATH:/usr/lib/hadoop/bin

# Set JAVA_HOME

export JAVA_HOME=/usr/local/jdk1.6.0_30

export PATH=$PATH:/usr/local/jdk1.6.0_30/bin

close terminals and open new one and test

echo $JAVA_HOME

echo $HADOOP HOME

6) Adding dedicated users hdfs and mapred to hadoop group

$ sudo gpasswd -a hdfs hadoop

$ sudo gpasswd -a mapred hadoop

7) Configuration

$ cd /usr/lib/hadoop/conf

Set Java Home in hadoop-env.sh

$ sudo gedit hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.6.0_30

8) core-site.xml

<property>

<name>hadoop.tmp.dir</name>

<value>/usr/lib/hadoop/tmp</value>

</property>

<property>

<name>fs.default.name</name>

<value>hdfs://localhost:8020</value>

</property>

$ sudo mkdir /usr/lib/hadoop/tmp

$ sudo chmod 750 /usr/lib/hadoop/tmp/

$ sudo chown hdfs:hadoop /usr/lib/hadoop/tmp/

9) hdfs-site.xml

<property>

<name>dfs.permissions</name>

<value>false</value>

</property>

<property>

<name>dfs.name.dir</name>

<value>/storage/name</value>

</property>

<property>

<name>dfs.data.dir</name>

<value>/storage/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

$ sudo mkdir /storage

$ sudo chmod 775 /storage/

$ chown hdfs:hadoop /storage/

10) mapred-site.xml

<property>

<name>mapred.job.tracker</name>

<value>hdfs://localhost:8021</value>

</property>

<property>

<name>mapred.system.dir</name>

<value>/mapred/system</value>

</property>

<property>

<name>mapred.local.dir</name>

<value>/mapred/local</value>

</property>

<property>

<name>mapred.temp.dir</name>

<value>/mapred/temp</value>

</property>

$ sudo mkdir /mapred

$ sudo chmod 775 /mapred

$ sudo chown mapred:hadoop /mapred

11) User Assignment

export HADOOP_NAMENODE_USER=hdfs

export HADOOP_SECONDARYNAMENODE_USER=hdfs

export HADOOP_DATANODE_USER=hdfs

export HADOOP_JOBTACKER_USER=mapred

export HADOOP_TASKTRACKER_USER=mapred

12) Format namenode

$ cd /usr/lib/hadoop/bin/

$ sudo -u hdfs hadoop namenode -format

You must get a successfully formatted message. Otherwise, check the error and correct it.

13) Start Daemons

$ sudo /etc/init.d/hadoop-0.20-namenode start

$ sudo /etc/init.d/hadoop-0.20-secondarynamenode start

$ sudo /etc/init.d/hadoop-0.20-jobtracker start

$ sudo /etc/init.d/hadoop-0.20-datanode start

$ sudo /etc/init.d/hadoop-0.20-tasktracker start

Check for any errors in /var/log/hadoop-0.20 for each daemon

check all ports are opened using $netstat -ptlen

14) Check UI

localhost:50070 - Hadoop Admin

localhost:50030 - Mapreduce