INTRODUCTION HADOOP
Apache Hadoop is
a Java software framework that allows for the distributed processing of large data sets
across clusters of computers spread across the world using a simpleprogramming model.
Hadoop can be configured in the following three modes:
1. Local Standalone (Non-distributed)
2. Pseudo Distributed
1.Install CentOS
I assume you
are
starting from scratch and want to
setup a small cluster with minimum OS components
(as
far as I could minimize it). To
start,
download
the
OS setup media, and boot each machine with it. Go through the
screens of the GUI installer, keeping in mind
to do the following:
- Disable IP V 6 during Network setup
- De-select all components,
including all GUIs and even "Server"
- On the component selection page, click "Customize now" radio
button and click next
- In "Base System" group, De-select "Dialup Networking Support"
It is important (at
least in
this tutorial) to
define the host names
in your DNS server (or in
the
host file, if no
DNS
server is used). Be careful with the hosts file,
one problem was
caused by the loop
back address (127.0.0.1)
being assigned to the name of your host (in this case,master). Apparently Java listeners use the hostname to
lookup the IP address
to listen to,
and they picked up
the
loop back address
from hosts file, instead of the network address.
2.Create user
3.Firewal Configuration
(IPtables & Selinux)
3.1.IPtables:
When the OS
installation is complete, make sure to turn off and disable firewalls:
(Check
the
iptables status)
(Check
the
ip6tables
status)
It may be a good idea to update your OS
before installing hadoop:
3.2.Firewall:
Hadoop uses a lot of ports for its
internal and external communications. We've just allowed all traffic between the servers in the cluster and clients. But if you don't want to do that you can also selectively open the
required ports. I try to mention them but they can all be changed in the configuration
files. I might also miss
some
due to our config so I'd be glad if someone could point those out to me.
1.[root@PRAVEEN PRAKASAN ~]# nano /etc/selinux/config (Edit selinux=disabled) (In nano [--> crtl+X
(save) press Y <--] This command working for root)
4.Change Hostname:
5.Java Installation&
Configuration
5.1.Install Java
* We're using JDK 7 Update 03
You have to download the JDK from Oracle's website yourself as license issues prevent it from being added to
the
repositories. Chose the correct
system (probably Linux x64) and make sure to download the file ending in
-rpm (i.e. jdk-7u3-linux-x64.rpm). You might have to do this from a client machine because you need a
browser that works with the Oracle site. So on any one machine execute the following:
5.1.1Download
Execute the following commands:
(To change the permissions of the file)
(If you have followed the instructions, java is installed in /usr/java/jdk1.7.0_03)
To make sure java was installed correctly:
5.2.Configure Java:
Setup some paths in the login script:
1.[root@PRAVEEN PRAKASAN ~]# nano /etc/profile/ (add the following to the end of the profile script)
=> export JAVA_HOME=/usr/java/jdk1.7.0_03
=> export PATH=${JAVA_HOME}/bin:$PATH
=> export CLASSPATH=${JAVA_HOME}/lib/tools.jar
6.Cloudera Hadoop installation:
Cloudera has now new software scm-express which make installation of hadoop and other component easier.
You can change directory to
/etc/yum.repos.d/
Then get file using
After that you can update repository by typing yum update yum
which should be able to see all new packages by typing command yum
search hadoop)
7.Install Hadoop deamons
On namenode (namenode only for master)
On secondarynamenode (secondarynamenode only for master1)
On datanode (datanode for all slaves)
8.Adding users to the hadoop group:
Add the users(hdfs
and mapred)
to
the Hadoop group in master and slave
9.Configure HDFS:
9.1.Setup some paths in the login script:
(add the following to the end of the profile script)
export HADOOP_HOME=/usr/lib/hadoop
export PATH=${HADOOP_HOME}/bin:${PATH}
export HADOOP_CLASSPATH=${HADOOP_HOME}/hadoop-tools.jar
export HADOOP_CONF=/etc/hadoop-0.20/conf
export HADOOP_BIN=/usr/lib/hadoop/bin
export HADOOP_CORE=/etc/hadoop-0.20/conf/core-site.xml
export HADOOP_HDFS=/etc/hadoop-0.20/conf/hdfs-site.xml
export HADOOP_MAPRED=/etc/hadoop-0.20/conf/mapred-siite.xml
export HADOOP_ENV=/etc/hadoop-020/conf/hadoop-env.sh
export HADOOP_LOGS=/var/logs/hadoop
export HADOOP_STORE=/home/hadoop/
9.2.Modify the /etc/hadoop-0.20/conf/hadoop-env.sh:
(Find the following exports and modify them accordingly:)
export JAVA_HOME=/usr/java/jdk1.7.0_03
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
export HADOOP_PID_DIR=${HADOOP_HOME}/tmp/pids
export HADOOP_NAMENODE_USER=hdfs
export HADOOP_DATANODE_USER=hdfs
export HADOOP_SECONDARYNAMENODE_USER=hdfs
export HADOOP_JOBTRACKER_USER=mapred
export HADOOP_TASKTRACKER_USER=mapred
9.3.Hadoop configuration
deafault-site.xml file
1.conf/core-site.xml
<!-- In: conf/core-site.xml -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>${HADOOP_STORE}/tmp</value>
</property>
<property>
<name>fs.default.name</name>
</property>
</configuration>
2.conf/hdfs-site.xml
- In:
conf/hdfs-site.xml -->
<configuration>
<property>
<name>dfs.namenode.plugins</name>
<value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
<description>
Comma-separated list of namenode plug-ins to be activated.
</description>
</property>
<property>
<name>dfs.datanode.plugins</name>
<value>org.apache.hadoop.thriftfs.DatanodePlugin</value>
<description>
Comma-separated list of datanode plug-ins to be activated.
</description>
</property>
<property>
<name>dfs.thrift.address</name>
<value>0.0.0.0:9090</value>
</property>
<property>
<name>dfs.permissions</name>
<value>true</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>${HADOOP_STORE}/datastore/name</value>
<final>true</final>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>${HADOOP_STORE}/dfs/namesecondary</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>${HADOOP_STORE}/datastore/data</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<configuration>
3.conf/mapred-site.xml
<!-- In: conf/mapred-site.xml -->
<configuration>
<!-- Enable Cloudera Desktop plugins
<property>
<name>mapred.jobtracker.plugins</name>
<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
<description>
Comma-separated list of jobtracker plug-ins to be activated.
</description>
</property>
<property>
<name>mapred.job.tracker</name>
</property>
<property>
<name>mapred.system.dir</name>
<value>${HADOOP_STORE}/taskstore/system</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>${HADOOP_STORE}/taskstore/mapred</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>4</value>
</property>
<property>
<name>mapred.userlog.retain.hours</name>
<value>8</value>
</property>
<configuration>
10.Create Hadoop Diractory & permissions:
11.IP hosts configuration
Edit the /etc/hosts:
(add the master and slaves ip and hostname like this)
|
192.168.1.2
|
master
|
|
192.168.1.3
|
slave0
|
|
192.168.1.4
|
slave1
|
12.Define the masters and slaves (Only on master node):
In this case hostname of master
master
Now modify the slaves file:
Slaves in our example are slave0 and slave1:
slave0
slave1
13.TO Configure Slaves_Datanode
14.Run Hadoop Cluster
14.1.NameNode Format
4.2.Deamons Run Command
OR
OR
OR
OR
15.MapReduce For Example
-record 10
