Error: Permission denied: user=dr.who, access=READ_EXECUTE, inode="/user/spark":spark:spark:drwxr-x--x or Spark-shell or Spark-submit

ERROR SparkContext: Error initializing SparkContext.
org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:216)
        at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:145)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6609)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6591)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:6543)



Solutions:

sudo -u hdfs hadoop fs -mkdir /user/root
sudo -u hdfs hadoop fs -chown root /user/root

Flume Maven Build Error: Mysql Source

Tests run: 20, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 0.285 sec <<< FAILURE! - in org.keedio.flume.source.SQLSourceHelperTest
checkStatusFileCorrectlyUpdated(org.keedio.flume.source.SQLSourceHelperTest)  Time elapsed: 0.024 sec  <<< FAILURE!
java.lang.AssertionError: expected:<10> but was:<0>
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.failNotEquals(Assert.java:834)
 at org.junit.Assert.assertEquals(Assert.java:645)
 at org.junit.Assert.assertEquals(Assert.java:631)
 at org.keedio.flume.source.SQLSourceHelperTest.checkStatusFileCorrectlyUpdated(SQLSourceHelperTest.java:170)

getCustomQuery(org.keedio.flume.source.SQLSourceHelperTest)  Time elapsed: 0.005 sec  <<< ERROR!
org.apache.flume.conf.ConfigurationException: status.column.name property not set
 at org.keedio.flume.source.SQLSourceHelper.checkMandatoryProperties(SQLSourceHelper.java:313)
 at org.keedio.flume.source.SQLSourceHelper.<init>(SQLSourceHelper.java:100)
 at org.keedio.flume.source.SQLSourceHelperTest.getCustomQuery(SQLSourceHelperTest.java:239)


Results :

Failed tests: 
  SQLSourceHelperTest.checkStatusFileCorrectlyUpdated:170 expected:<10> but was:<0>
Tests in error: 
  SQLSourceHelperTest.getCustomQuery:239 ? Configuration status.column.name prop...

Solution:
Use this commands
#mvn clean package -DskipTests=true

Error: GPG key retrieval failed: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'archive.cloudera.com'"

Cloudera Manager Installation Problem


[root@master cloudera-manager-installer]# tail -f 2.install-oracle-j2sdk1.7.log
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-j2sdk1.7.x86_64 0:1.7.0+update67-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================
 Package             Arch       Version              Repository            Size
========================================================================
Installing:
 oracle-j2sdk1.7     x86_64     1.7.0+update67-1     cloudera-manager     135 M

Transaction Summary
========================================================================
Install       1 Package(s)

Total download size: 135 M
Installed size: 279 M
Downloading Packages:
warning: rpmts_HdrFromFdno: Header V4 DSA/SHA1 Signature, key ID e8f86acd: NOKEY
Retrieving key from http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera


GPG key retrieval failed: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'archive.cloudera.com'"

Solutions:

Install Perl


[root@master cloudera-manager-installer]# yum install perl -y

Overview of Apache Flink: the 4G of Big Data Analytics Frameworks

This is an introductory level talk about Apache Flink: a multi-purpose Big Data analytics framework leading a movement towards the unification of batch and stream processing or stream processing-first in the open source. With the many technical innovations it brings along with its unique vision and philosophy, it is considered the 4 G (4th Generation) of Big Data Analytics frameworks providing the only hybrid (Real-Time Streaming + Batch) open source distributed data processing engine supporting many use cases.

In this talk, you will learn more about:

1. What is Apache Flink stack? Its streaming dataflow execution engine, APIs and domain-specific libraries for batch, streaming, machine learning and graph processing.

2. How Apache Flink integrates with Hadoop and other open source tools for data input and output as well as deployment?

3. Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark?

4. How Apache Flink is used at Capital One?

Hadoop Cluster:Passwordless SSH setup on CentOs In 3 Steps

Step 1:

[root@pravi ~]# yum install openssh-clients -y 

[root@pravi ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
e7:55:ae:cd:e0:b3:91:4a:36:53:a7:d7:62:d2:9a:d8 root@master.techvantage.com
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|              .  |
|             o   |
|        S . + o  |
|         o + X . |
|          * O B .|
|         o * X . |
|          o E    |
+-----------------+


~/.ssh/id_rsa [private key]
~/.ssh/id_rsa.pub [public key]

Step 2:

[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub <slave-ip>
Now try logging into the machine, with "ssh 'slave'", and check in:

  .ssh/authorized_keys


to make sure we haven't added extra keys that you weren't expecting.

Step 3

[root@master ~]# ssh <slave-hostname>

CDH3 HBase Installation

Install the HBase package on master node

#yum install hadoop-hbase-master

#yum install hadoop-zookeeper-server

#yum install hadoop-hbase-regionserver

Configuring Hadoop on master node

Now you have to configure the hadoop on master in /usr/local/hadoop/conf:- hadoop-env.sh

Set your JAVA_HOME to where the java package is installed (this should be already set from hadoop

installation). On CentOS 6:

The java implementation to use. Required.

#export JAVA_HOME=/usr/java/jdk1.7.0_03

Add the hbase path to the HADOOP_CLASSPATH:

Extra Java CLASSPATH elements. Optional.

#export HADOOP_CLASSPATH=/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar:/usr/lib/hbase-0.90.6-cdh3u4-tests.jar:/usr/lib/hbase/conf:/usr/lib/hbase/lib/zookeeper.jar

Copy the updated hadoop-env.sh to all slaves:

#hadoop@master:~$ scp -r /usr/lib/hadoop/conf/hadoop-env.sh ip_address_of_slavenode_i:/usr/lib/hadoop/conf/

Formatting the HDFS

Format the filesystem (which simply initializes the directory specified by the dfs.name.dir variable on the

namenode):

#hadoop@master:~$ /usr/lib/hadoop/bin/hadoop namenode -format

Configuring HBase

Now, login as hadoop and go to its conf directory:

#superuser@master:/usr/local$ sudo -i -u hdfs

without requesting password)

#hadoop@master:~$ cd /usr/lib/hbase/conf

(log in as hadoop user a/ Then update the Java classpath in hbase-env.sh, just like you did for hadoop-env.sh Change

The java implementation to use. Required.

#export JAVA_HOME=/usr/lib/j2sdk1.5-sun

to

The java implementation to use. Required.

#export JAVA_HOME=/usr/java/jdk1.7.0_03

and

add the hadoop conf directory to the HBASE_CLASSPATH:

#export HBASE_CLASSPATH=/usr/local/hadoop/conf

A distributed HBase depends on a running ZooKeeper cluster. All participating nodes and clients need to be

able to get to the running ZooKeeper cluster. HBase by default manages a ZooKeeper cluster for you, or you

can manage it on your own and point HBase to it. To toggle HBase management of ZooKeeper, use the

HBASE_MANAGES_ZK variable in ${HBASE_HOME}/conf/hbase-env.sh. This variable, which defaults to

true, tells HBase whether to start/stop the ZooKeeper quorum servers alongside the rest of the servers. So,

for the master you will want to say:

#export HBASE_MANAGES_ZK=true

b/ Inside hbase-site.xml, configure hbase.roodir and hbase.master. The result should look something like

<configuration>

        <property>

                <name>hbase.master</name>

                <value>master:60000</value>

                <description>The host and port that the HBase master runs at.</description>

        </property>

        <property>

                <name>hbase.zookeeper.quorum</name>

                <value>master</value>

        </property>

        <property>

                <name>hbase.cluster.distributed</name>

                <value>true</value>

                <description>The mode the cluster will be in. Possible values ar

                false: standalone and pseudo-distributed setups with managed Zookeeper

                ntrue: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)           <description>                                                           

        </property>

        <property>

                <name>hbase.zookeeper.property.maxClientCnxns</name>

                <value>300</value>

        </property>

        <property>

                <name>hbase.rootdir</name>

                <value>hdfs://master:8020/hbase</value>

        </property>

        <property>

                <name>hbase.tmp.dir</name>

                <value>/tmp/hbase</value>

                <description>Temporary directory on the local filesystem.</description>

        </property>

        <property>

                <name>hfile.block.cache.size</name>

                <value>0.2</value>

        </property>

        <property>

                <name>hbase.regionserver.lease.period</name>

                <value>300000</value>

        </property>

        <property>

                <name>zookeeper.session.timeout</name>

                <value>300000</value>

        </property>

</configuration>

Remote copy hbase folder from master node to slave nodes

Run the following command from the master node for each slave node:

#hadoop@master:~$ scp -r /usr/lib/hbase  ip_address_of_slavenode_i:/usr/lib/hbase

This should copy our entire set-up to the node without having to go through unzipping and all the

subsequent steps from that.

Start up HBase

#service hadoop-hbase-master start

#service hadoop-zookeeper-server start

#service hadoop-hbase-regionserver start

Check if HBase is up and running

Open up the HBase Web UI e.g. http://master_hostname:60010/

Installing cdh3 on Centos

INTRODUCTION HADOOP

Apache Hadoop is a Java software framework that allows for the distributed processing of large data sets across clusters of computers spread across the world using a simpleprogramming model.

Hadoop can be configured in the following three modes:

1. Local Standalone (Non-distributed)

2. Pseudo Distributed

3. Fully Distributed

1.Install CentOS

I assume you are starting from scratch and want to setup a small cluster with minimum OS components (as far as I could minimize it). To start, download the OS setup media, and boot each machine with it. Go through the screens of the GUI installer, keeping in mind to do the following:

- Disable IP V 6 during Network setup

- De-select all components, including all GUIs and even  "Server"

- On the component selection page, click "Customize now" radio button and click next

- In "Base System" group, De-select "Dialup Networking Support"

It is important (at least in this tutorial) to define the host names in your DNS server (or in the host file, if no DNS

server is used). Be careful with the hosts file, one problem was caused by the loop back address (127.0.0.1) being assigned to the name of your host (in this case,master). Apparently Java listeners use the hostname to lookup the IP address to listen to, and they picked up the loop back address from hosts file, instead of the network address.

2.Create user

3.Firewal Configuration (IPtables & Selinux)

3.1.IPtables:

When the OS installation is complete, make sure to turn off and disable firewalls:

1.[root@PRAVEEN PRAKASAN ~]# service iptables save

2.[root@PRAVEEN PRAKASAN ~]# service iptables stop

3.[root@PRAVEEN PRAKASAN ~]# chkconfig iptables off

4.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/iptables status

(Check the iptables status)

5.[root@PRAVEEN PRAKASAN ~]# service ip6tables stop

6.[root@PRAVEEN PRAKASAN ~]# chkconfig 1p6tables off

7.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/ip6tables status

(Check the ip6tables status)

It may be a good idea to update your OS before installing hadoop:

1.[root@PRAVEEN PRAKASAN ~]# yum update

3.2.Firewall:

Hadoop uses a lot of ports for its internal and external communications. We've just allowed all traffic between the servers in the cluster and clients. But if you don't want to do that you can also selectively open the required ports. I try to mention them but they can all be changed in the configuration files. I might also miss some due to our config so I'd be glad if someone could point those out to me.

1.[root@PRAVEEN PRAKASAN ~]# nano /etc/selinux/config (Edit selinux=disabled) (In nano [--> crtl+X (save) press Y <--] This command working for root)

2.[root@PRAVEEN PRAKASAN ~]# /usr/sbin/sestatus      (Check the selinux)

3.[root@PRAVEEN PRAKASAN ~]# init 6 (restart the system)

4.Change Hostname:

1.[root@PRAVEEN PRAKASAN ~]# hostname master

2.[root@PRAVEEN PRAKASAN ~]# nano /etc/sysconfig/network

5.Java Installation& Configuration

5.1.Install Java

* We're using JDK 7 Update 03

You have to download the JDK from Oracle's website yourself as license issues prevent it from being added to the repositories. Chose the correct system (probably Linux x64) and make sure to download the file ending in

-rpm (i.e. jdk-7u3-linux-x64.rpm). You might have to do this from a client machine because you need a browser that works with the Oracle site. So on any one machine execute the following:

5.1.1Download

1.[root@PRAVEEN PRAKASAN ~]# mkdir /home/projecte

2.[root@PRAVEEN PRAKASAN ~]# cd /home/projecte/

3.[root@PRAVEEN PRAKASAN projecte]# wget  http://download.oracle.com/otn-

pub/java/jdk/7u3-b04/jdk-7u3-linux-x64.rpm

Execute the following commands:

1.[root@PRAVEEN PRAKASAN projecte]# chmod a+x jdk-7u3-linux-x64-rpm

(To change the permissions of the file)

2.[root@PRAVEEN PRAKASAN projecte]# rpm -ivh jdk-7u3-linux-x64-rpm

(If you have followed the instructions, java is installed in /usr/java/jdk1.7.0_03)

To make sure java was installed correctly:

1.[root@PRAVEEN PRAKASAN projecte]# java -version

5.2.Configure Java:

Setup some paths in the login script:

1.[root@PRAVEEN PRAKASAN ~]# nano /etc/profile/ (add the following to the end of the profile script)

=> export JAVA_HOME=/usr/java/jdk1.7.0_03

=> export PATH=${JAVA_HOME}/bin:$PATH

=> export CLASSPATH=${JAVA_HOME}/lib/tools.jar

6.Cloudera Hadoop installation:

Cloudera has now new software scm-express which make installation of hadoop and other component easier.

    You can change directory to /etc/yum.repos.d/

1.[root@PRAVEEN PRAKASAN ~]# cd /etc/yum.repos.d/

    Then get file using

2.[root@PRAVEEN PRAKASAN yum.repos.d]# wget

http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo

    After that you can update repository by typing yum update yum

3.[root@PRAVEEN PRAKASAN yum.repos.d]# yum update yum -y

    which should be able to see all new packages by typing command yum search hadoop)

5.[root@PRAVEEN PRAKASAN yum.repos.d]# yum searsh hadoop

7.Install Hadoop deamons

    On namenode (namenode only for master)

1.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20

2.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20-namenode

3.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20-jobtracker

    On secondarynamenode (secondarynamenode only for master1)

1.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20

2.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20-secondarynamenode

    On datanode (datanode for all slaves)

1.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20

2.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20-datanode

3.[root@PRAVEEN PRAKASAN ~]# yum -y install hadoop-0.20-tasktracker

8.Adding users to the hadoop group:

Add the users(hdfs and mapred) to the Hadoop group in master and slave

1.[root@master /]# sudo gpasswd -a hdfs hadoop

2.[root@master /]# sudo gpasswd -a mapred hadoop

9.Configure HDFS:

9.1.Setup some paths in the login script:

1.[root@PRAVEEN PRAKASAN ~]# nano /etc/profile

(add the following to the end of the profile script)

  export HADOOP_HOME=/usr/lib/hadoop

  export PATH=${HADOOP_HOME}/bin:${PATH}

  export HADOOP_CLASSPATH=${HADOOP_HOME}/hadoop-tools.jar

2.[root@PRAVEEN PRAKASAN ~]# ee .bash_profile

  export HADOOP_CONF=/etc/hadoop-0.20/conf

  export HADOOP_BIN=/usr/lib/hadoop/bin

  export HADOOP_CORE=/etc/hadoop-0.20/conf/core-site.xml

  export HADOOP_HDFS=/etc/hadoop-0.20/conf/hdfs-site.xml

  export HADOOP_MAPRED=/etc/hadoop-0.20/conf/mapred-siite.xml

  export HADOOP_ENV=/etc/hadoop-020/conf/hadoop-env.sh

  export HADOOP_LOGS=/var/logs/hadoop

  export HADOOP_STORE=/home/hadoop/ 

9.2.Modify the /etc/hadoop-0.20/conf/hadoop-env.sh:

1.[root@PRAVEEN PRAKASAN ~]# nano /etc/hadoop-0.20/conf/hadoop-env.sh

(Find the following exports and modify them accordingly:)

  export JAVA_HOME=/usr/java/jdk1.7.0_03

  export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true

  export HADOOP_PID_DIR=${HADOOP_HOME}/tmp/pids

  export HADOOP_NAMENODE_USER=hdfs

  export HADOOP_DATANODE_USER=hdfs

  export HADOOP_SECONDARYNAMENODE_USER=hdfs

  export HADOOP_JOBTRACKER_USER=mapred

  export HADOOP_TASKTRACKER_USER=mapred

9.3.Hadoop configuration deafault-site.xml file

1.conf/core-site.xml

1.[root@PRAVEEN PRAKASAN ~]# ee $HADOOP_CORE

<!-- In: conf/core-site.xml -->

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>${HADOOP_STORE}/tmp</value>

</property>

<property>

<name>fs.default.name</name>

<value>www.hdfs.com:8020</value>

</property>

</configuration>

2.conf/hdfs-site.xml

1.[root@PRAVEEN PRAKASAN ~]# ee $HADOOP_HDFS

- In: conf/hdfs-site.xml -->

<configuration>

<property>

<name>dfs.namenode.plugins</name>

<value>org.apache.hadoop.thriftfs.NamenodePlugin</value>

<description>

Comma-separated list of namenode plug-ins to be activated.

</description>

</property>

<property>

<name>dfs.datanode.plugins</name>

<value>org.apache.hadoop.thriftfs.DatanodePlugin</value>

<description>

Comma-separated list of datanode plug-ins to be activated.

</description>

</property>

<property>

<name>dfs.thrift.address</name>

<value>0.0.0.0:9090</value>

</property>

<property>

<name>dfs.permissions</name>

<value>true</value>

</property>

<property>

<name>dfs.name.dir</name>

<value>${HADOOP_STORE}/datastore/name</value>

<final>true</final>

</property>

<property>

<name>fs.checkpoint.dir</name>

<value>${HADOOP_STORE}/dfs/namesecondary</value>

<final>true</final>

</property>

<property>

<name>dfs.data.dir</name>

<value>${HADOOP_STORE}/datastore/data</value>

<final>true</final>

</property>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<configuration>

3.conf/mapred-site.xml

1.[root@PRAVEEN PRAKASAN ~]# ee $HADOOP_MAPRED

<!-- In: conf/mapred-site.xml -->

<configuration>

<!-- Enable Cloudera Desktop plugins

<property>

<name>mapred.jobtracker.plugins</name>

<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>

<description>

Comma-separated list of jobtracker plug-ins to be activated.

</description>

</property>

<property>

<name>mapred.job.tracker</name>

<value>www.hdfs.com:8021</value>

</property>

<property>

<name>mapred.system.dir</name>

<value>${HADOOP_STORE}/taskstore/system</value>

<final>true</final>

</property>

<property>

<name>mapred.local.dir</name>

<value>${HADOOP_STORE}/taskstore/mapred</value>

<final>true</final>

</property>

<property>

<name>mapred.tasktracker.map.tasks.maximum</name>

<value>4</value>

</property>

<property>

<name>mapred.tasktracker.reduce.tasks.maximum</name>

<value>4</value>

</property>

<property>

<name>mapred.userlog.retain.hours</name>

<value>8</value>

</property>

    <configuration>

10.Create Hadoop Diractory & permissions:

1.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/tmp

2.[root@PRAVEEN PRAKASAN ~]# chmod 750 $HADOOP_STORE/tmp

3.[root@PRAVEEN PRAKASAN ~]# chwon hdfs:hadoop $HADOOP_STORE/tmp

4.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/datastore

5.[root@PRAVEEN PRAKASAN ~]#  chmod 775 $HADOOP_STORE/datastore

6.[root@PRAVEEN PRAKASAN ~]# chown hdfs:hadoop $HADOOP_STORE/datastore

7.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/datastore/name

8.[root@PRAVEEN PRAKASAN ~]# chown -R hdfs $HADOOP_STORE/datastore/name

9.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/datastore/data

10.[root@PRAVEEN PRAKASAN ~]# chown -R hdfs $HADOOP_STORE/datastore/data

11.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/taskstore

12.[root@PRAVEEN PRAKASAN ~]# chmod 775 $HADOOP_STORE/taskstore

13.[root@PRAVEEN PRAKASAN ~]# chown mapred:hadoop $HADOOP_STORE/taskstore

14.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/taskstore/system

15.[root@PRAVEEN PRAKASAN ~]# chown -R hdfs $HADOOP_STORE/taskstore/system

16.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/taskstore/mapred

17.[root@PRAVEEN PRAKASAN ~]# chmod 755 $HADOOP_STORE/taskstore/mapred

18.[root@PRAVEEN PRAKASAN ~]# chown mapred:hadoop $HADOOP_STORE/taskstore/mapred

19.[root@PRAVEEN PRAKASAN ~]# mkdir -p $HADOOP_STORE/tmp/pids

20.[root@PRAVEEN PRAKASAN ~]# chown -R hdfs $HADOOP_STORE/tmp/pids

11.IP hosts configuration

Edit the /etc/hosts:

1.[root@PRAVEEN PRAKASAN ~]# nano /etc/hosts

(add the master and slaves ip and hostname like this)

	192.168.1.2	master
	192.168.1.3	slave0
	192.168.1.4	slave1

12.Define the masters and slaves (Only on master node):

1.[root@PRAVEEN PRAKASAN ~]# nano $HADOOP_CONF/masters

    In this case hostname of master

    master

    Now modify the slaves file:

2.[root@PRAVEEN PRAKASAN ~]#nano $HADOOP_CONF/slaves

    Slaves in our example are slave0 and slave1:

    slave0

    slave1

 13.TO Configure Slaves_Datanode

14.Run Hadoop Cluster

14.1.NameNode Format

1.[hdfs@PRAVEEN PRAKASAN ~]# sudo –u hdfs $HADOOP_HOME/bin/hadoop namenode -format

 4.2.Deamons Run Command

1.[hdfs@PRAVEEN PRAKASAN ~]# start-all.sh

OR

1.[hdfs@PRAVEEN PRAKASAN ~]# start-dfs.sh

2.[mapred@PRAVEEN PRAKASAN ~]# start-mapred.sh

OR

1.[root@PRAVEEN PRAKASAN ~]# sudo -u hdfs /usr/lib/hadoop/bin/hadoop namenode

2.[root@PRAVEEN PRAKASAN ~]# sudo -u hdfs /usr/lib/hadoop/bin/hadoop secondarynamenode

3.[root@PRAVEEN PRAKASAN ~]# sudo -u hdfs /usr/lib/hadoop/bin/hadoop datanode

4.[root@PRAVEEN PRAKASAN ~]# sudo -u mapred /usr/lib/hadoop/bin/hadoop jobtracker

5.[root@PRAVEEN PRAKASAN ~]# sudo -u mapred /usr/lib/hadoop/bin/hadoop tasktracker

OR

1.[root@PRAVEEN PRAKASAN ~]# service hadoop-0.20-namenode start

2.[root@PRAVEEN PRAKASAN ~]# service hadoop-0.20-secondarynamenode start

3.[root@PRAVEEN PRAKASAN ~]# service hadoop-0.20-datanode start

4.[root@PRAVEEN PRAKASAN ~]# service hadoop-0.20-tasktracker start

5.[root@PRAVEEN PRAKASAN ~]# service hadoop-0.20-jobtracker start

OR

1.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/hadoop-0.20-namenode start

2.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/hadoop-0.20-secondarynamenode start

3.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/hadoop-0.20-datanode start

4.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/hadoop-0.20-tasktracker start

5.[root@PRAVEEN PRAKASAN ~]# /etc/init.d/hadoop-0.20-jobtracker start

15.MapReduce For Example

1.[root@PRAVEEN PRAKASAN ~]# cd  $HADOOP_HOME

2.[root@PRAVEEN PRAKASAN ~]#bin/hadoop jar hadoop-example.jar sleep –m 10 -r 10 -mt 10 -rt 10

-record 10