Introduction to Hadoop

Apache Hadoop is a Java software framework that allows for the distributed processing of large data sets across clusters of computers spread across the world using a simple programming model.


Multiple autonomous systems appear as one, interacting via a message passing interface, no single point of failure. Hadoop has components which take care of all complexities for us and by using a simple map reduce framework we are able to harness the power of distributed computing without having to worry about complexities like fault tolerance, data loss. It has replication mechanism for data recovery and job scheduling and blacklisting of faulty nodes by a configurable blacklisting policy.


Following are major components.
1. Map-reduce (Job Tracker and task tracker)
2. Namenode  and Secondary namenode (A HDFS NameNode stores Edit logs and File system Image).
3. Datanode (Runs on slaves)
4. JobTracker (Runs on server)
5. TaskTracker (Runs on slaves)

Hadoop can be configured in the following three modes:
  1. Local Standalone (Non-distributed)
    All hadoop deamons run as a  single java process on a single system
    Useful for debugging


  2. Pseudo Distributed
    Daemons run a single-node
    Each  Hadoop Daemons runs in a separate java process


  3. Fully Distributed
    Master-slave Architecture
    One machine  is designed as the NameNode and  other as JobTracker (Can be  with same 

    machine as well)
    Reset of the machines in the cluster act as both DataNode and TaskTracker

MapReduce:
The map reduce framework is introduced by google. A simple and powerful interface that enables automatic parallelization and distribution of large-scale computations, combined with an implementation of this interface that achieves high performance on large clusters of commodity PCs. (definition by Google paper on mapred)


This broadly consists of two mandatory functions to implement: “Map and reduce”. A map is a function which is executed on each key-value pair from an input split, does some processing and emits again a key and value pair. After map and before reduce can begin there is a phase called shuffle which copies and sorts the output on key and aggregates values.


These key and “aggregated value” pairs are captured by reduce and outputs a reduced key value pair . This process is also called aggregation as you get values aggregated for a particular key as input to reduce method. Again in reduce you may play around as you want with key-values and what you emit know is also key value pairs which are dumped directly to a file.


Now simply by expressing a problem in terms of map-reduce we can execute a task in parallel and distribute it across a broad cluster and be relieved of taking care of all complexities of distributed computing. Indeed “Life made easy”, had you tried doing the same thing with MPI libraries you can understand the complexity there scaling to thousands or even hundreds of nodes.


There is a lot more going in map-reduce than just map-reduce. But the beauty of the hadoop is that it takes care of most of those things and a user may not dig into details for simply running a job, though it is good if one has knowledge of those features and can help in tuning parameters and thus improve efficiency and fault tolerance.


HDFS:
Hadoop has its own implementation of distributed file system called hadoop distributed filesystem, which is coherent and provides all facilities of a file system. It implements ACLs and provides a subset of usual UNIX commands for accessing or querying the filesystem and if one mounts it as a fuse dfs then it is possible to access it as any other linux filesystem with standard unix commands.


Namenode:
A single point of failure for an HDFS installation. It contains information regarding a block’s location as well as the information of entire directory structure and files. By saying It is a single point of failure - I mean, if namenode goes down - whole filesystem is offline. Hadoop also has a secondary namenode which contains edit log, which in case of a failure of namenode can be used to replay all the actions of the filesystem and thus restore the state of the filesystem. A secondary namenode regularly contacts namenode and takes checkpointed snapshot images. At any time of failure these checkpointed images can be used to restore the namenode. Current efforts are going on to have high availability for Namenode.


Datanode:
Datanode stores actual blocks of data and stores and retrieves blocks when asked. They periodically report back to Namenode with list of blocks they are storing.


Jobtracker and tasktracker:
There is one JobTracker(is also a single point of failure) running on a master node and several tasktracker running on slave nodes. Each tasktracker has multiple task-instances running and every task tracker reports to jobtracker in the form of heart beat at regular intervals which also carries message of the progress of the current job it is executing and idle if it has finished executing. Jobtracker schedules jobs and takes care of failed ones by re-executing them on some other nodes. In Mrv2 efforts are made to have high availability for Jobtracker, which would definitely change the way it has been.