Hadoop JobConf

Job Conf

It is a primary interface created by the user to describe the map reduce concept in Hadoop main frame for execution. The frame works try to execute the function to the best, due to certain reasons as following:

Some parameters will be marked as final by the admin, it cannot be altered.

Some parameters are declared straight but, some other parameters are slightly related to the rest of the frame work and job configuration and is more complex.

In map reduce concept, job conf is used in mapper class, reducer class, partitioner, input format and output format. It is used in map reduce because optionally job conf is used in advanced effects such as Comparators and files in distributed cache.

The configuration using JobConf is as follows:

Step1: create a new JobConf,

JobConf job=new Jobconf(new Configuration(), classname.class);

Step2: Specify carious job specific parameters,
    It’s already mentioned jobconf is used to specify functions for mapper class, reducer class, input format, output format and partitioner. So they are configured using the new job conf object.

job.setJobName("myjob"); /* job name is set*/
    
     FileInputFormat.setInputPaths(job, new Path("in"));
     FileOutputFormat.setOutputPath(job, new Path("out"));
    
     job.setMapperClass(MyJob.MyMapper.class);
     job.setCombinerClass(MyJob.MyReducer.class);
     job.setReducerClass(MyJob.MyReducer.class);
    
     job.setInputFormat(SequenceFileInputFormat.class);
     job.setOutputFormat(SequenceFileOutputFormat.class);


it is included in the packages,
java.lang.Object
  extended by org.apache.hadoop.conf.Configuration
      extended by org.apache.hadoop.mapred.JobConf



Example with the wordcount program:

public static void main(String[] args) throws Exception {
    JobConf conf = new JobConf(WordCount.class); /* new job conf object is created.*/
    conf.setJobName("wordcount"); /* job name is set*/
       
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);
 /*The setOutputKeyClass() and setOutputValueClass() methods control the output types for the map and the reduce functions,*/
           
    conf.setMapperClass(Map.class);
    conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);
       
    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);
       
    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));
       
    JobClient.runJob(conf);/* JobClient is the primary interface for the user-job to interact with the cluster. JobClient provides facilities to submit jobs, track their progress, access component-tasks' reports/logs, get the Map-Reduce cluster status information etc.*/
 }