Hadoop streaming is a utility that is used along with distributed system. This allows a user to create and run map reduce jobs with any other executable language, and script (shell scripting) as the mapper and the reducer. Before starting work with streaming. The path and the directories have to be set as in Java. The path have to be set for input, output, mapper, combiner, and reducer.
input <path> DFS input file(s) for the Map step
-output <path> DFS output directory for the Reduce step
-mapper <cmd|JavaClassName> The streaming command to run
-combiner <JavaClassName> Combiner has to be a Java class
-reducer <cmd|JavaClassName> The streaming command to run
The mapper and the reducer will be read from the appropriate directories using stdin and stdout commands. Then the whole utility will create a job, submit the job to appropriate cluster and monitor the job until it completes. When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper.
When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the the rest of the line (excluding the tab character) is the value.
In practical application , streaming in hadoop can be developed with limited knowledge of java. But in developing two shell script have to write. They are mapper.sh and reducer.sh . Both files have to work in sync. The real place the logical correspondence breaks down is that in a one machine scripting environment shellMapper.sh and shellReducer.sh will each run as a single process and data will flow directly from one process to the other. With Hadoop the shellMapper.sh file will be sent to every machine on the cluster that has data chunks and each such machine will run it's own chunk through the shellMapper.sh process on each machine. The output from those scripts doesn't run a reduce on each of those machines. Instead the output is sorted so that different lines from various mapping jobs are streamed across the network to different machines (Hadoop defaults to four machines) where the reduce(s) can be performed.
While job submission, the executable may not be in the system. if they don't, you will need to use "-file" option to tell the framework to pack your executable files as a part of job submission.
-file myPythonScript.py
-file myDictionary.txt
For mapping jobs only, Often want to process input data using a map function only. To do this, simply set mapred.reduce.tasks to zero. The map/reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.
Fro adding additional plugins, just as map and reduce process, specify other plugins also.
-inputformat JavaClassName
-outputformat JavaClassName
-partitioner JavaClassName
-combiner JavaClassName
The class supplied for the input format will should return the key/value pairs from the input format class. If it is not mentioned TextInputFormat is used as the default. The -cacheFile and -cacheArchive options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can specify additional configuration variables by using jobconf.
There are so many other options that can be specified in the streaming. They are:
input <path> DFS input file(s) for the Map step
-output <path> DFS output directory for the Reduce step
-mapper <cmd|JavaClassName> The streaming command to run
-combiner <JavaClassName> Combiner has to be a Java class
-reducer <cmd|JavaClassName> The streaming command to run
The mapper and the reducer will be read from the appropriate directories using stdin and stdout commands. Then the whole utility will create a job, submit the job to appropriate cluster and monitor the job until it completes. When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper.
When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the the rest of the line (excluding the tab character) is the value.
In practical application , streaming in hadoop can be developed with limited knowledge of java. But in developing two shell script have to write. They are mapper.sh and reducer.sh . Both files have to work in sync. The real place the logical correspondence breaks down is that in a one machine scripting environment shellMapper.sh and shellReducer.sh will each run as a single process and data will flow directly from one process to the other. With Hadoop the shellMapper.sh file will be sent to every machine on the cluster that has data chunks and each such machine will run it's own chunk through the shellMapper.sh process on each machine. The output from those scripts doesn't run a reduce on each of those machines. Instead the output is sorted so that different lines from various mapping jobs are streamed across the network to different machines (Hadoop defaults to four machines) where the reduce(s) can be performed.
While job submission, the executable may not be in the system. if they don't, you will need to use "-file" option to tell the framework to pack your executable files as a part of job submission.
-file myPythonScript.py
-file myDictionary.txt
For mapping jobs only, Often want to process input data using a map function only. To do this, simply set mapred.reduce.tasks to zero. The map/reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.
Fro adding additional plugins, just as map and reduce process, specify other plugins also.
-inputformat JavaClassName
-outputformat JavaClassName
-partitioner JavaClassName
-combiner JavaClassName
The class supplied for the input format will should return the key/value pairs from the input format class. If it is not mentioned TextInputFormat is used as the default. The -cacheFile and -cacheArchive options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can specify additional configuration variables by using jobconf.
There are so many other options that can be specified in the streaming. They are:
- Switching between local hadoop and remote clusters.
- dfs configuration
- job tracker configuration
- passing environment variable to streaming commands.
- File uploaded to HDFS
No comments:
Post a Comment