org.apache.hadoop.mapreduce.lib.output

Class FileOutputCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

public class FileOutputCommitter
extends OutputCommitter

An OutputCommitter that commits files specified in job output directory i.e. ${mapred.output.dir}.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	SUCCEEDED_FILE_NAME
protected static java.lang.String	TEMP_DIR_NAME Temporary directory name

Constructor Summary

Constructors

Constructor and Description
FileOutputCommitter(org.apache.hadoop.fs.Path outputPath, TaskAttemptContext context) Create a file output committer

Method Summary

Methods

Modifier and Type	Method and Description
void	abortJob(JobContext context, JobStatus.State state) Delete the temporary directory, including all of the work directories.
void	abortTask(TaskAttemptContext context) Delete the work directory
void	cleanupJob(JobContext context) Deprecated.
void	commitJob(JobContext context) Delete the temporary directory, including all of the work directories.
void	commitTask(TaskAttemptContext context) Move the files from the work directory to the job output directory
org.apache.hadoop.fs.Path	getWorkPath() Get the directory that the task should write results into
boolean	needsTaskCommit(TaskAttemptContext context) Did this task write any files in the work directory?
void	setupJob(JobContext context) Create the temporary directory that is the root of all of the task work directories.
void	setupTask(TaskAttemptContext context) No task setup required.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEMP_DIR_NAME

protected static final java.lang.String TEMP_DIR_NAME

Temporary directory name

See Also:

Constant Field Values

SUCCEEDED_FILE_NAME

public static final java.lang.String SUCCEEDED_FILE_NAME

See Also:

Constant Field Values

Constructor Detail

FileOutputCommitter

public FileOutputCommitter(org.apache.hadoop.fs.Path outputPath,
                   TaskAttemptContext context)
                    throws java.io.IOException

Create a file output committer

Parameters:

outputPath - the job's output path

context - the task's context

Throws:

java.io.IOException

Method Detail

setupJob

public void setupJob(JobContext context)
              throws java.io.IOException

Create the temporary directory that is the root of all of the task work directories.

Specified by:

setupJob in class OutputCommitter

Parameters:

context - the job's context

Throws:

java.io.IOException - if temporary output could not be created

commitJob

public void commitJob(JobContext context)
               throws java.io.IOException

Delete the temporary directory, including all of the work directories. This is called for all jobs whose final run state is SUCCEEDED

Overrides:

commitJob in class OutputCommitter

Parameters:

context - the job's context.

Throws:

java.io.IOException

cleanupJob

@Deprecated
public void cleanupJob(JobContext context)
                throws java.io.IOException

Deprecated.

Description copied from class: OutputCommitter

For cleaning up the job's output after job completion

Overrides:

cleanupJob in class OutputCommitter

Throws:

java.io.IOException

abortJob

public void abortJob(JobContext context,
            JobStatus.State state)
              throws java.io.IOException

Delete the temporary directory, including all of the work directories.

Overrides:

abortJob in class OutputCommitter

Parameters:

context - the job's context

state - final run state of the job, should be FAILED or KILLED

Throws:

java.io.IOException

setupTask

public void setupTask(TaskAttemptContext context)
               throws java.io.IOException

No task setup required.

Specified by:

setupTask in class OutputCommitter

Parameters:

context - Context of the task whose output is being written.

Throws:

java.io.IOException

commitTask

public void commitTask(TaskAttemptContext context)
                throws java.io.IOException

Move the files from the work directory to the job output directory

Specified by:

commitTask in class OutputCommitter

Parameters:

context - the task context

Throws:

java.io.IOException - if commit is not

abortTask

public void abortTask(TaskAttemptContext context)
               throws java.io.IOException

Delete the work directory

Specified by:

abortTask in class OutputCommitter

Throws:

java.io.IOException

needsTaskCommit

public boolean needsTaskCommit(TaskAttemptContext context)
                        throws java.io.IOException

Did this task write any files in the work directory?

Specified by:

needsTaskCommit in class OutputCommitter

Parameters:

context - the task's context

Returns:

true/false

Throws:

java.io.IOException

getWorkPath

public org.apache.hadoop.fs.Path getWorkPath()
                                      throws java.io.IOException

Get the directory that the task should write results into

Returns:

the work directory

Throws:

java.io.IOException