org.apache.hadoop.mapreduce.lib.input

Class CombineFileRecordReader<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.RecordReader<K,V>

org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader<K,V>

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class CombineFileRecordReader<K,V>
extends RecordReader<K,V>

A generic RecordReader that can hand out different recordReaders for each chunk in a CombineFileSplit. A CombineFileSplit can combine data chunks from multiple files. This class allows using different RecordReaders for processing these data chunks from different files.

See Also:

CombineFileSplit

Field Summary

Fields

Modifier and Type	Field and Description
protected TaskAttemptContext	context
protected RecordReader<K,V>	curReader
protected org.apache.hadoop.fs.FileSystem	fs
protected int	idx
protected long	progress
protected java.lang.Class<? extends RecordReader<K,V>>	rrClass
protected java.lang.reflect.Constructor<? extends RecordReader<K,V>>	rrConstructor
protected CombineFileSplit	split

Constructor Summary

Constructors

Constructor and Description
CombineFileRecordReader(CombineFileSplit split, TaskAttemptContext context, java.lang.Class<? extends RecordReader<K,V>> rrClass) A generic RecordReader that can hand out different recordReaders for each chunk in the CombineFileSplit.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Close the record reader.
K	getCurrentKey() Get the current key
V	getCurrentValue() Get the current value.
float	getProgress() return progress based on the amount of data processed so far.
void	initialize(InputSplit split, TaskAttemptContext context) Called once at initialization.
protected boolean	initNextRecordReader() Get the record reader for the next chunk in this CombineFileSplit.
boolean	nextKeyValue() Read the next key, value pair.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

split

protected CombineFileSplit split

rrClass

protected java.lang.Class<? extends RecordReader<K,V>> rrClass

rrConstructor

protected java.lang.reflect.Constructor<? extends RecordReader<K,V>> rrConstructor

fs

protected org.apache.hadoop.fs.FileSystem fs

context

protected TaskAttemptContext context

idx

protected int idx

progress

protected long progress

curReader

protected RecordReader<K,V> curReader

Constructor Detail

CombineFileRecordReader

public CombineFileRecordReader(CombineFileSplit split,
                       TaskAttemptContext context,
                       java.lang.Class<? extends RecordReader<K,V>> rrClass)
                        throws java.io.IOException

A generic RecordReader that can hand out different recordReaders for each chunk in the CombineFileSplit.

Throws:

java.io.IOException

Method Detail

initialize

public void initialize(InputSplit split,
              TaskAttemptContext context)
                throws java.io.IOException,
                       java.lang.InterruptedException

Description copied from class: RecordReader

Called once at initialization.

Specified by:

initialize in class RecordReader<K,V>

Parameters:

split - the split that defines the range of records to read

context - the information about the task

Throws:

java.io.IOException

java.lang.InterruptedException

nextKeyValue

public boolean nextKeyValue()
                     throws java.io.IOException,
                            java.lang.InterruptedException

Description copied from class: RecordReader

Read the next key, value pair.

Specified by:

nextKeyValue in class RecordReader<K,V>

Returns:

true if a key/value pair was read

Throws:

java.io.IOException

java.lang.InterruptedException

getCurrentKey

public K getCurrentKey()
                throws java.io.IOException,
                       java.lang.InterruptedException

Description copied from class: RecordReader

Get the current key

Specified by:

getCurrentKey in class RecordReader<K,V>

Returns:

the current key or null if there is no current key

Throws:

java.io.IOException

java.lang.InterruptedException

getCurrentValue

public V getCurrentValue()
                  throws java.io.IOException,
                         java.lang.InterruptedException

Description copied from class: RecordReader

Get the current value.

Specified by:

getCurrentValue in class RecordReader<K,V>

Returns:

the object that was read

Throws:

java.io.IOException

java.lang.InterruptedException

close

public void close()
           throws java.io.IOException

Description copied from class: RecordReader

Close the record reader.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in class RecordReader<K,V>

Throws:

java.io.IOException

getProgress

public float getProgress()
                  throws java.io.IOException,
                         java.lang.InterruptedException

return progress based on the amount of data processed so far.

Specified by:

getProgress in class RecordReader<K,V>

Returns:

a number between 0.0 and 1.0 that is the fraction of the data read

Throws:

java.io.IOException

java.lang.InterruptedException

initNextRecordReader

protected boolean initNextRecordReader()
                                throws java.io.IOException

Get the record reader for the next chunk in this CombineFileSplit.

Throws:

java.io.IOException