org.apache.hadoop.streaming

Class StreamBaseRecordReader

java.lang.Object

org.apache.hadoop.streaming.StreamBaseRecordReader

All Implemented Interfaces:

RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Direct Known Subclasses:

StreamXmlRecordReader

public abstract class StreamBaseRecordReader
extends java.lang.Object
implements RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Shared functionality for hadoopStreaming formats. A custom reader can be defined to be a RecordReader with the constructor below and is selected with the option bin/hadoopStreaming -inputreader ...

See Also:

StreamXmlRecordReader

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.apache.commons.logging.Log	LOG

Constructor Summary

Constructors

Constructor and Description
StreamBaseRecordReader(org.apache.hadoop.fs.FSDataInputStream in, FileSplit split, Reporter reporter, JobConf job, org.apache.hadoop.fs.FileSystem fs)

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Close this to future operations.
org.apache.hadoop.io.Text	createKey() Create an object of the appropriate type to be used as a key.
org.apache.hadoop.io.Text	createValue() Create an object of the appropriate type to be used as a value.
long	getPos() Returns the current position in the input.
float	getProgress() How much of the input has the RecordReader consumed i.e.
abstract boolean	next(org.apache.hadoop.io.Text key, org.apache.hadoop.io.Text value) Read a record.
abstract void	seekNextRecordBoundary() Implementation should seek forward in_ to the first byte of the next record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.apache.commons.logging.Log LOG

Constructor Detail

StreamBaseRecordReader

public StreamBaseRecordReader(org.apache.hadoop.fs.FSDataInputStream in,
                      FileSplit split,
                      Reporter reporter,
                      JobConf job,
                      org.apache.hadoop.fs.FileSystem fs)
                       throws java.io.IOException

Throws:

java.io.IOException

Method Detail

next

public abstract boolean next(org.apache.hadoop.io.Text key,
           org.apache.hadoop.io.Text value)
                      throws java.io.IOException

Read a record. Implementation should call numRecStats at the end

Specified by:

next in interface RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Parameters:

key - the key to read data into

value - the value to read data into

Returns:

true iff a key/value was read, false if at EOF

Throws:

java.io.IOException

getPos

public long getPos()
            throws java.io.IOException

Returns the current position in the input.

Specified by:

getPos in interface RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Returns:

the current position in the input.

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Close this to future operations.

Specified by:

close in interface RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

java.io.IOException

getProgress

public float getProgress()
                  throws java.io.IOException

Description copied from interface: RecordReader

How much of the input has the RecordReader consumed i.e. has been processed by?

Specified by:

getProgress in interface RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Returns:

progress from 0.0 to 1.0.

Throws:

java.io.IOException

createKey

public org.apache.hadoop.io.Text createKey()

Description copied from interface: RecordReader

Create an object of the appropriate type to be used as a key.

Specified by:

createKey in interface RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Returns:

a new key object.

createValue

public org.apache.hadoop.io.Text createValue()

Description copied from interface: RecordReader

Create an object of the appropriate type to be used as a value.

Specified by:

createValue in interface RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Returns:

a new value object.

seekNextRecordBoundary

public abstract void seekNextRecordBoundary()
                                     throws java.io.IOException

Implementation should seek forward in_ to the first byte of the next record. The initial byte offset in the stream is arbitrary.

Throws:

java.io.IOException