org.apache.hadoop.mapreduce.lib.db

Class DBInputFormat<T extends DBWritable>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.LongWritable,T>

org.apache.hadoop.mapreduce.lib.db.DBInputFormat<T>

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

DataDrivenDBInputFormat, DBInputFormat

public class DBInputFormat<T extends DBWritable>
extends InputFormat<org.apache.hadoop.io.LongWritable,T>
implements org.apache.hadoop.conf.Configurable

A InputFormat that reads input data from an SQL table.

DBInputFormat emits LongWritables containing the record number as key and DBWritables as value. The SQL query, and input class can be using one of the two setInput methods.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DBInputFormat.DBInputSplit A InputSplit that spans a set of rows
static class	DBInputFormat.NullDBWritable A Class that does nothing, implementing DBWritable

Constructor Summary

Constructors

Constructor and Description
DBInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	closeConnection()
protected RecordReader<org.apache.hadoop.io.LongWritable,T>	createDBRecordReader(DBInputFormat.DBInputSplit split, org.apache.hadoop.conf.Configuration conf)
RecordReader<org.apache.hadoop.io.LongWritable,T>	createRecordReader(InputSplit split, TaskAttemptContext context) Create a record reader for a given split.
org.apache.hadoop.conf.Configuration	getConf()
java.sql.Connection	getConnection()
protected java.lang.String	getCountQuery() Returns the query for getting the total number of rows, subclasses can override this for custom behaviour.
DBConfiguration	getDBConf()
java.lang.String	getDBProductName()
java.util.List<InputSplit>	getSplits(JobContext job) Logically split the set of input files for the job.
void	setConf(org.apache.hadoop.conf.Configuration conf)
static void	setInput(Job job, java.lang.Class<? extends DBWritable> inputClass, java.lang.String inputQuery, java.lang.String inputCountQuery) Initializes the map-part of the job with the appropriate input settings.
static void	setInput(Job job, java.lang.Class<? extends DBWritable> inputClass, java.lang.String tableName, java.lang.String conditions, java.lang.String orderBy, java.lang.String... fieldNames) Initializes the map-part of the job with the appropriate input settings.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DBInputFormat

public DBInputFormat()

Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getDBConf

public DBConfiguration getDBConf()

getConnection

public java.sql.Connection getConnection()

getDBProductName

public java.lang.String getDBProductName()

createDBRecordReader

protected RecordReader<org.apache.hadoop.io.LongWritable,T> createDBRecordReader(DBInputFormat.DBInputSplit split,
                                                                     org.apache.hadoop.conf.Configuration conf)
                                                                                             throws java.io.IOException

Throws:

java.io.IOException

createRecordReader

public RecordReader<org.apache.hadoop.io.LongWritable,T> createRecordReader(InputSplit split,
                                                                   TaskAttemptContext context)
                                                                                        throws java.io.IOException,
                                                                                               java.lang.InterruptedException

Create a record reader for a given split. The framework will call RecordReader.initialize(InputSplit, TaskAttemptContext) before the split is used.

Specified by:

createRecordReader in class InputFormat<org.apache.hadoop.io.LongWritable,T extends DBWritable>

Parameters:

split - the split to be read

context - the information about the task

Returns:

a new record reader

Throws:

java.io.IOException

java.lang.InterruptedException

getSplits

public java.util.List<InputSplit> getSplits(JobContext job)
                                     throws java.io.IOException

Logically split the set of input files for the job.

Each InputSplit is then assigned to an individual Mapper for processing.

Note: The split is a logical split of the inputs and the input files are not physically split into chunks. For e.g. a split could be <input-file-path, start, offset> tuple. The InputFormat also creates the RecordReader to read the InputSplit.

Specified by:

getSplits in class InputFormat<org.apache.hadoop.io.LongWritable,T extends DBWritable>

Parameters:

job - job configuration.

Returns:

an array of InputSplits for the job.

Throws:

java.io.IOException

getCountQuery

protected java.lang.String getCountQuery()

Returns the query for getting the total number of rows, subclasses can override this for custom behaviour.

setInput

public static void setInput(Job job,
            java.lang.Class<? extends DBWritable> inputClass,
            java.lang.String tableName,
            java.lang.String conditions,
            java.lang.String orderBy,
            java.lang.String... fieldNames)

Initializes the map-part of the job with the appropriate input settings.

Parameters:

job - The map-reduce job

inputClass - the class object implementing DBWritable, which is the Java object holding tuple fields.

tableName - The table to read data from

conditions - The condition which to select data with, eg. '(updated > 20070101 AND length > 0)'

orderBy - the fieldNames in the orderBy clause.

fieldNames - The field names in the table

See Also:

setInput(Job, Class, String, String)

setInput

public static void setInput(Job job,
            java.lang.Class<? extends DBWritable> inputClass,
            java.lang.String inputQuery,
            java.lang.String inputCountQuery)

Initializes the map-part of the job with the appropriate input settings.

Parameters:

job - The map-reduce job

inputClass - the class object implementing DBWritable, which is the Java object holding tuple fields.

inputQuery - the input query to select fields. Example : "SELECT f1, f2, f3 FROM Mytable ORDER BY f1"

inputCountQuery - the input query that returns the number of records in the table. Example : "SELECT COUNT(f1) FROM Mytable"

See Also:

setInput(Job, Class, String, String, String, String...)

closeConnection

protected void closeConnection()