org.apache.hadoop.mapred.lib

Class TotalOrderPartitioner<K extends org.apache.hadoop.io.WritableComparable,V>

java.lang.Object

org.apache.hadoop.mapred.lib.TotalOrderPartitioner<K,V>

All Implemented Interfaces:

JobConfigurable, Partitioner<K,V>

public class TotalOrderPartitioner<K extends org.apache.hadoop.io.WritableComparable,V>
extends java.lang.Object
implements Partitioner<K,V>

Partitioner effecting a total order by reading split points from an externally generated source.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_PATH

Constructor Summary

Constructors

Constructor and Description
TotalOrderPartitioner()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(JobConf job) Read in the partition file and build indexing data structures.
int	getPartition(K key, V value, int numPartitions) Get the paritition number for a given key (hence record) given the total number of partitions i.e.
static java.lang.String	getPartitionFile(JobConf job) Get the path to the SequenceFile storing the sorted partition keyset.
static void	setPartitionFile(JobConf job, org.apache.hadoop.fs.Path p) Set the path to the SequenceFile storing the sorted partition keyset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_PATH

public static final java.lang.String DEFAULT_PATH

See Also:

Constant Field Values

Constructor Detail

TotalOrderPartitioner

public TotalOrderPartitioner()

Method Detail

configure

public void configure(JobConf job)

Read in the partition file and build indexing data structures. If the keytype is BinaryComparable and total.order.partitioner.natural.order is not false, a trie of the first total.order.partitioner.max.trie.depth(2) + 1 bytes will be built. Otherwise, keys will be located using a binary search of the partition keyset using the RawComparator defined for this job. The input file must be sorted with the same comparator and contain JobConf.getNumReduceTasks() - 1 keys.

Specified by:

configure in interface JobConfigurable

Parameters:

job - the configuration

getPartition

public int getPartition(K key,
               V value,
               int numPartitions)

Description copied from interface: Partitioner

Get the paritition number for a given key (hence record) given the total number of partitions i.e. number of reduce-tasks for the job.

Typically a hash function on a all or a subset of the key.

Specified by:

getPartition in interface Partitioner<K extends org.apache.hadoop.io.WritableComparable,V>

Parameters:

key - the key to be paritioned.

value - the entry value.

numPartitions - the total number of partitions.

Returns:

the partition number for the key.

setPartitionFile

public static void setPartitionFile(JobConf job,
                    org.apache.hadoop.fs.Path p)

Set the path to the SequenceFile storing the sorted partition keyset. It must be the case that for R reduces, there are R-1 keys in the SequenceFile.

getPartitionFile

public static java.lang.String getPartitionFile(JobConf job)

Get the path to the SequenceFile storing the sorted partition keyset.

See Also:

setPartitionFile(JobConf,Path)